Skip to content

FOSDEM 2014

As usual, there will be a lot of Mozillians attending FOSDEM this year, including 3 members of the graphics team. Bas Schouten will give a talk about Utilizing GPUs to accelerate 2D content on Saturday (16:30 in the Mozilla devroom). It’s going to be a very interesting and also fairly technical talk. If you are interested in the challenges of hardware accelerated 2D rendering, we hope to see you there!

And as Mozilla is not all about graphics, there will be plenty of other interesting talks in the Mozilla devroom this year that you should checkout!

Removing old OpenGL layers

This post is only interesting for advanced Firefox users on Linux who manually activated OpenGL compositing.

On more and more platforms we perform compositing in a separate thread from content rendering. This is awesome for smooth panning and zooming, as well as smooth video playback and CSS animations. We refer to this as “off-main-thread compositing” (OMTC).

Currently, we use OMTC on android, Firefox OS, Mac, and very soon on (some flavors of) Windows. On Linux, it can be activated but it is turned off by default as there are some bugs to fix (just like hardware accelerated compositing).

Main-thread compositing and off-main-thread compositing have a different infrastructure and as we move toward using OMTC, the main-thread compositing code becomes more and more obsolete, and today we don’t ship main-thread OpenGL compositing on any platform by default.

Very soon we will remove the main-thread OpenGL compositing code. This will feel great because that’s a few thousand lines of code that we won’t have to support anymore. This will not affect any user running the default configuration. However, for adventurous users that manually opted in OpenGL compositing on Linux, compositing will suddenly fallback to the CPU.

Don’t panic.

OpenGL compositing will not be gone. It will be gone for main-thread compositing only. If you opted in for OpenGL compositing on Linux, then you will still be able to activate OMTC and enjoy OpenGL compositing there.

At the moment the only place where OpenGL layers on linux is really useful is WebGL, because compositing on the CPU forces reading back the output of the WebGL canvas, which is slow. For other use cases, OpenGL compositing trades faster blitting with slower texture uploads (as we are not yet using a good solution for direct texturing on Linux such as texture from pixmap).

So, if you decide that you want GPU compositing on Linux, here is how to activate OMTC on this platform:

In about:config, set the following prefs to “true”

  • layers.offmainthreadcomposition.enabled
  • layers.acceleration.force-enabled
  • layers.async-video.enabled (this is optional)

Also set this pref to “false”

  • layers.use-deprecated-textures

As I write this post, we have not yet removed main-thread OpenGL layers. Power users don’t need to jump on OMTC yet, unless they want to report lots of bugs! I will write another blog post when we actually make the change in nightly (soon-ish) and when the change hits the stable version of Firefox.

Again, this does not affect the vast majority of our users.

Edit: We recently removed the need for an environment variable.

Looking for a good first place to contribute to Gecko gfx?

I heard we don’t have enough mentored bugs filed for the gfx code. If you are interested in contributing to Gecko’s graphics code, read on.

Contributing to Gecko for the first time can be scary because Gecko is a complex beast and it is very easy to get overwhelmed by the amount of code. Trying to understand it all is impossible, and knowing what parts are important to understand and where to start is hard when you approach a code base for the first time.

This is why mentoring bugs is useful.
A mentored bug is a bug that is approachable for new contributors, and that has someone who feels familiar with the code volunteering to mentor.
The mentor a bug already has a good understanding of the code and can help new contributors get started, show them where to look and explain the non-trivial things that may get in the way. Actually, the mentor probably already knows how to fix the bug, but he chooses to let someone else do it because he thinks the task is a good way to get started contributing.

Josh Matthews made a pretty cool tool to help finding mentored bugs.

Enough with the introduction, let’s talk about a good place where newcomers can start contributing to Gecko’s graphics code.

In gecko we have two drawing APIs:

Thebes, which is for the most part a wrapper around Cairo, is what we have been using for a while.

Moz2D, a higher level API that has several backends (including Cairo) is a more recent API that we want to use instead of thebes. Migrating from Thebes to Moz2D is a long process that has been going on for a while and will keep us busy for while too. It is a very good place to start contributing, because a lot of it isn’t very hard. Both APIs fulfill the same roles. So in many cases it is mostly about finding uses of thebes class (such as gfxIntSize, gfxIntRect, gfxImageSurface, etc.) and replace them with Moz2D equivalent (gfx::IntSize, gfx::IntRect, gfx::DataSourceSurface, etc.).

Moving from Thebes to Moz2D is important for us to make Gecko’s code more awesome because it makes Gecko easier to maintain and lets us build our graphics stack on top of better and more future-proof foundations.

We already filed a few mentored bugs to port from Thebes to Moz2D, such as https://bugzilla.mozilla.org/show_bug.cgi?id=882113 and the dependent bugs, and already received contributions, which is great! More of these bugs will be filed as we still have a lot of Thebes in the code.

If you are interested in helping there, the rest of this post contains some useful information:

Moz2D code lives in the gfx/2d/ directory.
Thebes code lives in the gfx/thebes/ directory.
gfx2DGlue.h contains a lot of small helper to convert from one API to the other.

Edit: For the curious, more info about Moz2D here: https://wiki.mozilla.org/Platform/GFX/Moz2D

Converting the simple classes like size, point and rectangle classes is rather trivial because they mostly have the same interfaces.
A class that is also interesting to convert is Thebes’ gfxImageSurface.
gfxImageSurface represents an image which pixels are stored in memory and can be initialized and accessed through pointers (as opposed to, say a texture on the GPU or some vector shapes that haven’t been rasterized yed).
Typically we use gfxImageSurface when we have some producer code that creates an image in memory and we want to wrap it in an object:

RefPtr<gfxImageSurface> thebesSurface = new gfxImageSurface(dataPointer, size, stride, format);
// we can use this image as any other surface type and also access the buffer through pointers

Or when we want to access an image’s buffer to pass it to some other API that doesn’t talk in term of thebes surfaces (like OpenGL texture uploads).

So for these two use cases we need an image abstraction that ensures that the image data is constituted of pixels and accessible by pointers, and that is what gfxImageSurface provides.
The Moz2D equivalent is mozilla::gfx::DataSourceSurface.
It offers pretty much the same thing and passing from one to the other in the use cases described above is rather easy.

One difference though is that while gfxImageSurface can be instanciated directly with the new operator, DataSourceSurface is created using gfx::Factory::CreateDataSourceSurface and gfx::Factory::CreateWrappingDataSourceSurface (static methods so you don’t need to hold on to a Factory object).

Some pieces of advise:

  • It is very easy to fall in the snowball effect of converting one function parameter from Thebes to Moz2D and then fix all the code that depends on it, and then all the code that depends on that code, etc. Don’t hesitate to make small patches that convert a method or a class to Moz2D and then use the helpers in gfx2DGlue.h to do the glue on the calling code. Then on a subsequent patch the calling code can be ported to Moz2D, etc. This way we avoid hug monster-patches that are awful to review and rebase. Using mercurial queues help a lot with this workflow (having several patches in a certain order and going back and forth between them to modify them).
  • When the task is to do some mechanical conversion, you can try to understand the surrounding code out of curiosity but you really don’t need to. I have seen motivated new contributors trying to understand all of Gecko’s graphics code before doing a simple patch and ending up not doing the patch because understanding everything all at once is hard and discouraging. Try to limit yourself to what you think is needed for the task. When in doubt ask questions to the mentor of the bug, ask about what parts of the code are important to understand, and don’t hesitate voice what you intend to do before doing it so that the mentor can tell you if your are going in the wrong direction before you have spent too much time on it.
  • Again, ask questions. On the bug, email the mentor and on IRC (#gfx about graphics stuff #introduction about problems like building firefox). Sometimes on IRC someone asks a question and nobody answers it. This may feel intimidating and rude, but in fact it just means no-one that is watching IRC at this very moment knows the answer to your question. Perhaps the people who could answer are in a different time-zone or a different IRC channel, In any case you can send an email or ping your mentor on IRC and if he is not swamped with work he will try to help you asap or direct you to the right person to ask.

So if you are looking to contribute to gfx but haven’t yet found a starting point, this is a good one.

Hardware acceleration and compositing

Let’s talk about some common misconceptions about hardware acceleration.

Hardware acceleration conveys the idea that some parts of the rendering use the GPU in order to speed things up.

In a web browser there are two main topics for hardware acceleration:

  • Hardware accelerated content rendering: Drawing content on the GPU, like rendering text, shadows, shapes, etc. I also place video decoding in that bag but it is a somewhat different topic and I am not going to talk about it in this post.
  • Hardware accelerated compositing: We render different elements of a web page in different layers (think of layers as photoshop layers) and compositing is the action of flattening all these layers into the final image that gets to the screen.

The former is generally very hard to achieve. Rendering content on the GPU is hard because GPUs are not always good at it. But it depends and there are things that we can do efficiently on the GPU and things that we can’t do efficiently on the GPU. For example graphics cards are not good at rendering text (text that looks good). Some canvas operations are very fast on the gpu (like blitting surfaces) but others are not (drawing bezier curves and shapes in general), so there is a trade-off and “hardware-accelerated canvas” is not always “accelerated” depending on what you are doing with your canvas.

The latter, GPU compositing, is a much more interesting optimization for browser implementors because it is a simple task that the GPU is very good at doing: blitting quads. Gecko performs compositing on the GPU except when the graphics card model or driver is black-listed and unfortunately, except on desktop Linux because we haven’t yet dedicated enough resources to support it efficiently (it can be activated manually but the results will not always be better). Note that Linux accelerated compositing is a very good area to contribute to.

When people around the internet speak of hardware acceleration in web browsers, they almost always refer to accelerated compositing.

Now let’s talk about a very bad misconception that has spread: I have read in various places on the internet that using 3D transforms enable hardware acceleration. This is very wrong. Gecko, Webkit, Blink, etc. use heuristic to determine when elements of a web page should be placed in a separate layer. Doing so let’s us do things like not invalidating (and re-drawing) an element that is overlapping with something that is animated, and some other optimizations. Applying a 3D transform to an element just happens to trigger this element to have its own layer. Layerizing an element is different from enabling hardware acceleration. when the browser starts up, it decides whether it will do compositing on the CPU or the GPU, and it won’t switch between the two because someone applied a transform on some element of a web page.

Layerizing an element is good if the element is going to be animated, but it is not something that will just make things faster. It can, and most likely will, use more memory because it will require us to allocate an extra texture to host the pixels of the element. For example, Gecko tends to layerize elements during a CSS transition, and merge back the element with the rest as soon as the transition is over, to avoid having too many layers at the same time.

Sometimes people blog about how applying a 3D transform can accelerate rendering, then other people read it and believe that applying transforms is the silver bullet to making web pages faster. Please, please, don’t blindly do things like that. I realize that we need to document more what layers acceleration in a web browser is, and how compositing works. Thankfully most browsers today have roughly the same notions of layers and compositing, so talking about layers in Gecko or Webkit should be equivalent from a web developer’s point of view. Layerizing for the sake of layerizing can degrade performances and memory usage. Heuristics evolve, and using hacks in websites today can prevent us from doing clever optimizations in the browser tomorrow.

So only use this kind of hacks when you understand what it actually does in the browser and when you can observe a real difference. Always remember that using a hack for something it is not meant to can probably speed things up in some browser today, and make things worse later as browsers evolves.

FOSDEM 2013

We’ve been quite busy lately and, I must confess, not very good at feeding this blog. Sorry about that. If you want some news about Gfx and you are attending FOSDEM this year, I invite you to see the the two talks by Gfx members:

In the first one, Sunday in the morning, I will give a quick overview of some parts of our rendering pipeline, and focus on the Layers system that we are in the process of refactoring. I think it should be an interesting talk for anyone curious about how things work under the hood, or for those who would like to contribute specifically on the topic of layers and off-main-thread compositing.

https://fosdem.org/2013/schedule/event/geckos_asynchronous_compositing_architecture/

In the second talk (Sunday afternoon), Bas will give useful pointers for people interested in contributing to graphics in Gecko.

https://fosdem.org/2013/schedule/event/contributing_to_graphics_in_gecko/

There are plenty of other cool Mozilla talks at FOSDEM this year.

See you tomorrow morning!

How to help testing off-main-thread compositing

I have been asked several times on irc and Bugzilla about how to help getting off-main-thread compositing (OMTC) ready to be turned on by default (especially by Linux users). It is awesome to hear from such people and yes, you can help. This blog post is about reporting bugs for off-main-thread compositing, a feature that works very well on Firefox for Android and Firefox OS but is in development on Linux and Mac, and not yet started on Windows. On this specific topic, you can help us if you are using Linux or Mac. I will give informations about testing on Windows when the Windows implementation is ready, which should happen soon.
If you are asking “how can I help?”, it most likely means that you are not familiar with Firefox’s source code, and it is all right. We need a lot of testing. Since OMTC uses hardware acceleration, it is subject to driver bugs and problems can vary from one machine to another. We need people to test and give us feedbacks about their experience using OMTC:
  •  is it improving/regressing your user experience? In what ways?
  •  do you see rendering bugs when OMTC?  If so, on which web pages?
  •  is it crashing?
  •  most likely the experience will be less smooth at this point, since we have not played with optimizing OMTC on desktop platforms yet. However if it turns out to be faster it’s worth.
Of course, it is also possible to help us by writing code, and some community members do help us this way, but this is not the topic of this post. If you’re interested you can check out https://wiki.mozilla.org/Platform/GFX/Contribute.

Setup a testing profile in Firefox nightly

OMTC is changing a lot, so we need feedbacks from people using nightly builds of Firefox (no need to build it from source, you can download Firefox Nightly from here: http://nightly.mozilla.org/ )
First, create a separate profile. A profile contains your preferences, and we are about to try experimental features, so you don’t want to break your default profile, so we will make sure you can try the fancy stuff without breaking your normal user experience.
open a terminal and go to the directory containing Firefox Nightly.
Type:
./firefox --no-remote -P
This will open the profile manager. from there you can create a new profile. You will have to give it a name, I will assume you named it “testing”, but if you give it a different name, just remember to put the name you choose in place of testing in the following instructions. Do not tick the “Don’t ask at startup” option because it would set it as your default profile which you don’t want until you are sure that it will not break.
We will now make sure that hardware acceleration is available by typing about:config in the address bar, and setting the following preferences to “true”:
  • layers.acceleration.force-enabled
  • layers.acceleration.draw-fps
Then create a new window by detaching a tab or pressing ctrl+n.
You should see one or two pink counters appear on the top left of every new window. The content of the counters is not interesting to us in this case, but the presence of the counter means that layers acceleration is enabled and used.
If the pink counter did not appear, it means that Firefox could not enable layers acceleration, which I think is pretty rare. At this point you can’t help us testing OMTC yet (until we implement OMTC on top of non-accelerated layers), sorry… But there may be other areas you can test and give us feedbacks on! You can connect to the #developers and/or #gfx channels at irc.mozilla.org with an irc client and tell us that you want to help testing new features, and I am sure someone will be thrilled to hear that!
Back to testing OMTC:
you just checked that layers acceleration works. It does. Sweet. Keep layers acceleration enabled and if you are on linux close Firefox.
Now there are two scenarios:
  • you are using Linux:
    you need to set an environment variable before running Nightly. This is the most annoying part. in your terminal, in the directory containing Firefox Nightly, type the following command:
 export MOZ_USE_OMTC=1

start firefox again by entering in your terminal

./firefox --no-remote -p testing
  • you are using OS X
    • no need to set an environment vairable. start firefox with the testing profile by typing ./firefox –no-remote -p testing if you closed it, and set the preference layers.offmainthreadcomposition.enabled in about:config to “true”.
You are now running off-main-thread compositing \o/
Before you start playing around, also set to “true” the following preferences in about:config
  • layers.offmainthreadcomposition.animate-transform
  • layers.offmainthreadcomposition.animate-opacity
  • layers.async-video.enabled
These three features are using the OMTC architecture to move some of the animations off the main thread, making them smooth even when the browser is slowed down by a heavy javascript workload or an addon.
The inital setup was a bit tedious but you will not have to repeat all the steps every time. all the options that you have set in about:config are stored in the testing profile, so you will not have to re do these steps.
If you are on linux you will,  however, have to set the environment variable MOZ_USE_OMTC each time. I recommend not to place the command for the environment variable in your bashrc/zshrc/etc. because if your non-testing instances of Firefox see this environment variable, they will try to enable OMTC, and you don’t want to risk breaking your non-testing instances.
When you want to run your testing instance of firefox, run
./firefox --no-remote -p testing 
from a terminal in the directory containing Firefox Nigthly.

Now let’s test.

Browse the web, use your browser like you normally would, think of whether your experience is better or worse than before, and what exactly is better or worse.

Report the bugs.

To report problems, please use https://bugzilla.mozilla.org
Create a bug in the product “Core” and the component Graphics: Layers.
In the “CC:” field, add “:nical, :BenWa, :bjacob” without the quotes, so that I, and a few others from the gfx team, receive notifications about the bug.
In the “Blocks:” field, put “722012″ without quotes, if you are using Linux, or “756601″ if you are using OS X. This will tell us that there is a dependency between OMTC Linux/Mac and the bug you found, maximizing the chances that we see your bug in the first place and that  we will see it when we are looking for bugs related to OMTC.
When you fill in the description of the bug, try to give us as many informations as you can. We are interested in what OS you use, which version, what graphics card you have…
If you are on linux, type
glxinfo
in a terminal and copy-paste the result of this command in the description of the bug. glxinfo provides with informations about your graphics card, the version of its pilots and what are the capabilities of hardware acceleration on your machine, which is helpful for us to diagnose the problems.
Think of a bug in Bugzilla has anything that is misbehaving compared to the stable version of Firefox. If the page is not rendered correctly, or firefox is crashing, or some things like scrolling getting slow… Anything that represents a regression compared to stable Firefox is a bug for OMTC.
If you are familiar with tools like gdb, providing us with a backtrace of the crash if you find one is extremely helpful.
to run firefox inside gdb, you can type:
gdb --args ./firefox --no-remote -p testing
then type r and press enter.
If firefox crashes while running in gdb, you can get the backtrace by typing bt and pressing enter. Then you just have to copy-paste the backtrace in the description of the bug.

If you need help

I just wrote a lot of things in here, and it may be confusing. I don’t want to scare you away, so don’t hesitate to ask me questions on the #gfx irc channel at irc.mozilla.org. My nick name on irc is nical, and I’ll do my best to help you if I am around when you ask. Note that I also speak French, in case you are a French speaker and not at ease with conversing in English.
Thank you to all the people who have proposed to help us. The making of Firefox does not only involve only Mozilla’s staff. Firefox is the making of its entire community without which it would not nearly be as awesome as it is today.

Created the Mozilla Gfx Team Blog

The primary purpose of this blog is to help communication across and around the Mozilla Gfx team. In particular, we want to help non-employee contributors be better connected and better informed of any relevant developments.

In particular, we aim for this blog to have:

  • team goal/project updates;
  • announcements of new projects, or developments in existing projects;
  • early feature announcements;
  • posts introducing anyone in the gfx team, regardless of employee status;
  • posts from anyone in the gfx team, regardless of employee status.
Follow

Get every new post delivered to your Inbox.