Android Performance Case Study Follow-up

Two years ago, I published an articled titled Android Performance Case Study to help Android developers understand what tools and technique can be used to identify, track down, and fix performance issues.

This article focused on Falcon Pro, a Twitter client designed and developed by Joaquim Vergès. Joaquim was kind enough to let me use his application in my article and quickly addressed all the issues I found. All was well until Joaquim started working on Falcon Pro 3, written from scratch. Shortly before releasing his new application, Joaquim contacted me because he needed help figuring out a performance problem that was affecting scrolling (and once again, I did not have access to the source code).

Joaquim used all the right tools and was able to quickly determine what was not causing the issue. For instance, he found that overdraw was not an issue. He was however able to narrow down the problem to the use of a ViewPager. He sent me the following screenshots:

Falcon Pro 3

Joaquim used the system’s on-screen GPU profiling tool to detect framerate drops. The screenshot on the left shows the performance of scrolling a timeline without a ViewPager and the screenshot on the right shows performance with a ViewPager (he used a 2014 Moto X to capture this data). The root cause seems pretty obvious.

My first idea was to see whether the ViewPager was somehow misusing hardware layers. The performance issue we observed could have been caused by a hardware layer updated on every frame by the list’s scroll. The system’s hardware layers updates debugging tool did not reveal anything. I double checked with HierarchyViewer and I was satisfied that the ViewPager was behaving correctly (the contrary was unlikely anyway and would have been troublesome).

I then turned to another powerful, seldom used, tool called Tracer for OpenGL. My previous article explains how the tool works in more details. All you need to know is that this tool collects all the drawing commands sent by the UI toolkit to the GPU.

Android 4.3 and up: Tracer has unfortunately become a little more difficult to use since Android 4.3 when we introduced reordering and merging of drawing commands. It’s an amazingly useful optimization but it prevents Tracer from grouping drawing commands by view. You can restore the old behavior by disabling display lists optimization using the following command (before you start your application):

adb shell setprop debug.hwui.disable_draw_reorder true

Reading OpenGL traces: Commands shown in blue are GL operations that draw pixels on screen. All other commands are used to transfer data or set state and can easily be ignored. Every time you click on one of the blue commands, Tracer will update the Details tab and show you the content of the current render target right after the command you clicked is executed. You can thus reconstruct a frame by clicking on each blue command one after another. It’s pretty much how I analyze performance issues with Tracer. Seeing how a frame is rendered gives a lot of insight on what the application is doing.

While perusing the traces collected during a scroll in Falcon Pro I was surprised to see a series of SaveLayer/ComposeLayer blocks of commands (click the picture to enlarge):

Tracer for OpenGL

These blocks indicate the creation and composition of a temporary hardware layer. These temporary layers are created by the different variants of Canvas.saveLayer(). The UI toolkit uses Canvas.saveLayer() to draw Views with an alpha < 1 (see View.setAlpha()) when specific conditions are met:

Chet and I explained in several presentations why you should use alpha with care. Every time the UI toolkit has to use a temporary layer, drawing commands are sent to a different render target, and switching render target is an expensive operation for the GPU. GPUs using a tiling/deferred architecture (ImaginationTech’s SGX, Qualcomm’s Adreno, etc.) are particularly hurt by this behavior. Direct rendering architectures such as Nvidia’s fare better. Since the Moto X 2014 devices Joaquim and I were working with use a Qualcomm Adreno GPU, the use of multiple temporary hardware layers was most likely the root cause of our performance problem.

The big question thus become: what is creating all these temporary layers? Tracer gave us the answer. If you look at the screenshot of Tracer you can see that the only drawing command in the SaveLayer group of OpenGL operations renders what appears to be a circle in a small render target (the tool magnifies the result). Now let’s look at a screenshot of the application:

Falcon Pro 3

Do you see these little circles at the top? That’s a ViewPager indicator, used to show the user her position. Joaquim was using a third party library (I don’t remember which one) to draw these indicators. What’s interesting is how that library draws the indicator: the current page is indicated by a white circle, the other pages with what appears to be a gray circle. I say “what appears to be a gray” because the circles are actually translucent white circles. The library uses a View for each circle (which is in itself wasteful) and calls setAlpha() to change their color.

There are several solutions to fix this problem:

  • Use a customizable “inactive” color instead of setting an opacity on the View
  • Return false from hasOverlappingRendering() and the framework will set the proper alpha on the Paint for you
  • Return true from onSetAlpha() and set an alpha on the Paint used to draw the “gray” circles

The easiest solution is the second one but it is only available from API level 16. If you must support older versions of Android, use one of the other two solutions. I believe Joaquim simply ditched the third party library and used his own indicator.

I hope this article makes it clear that performance issues can arise from what appears to be innocent and harmless operations. So remember: don’t make assumptions, measure!

29 Responses to “Android Performance Case Study Follow-up”

  1. Julio Reyes says:

    Great Article!

  2. Andreas says:

    This is a fun and interesting post to read through, but I cant help to ask my self the question:
    Who could actually do this?

    I recon most android developers cannot even perform an OpenGL trace and of those who can even fewer will understand the result.

    For those of us that can do the trace and interpret the result very few would be able to deduct how this relates to the android graphics framework. I.e. The trace show what happens but not why it happens.

    This either proves that we are missing an important tool and/or that the graphics framework is not smart enough. I think there is a case for both but understand that writing a smart graphics framework is not easy.

  3. Romain Guy says:


    I recognize that and that’s why I always explain *how* things work in my talks and articles (and I did exactly the same in the book Filthy Rich Clients I wrote with Chet).

    The fact is that whenever you use a sufficiently complex system (library, framework, OS or even CPU), you need to understand how it works internally, even at a high level, to be able to understand its behaviors. Here’s a good example: you iterate over a large bitmap column by column, instead of row by row. You will not understand why it is slow unless you understand how memory caching work in a CPU.

    I realize it is difficult and there is no way one can know everything there is to know (although in this particular instance, the information is out there since I’ve talked about the case of saveLayer in several presentations). That’s why we’ve added more and more tools to Android over the years and that’s why I’m sure the Android team will keep adding more tools.

    In this particular case, the framework could do something smart. There are various solutions at the framework level but they all have their own drawbacks.

  4. Akop says:

    Romain, is there a reason why an onscreen soft keyboard interferes with the GPU profiler? Is there some way to work around this?

    I was working on a View that accepts input (not an EditText subclass), and I was unable to use the profiler tool when the keyboard was visible.

  5. Cal says:

    Great article, Romain. Just wanted to point out a possible error:

    “That’s a ViewPager indicator, used to show the user her position.”

  6. Trane says:

    Great article, I love your android performance posts!

  7. Rakhi Dhavale says:

    Nice article Romanian guy! There’s so much of depth involved in what your trying to explain, which most of the developers tend to ignore!

  8. Dmitry Zaytsev says:

    Enlightening, as usual

    But coming back to topic brought by Andreas – I’m working with Android for several years now and I always felt like I am missing something when working with graphics framework. As you said, when working with complex systems it is important to understand how things work underneath – in case with GPU and Android’s graphic framework which literature can you suggest to catch-up?

    Of course, I can just type “How GPU works?” in Google, but I believe that would be harder way than it could’ve been (one way to start with Java is by exploring JVM specification, but as a beginner in a field you won’t normally do that, won’t you?).

    And once again thanks for great articles – they really improved percieved speed of the apps I worked on.

  9. Greg Loesch says:

    I share concerns with Andreas and Dmitry. I’m not sure how long it would have taken me to arrive at the conclusion you arrived at. :P

    I would love to see some practical resources on the GPU, Android’s graphics framework, and even OpenGL with specifics of what is feasible on each sdk version. I’ve been doing Android dev work for 2-3 years, and this is definitely an area of weakness. Do you have a suggested resources?

  10. Ivan Davletshin says:

    Great article. Thank you. Just checked the application I am currently working on and found one view which takes significant amount of time to draw its frames.

  11. It’s really interesting! I don’t know which third party library he would have used but the following link of my article explains how to create the View Pager indicator in a really simple way (Not a big/standard coding :D ) which won’t affect the performance issue at any cost. There is a video to check the demo

  12. Thierry-Dimitri Roy says:

    I think it’s clear that the vast majority of the Android developper don’t understand the graphics pipeline. Maybe it’s because most of us comes from Java and we mostly work with web server and don’t usually work with OpenGL, as opposed to our iOS college that comes from a C++ background. Additionnally, the tooling that we currently have is very poor, compared to what we have in the java server world.

    But I do have a proposition. Instead of writing these blog post, why don’t you just record a video of youself going though this kind of bug? I would love to see which part you looked that worked, and the one that didn’t. It is my belief that most developper learn by watching another one work instead of watching a tutorial.

  13. Udi Cohen says:

    Great article!

    There’s another performance issue that I found with ViewPager:
    When you scroll between pages, the ViewPager creates a hardware layer for each page. In the case that you move views around the page when scrolling (to create cool transition effect), the HW layer will be updated for every frame when scrolling. This will make the scroll slower than it should.
    The solution for me was to disable the creation of that HW layer. I explained the process in my blog:

  14. amr says:

    i wonder if android team make a standalone layout design tool?

  15. This is one of the most joyful articles I’ve read recently. It contains really great and useful information. Thank you. I hope you write more frequently. :)

  16. Anxiaoyi says:

  17. Dmitry says:

    > Return false from hasOverlappingRendering() and the framework will set the proper alpha on the Paint for you

    Which Paint are you talking about here? A layer paint? So in this case the View should just draw what it usually draws, no alpha manipulation anywhere, right?
    And in the case of API < 16 it should instead use onSetAlpha to update some internal Paint which it uses to draw. Please correct me if I'm wrong.

  18. Commenting test says:

    I like your commenting system. It’s simple and clean.

  19. Martin says:

    Would this also apply when you use a png image with the transparancy set instead of the setAlpha() function? And does this also apply to using textviews that use a transparant text color?

  20. Romain Guy says:

    Martin: this does not apply to images with translucent pixels nor to TextViews that use a translucent color.

  21. Andy says:

    It seems pretty crazy that a series of tiny views are allowed to reek havoc on scroll performance for the whole screen, especially given that the dots aren’t actually changing during the scroll, given how easy this is to accidentally do, and how hard it is to debug. I assume you’ve considered caching a texture so that you don’t have to change render targets on each frame — what are the drawbacks to that that has made you decide not to do it?

  22. Romain Guy says:

    The problem comes from using Canvas.saveLayer(), a command that tells the graphics pipeline to change the render target. It wasn’t an issue with the software pipeline but it’s of course a problem with OpenGL. That command was thankfully rarely used and it was hard to justify the complexity of caching data for this arbitrary command in the stream. I believe that the M release simply forces a hardware layer on any View that uses setAlpha().

  23. Andy says:

    I would argue any problem with a tool the software pipeline uses that gets exposed like this is an issue with the software pipeline itself. But really glad to hear this is getting fixed in M! (btw, any eta on automatic overdraw elimination?)

  24. Romain Guy says:

    Automatic overdraw removal is in 4.4 and 5.0.

  25. Andy says:

    Is that as far as the team is planning on taking it? As long as developers are expected to manually inspect their view hierarchies for overdraw, there’s room for improvement. Or, conversely, devs will continue to put out android apps with subpar graphics perf. (Don’t misconstrue: obviously, there are benefits to reducing the extent of the view hierarchy outside of overdraw — those could be tackled automatically and separately though).

  26. Romain Guy says:

    Since I don’t work on the Android framework team anymore I cannot speak for their plans. The overdraw present in 4.4/5.0 does remove drawing instructions whose result would be covered by another opaque draw instruction. Further removing overdraw per pixel would be difficult in a lot of cases. I also don’t think it’s reasonable to expect a system to automatically optimize everything away (JITs, GCs, databases, filesystems, accelerated graphics APIs, compilers, etc. all do tons of optimizations on the developer’s behalf, but it remains possible to write code that leads to poor performance). Better tools and better implementations on Android’s side will always be needed but app developers should not ignore performance matters.

  27. Andy says:

    Is this the overdraw optimization you were referring to: or were there more? It seems like it’s good at removing full screen backgrounds. Do you remember if you all didn’t find it worth the CPU cost to more aggressively target overdraw, e.g. by discarding overdrawn rects at a more granular level while adding ops? I guess it’d probably depend on whether you think apps are more GPU bound or CPU bound?

  28. Romain Guy says:

    That’s the one. But overdraw removal only works when drawing opaque ops which are actually fairly rare. Patches could be split in opaque/translucent chunks but this would have other performance side effects.

  29. Andy says:

    Cool, thanks for the info! Since you’re not working on the team anymore, do you have a recommendation for someone else to follow and ask questions about future graphics perf plans?