Scalability cont.

Thanks for all the comments on my last post! Due to popular demand I was also running the tests on Opera. During the first run Opera crashed after opening 140 tabs. The crash ID was “crash20110803153559″, whatever that means.
The following runs were more successful but I noticed a slowdown during rendering after about 120 tabs.
The timing results for Opera 11.50 are:

real    6m55.074s
user   5m23.217s
sys     1m13.607s

The results are similar to Firefox. Opera uses a single process with 290 threads and 2.5GB of RAM. For comparison: Firefox uses 27 threads and 2.02GB of RAM.

I was also running the V8 benchmarks on Opera with 150 other tabs:
Score: 2719
Richards: 2665
DeltaBlue: 2027
Crypto: 3213
RayTrace: 4440
EarleyBoyer: 3135
RegExp: 1008
Splay: 4515

Opera with a single tab:
Score: 4028
Richards: 3901
DeltaBlue: 2999
Crypto: 4625
RayTrace: 5596
EarleyBoyer: 4744
RegExp: 1543
Splay: 7766

The JavaScript performance is pretty good. Firefox got 3954 for 150 open tabs and 5125 with a single tab. The browser is still responsive but closing 150 tabs at once takes about 30 seconds. I can’t provide any insights to the Opera event loop. Maybe some developers can help me here?

This entry was posted in Firefox GC. Bookmark the permalink.

21 Responses to Scalability cont.

  1. WhateverThisMeans says:

    The crashid is completely uninteresting btw but I guess you knew that already and the “whatever this means” was a rethorical question anyways.
    “crash 2011-08-03 15:35:59″

  2. Alexander says:

    Why you count threads for Opera with flash and not add threads from plugins contatiner for Firefox? Opera with disabled flash use mach less threads.

  3. jgraham says:

    Have you thought about how to make this work without depending on live sites? Obviously having a realistic workload is nice, but live sites vary in realtime (e.g. loading flash adverts sometimes and simple text or images others) and vary substantially over longer time-periods. This means that the data will be a bit noisy for cotemporal runs — although I expect not enough to invalidate your findings — but useless historically; one can’t usefully compare a run from today with a run from two months ago.

    • Gregor says:

      I am not claiming that I have invented a new standard benchmark and I am not interested in making a static version that has every page in a cache. Gregor Richards from purdue is working on such a system.
      I am more interested in the changes over time. So if the memory footprint of the browser with 150 tabs increases, we have to make sure that it’s not a regression within the browser.
      I think it’s also very interesting to see how real workload changes over time and how much JavaScript do we see now and in one year for example.

  4. Sadly you fail to mention the OS. On Linux Opera is rather sluggish and if those numbers are for Linux then i suspect you’d get much different numbers on Windows.

    • Looking at your time output you actually are on linux. If you would like to try this on windows, you can get time by installing Cygwin and calling it directly (`c:\cygwin\bin\time`), and all the process stats with Process Explorer: http://technet.microsoft.com/en-us/sysinternals/bb896653

      Running it there would also mean that you actually get tests relevant for the majority of your userbase.

      • Andrei says:

        In the previous post, which is continued by this one, the author says that those test were running on Mac OS

    • Gregor says:

      This is on OS X 10.6. You are right that it might look very different on Windows but I don’t have a PC for testing.

      • If you direct me to a download of the Firefox version you used to test this i can run the two on my machine under equal conditions. :)

        • Gregor says:

          Thanks! It’s a mozilla-central nightly build from 7/27. You can download it from our nightly build server.

          • Alright, i’m currently running the tests, but i have an issue with Firefox, over 30 tabs the popup blocker kicks in even when it’s explicitly disabled in the preferences, which stops the script from opening more tabs. Any suggestions?

          • Gregor says:

            you have to go to about:config and adjust dom.popup_maximum

          • Thanks, that helped. Sadly Opera disqualified itself though. :(

            Around when it tries to load “howstuffworks” it starts to thrash the disk a bit and then loads further pages only as blank pages. This means any comparison is meaningless.

            I’ve reported the issue to Opera and they may do something about it sometime.

          • Gregor says:

            I won’t delete the scripts so we can compare when they fixed the issue.
            Any comments on the Firefox results? Was the browser still responsive?
            Thanks for testing and your time!

          • Thanks for keeping it around. I hope we get a chance for comparison.

            Some more information: Firefox was … working. I wouldn’t call it responsive. Every single aspect of it was laggy, scrolling through the tab bar, switching tabs, scrolling pages, opening context menus, you name it. The V8 test gave it 1709 points. This does not sound good, but on the upside, it did load all content, which is admirable in light of the following:

            Opera is, for a while after loading the tabs, very responsive and speedy. I tried to wait out the 90 seconds of the test script and then open the V8 test, but then the UI began to break apart. The tab bar flickered between showing all tabs, and only 8 of them at full-width, i couldn’t switch tabs anymore and closing was impossible too. So a test was impossible.

            However, to counterbalance that: This is my current Opera:

            http://i.imgur.com/9uFUr.png

            That’s 170 tabs, and it loaded those not one after the other, but when i started it up, simultaneously. And all of them are loaded and working fine. The difference: My work Opera is set to wrap flashes in containers that skip loading until i trigger them and contains a number of URL blocks similar to adblock plus.

            When i ran the V8 test with it, i got 3081 points, which compares to 3200 points with one tab open in the newest dev release.

            If you have any suggestion as to how i could import an opera session file into firefox i could try loading APB in there, switching off flash, loading all those pages and then running V8 to see how it compares. :)

            In summary: Firefox seems to take the path of a reference rendering, taking pains to be correct at all times and pays a bit for it in smoothness (not the same as speed) under extreme conditions. Meanwhile Opera focuses on smoothness at all times, paying for it by sacrificing stability under extreme conditions (though that seems to mostly become an issue when flash is involved).

  5. Dillon says:

    You should update the test for the latest Opera public build since you’re comparing it against a Firefox nightly build (and Chrome Canary). There have been some memory usage improvements since 11.50 and the performance on this test might have improved.

  6. Caspy7 says:

    You rock Gregor.

  7. MuadDib says:

    Gregor, there seems to be a difference in memory usage between Opera and Mozilla Firefox.
    Opera feels much snappier and faster, but tries tries to keep everything in memory and has a rather bad spatial locality, esp. with JavaScript switched on, it feels like it is constantly touching all of its pages.

    On the other hand, Mozilla Firefox feels slower and a bit sluggish in comparison with Opera, however, it is apparently capable of releasing the contents of its least recently used tabs, so its memory usage remains still reasonable and the browser usable, when Opera’s memory usage causes trashing and becomes unresponsive.

    Just think of loading a Mozilla Firefox session with, say, 800 tabs or so… the browser will still be usable, you can open another tab and do not need to care about the rest (most of which have not even been loaded; their loading will be on-demand, thus delayed until the actual need).

    Now, try this in Opera… if it has not crashed, you will find it struggling hard and trashing the whole system even after more than an hour, being barely responsive…

    BTW, just for the record, I am an avid Opera user, for it is the most socialist browser (http://www.evilsoft.org/2009/08/25/libertarians-are-just-crazy-for-firefox-and-other-observations) and I find it still the best for porn-browsing…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>