By Scott M. Fulton, III, Net1News
The problem with the developers of any product releasing just a single public beta, without periodic updates, several months before its anticipated final release, is that it creates a fixed target for its competition. Microsoft spokespersons have indicated to me that scheduled updates for the Internet Explorer 9 public beta have not been planned, and even characterized intermediate updates for purposes other than vulnerability patching, to be unorthodox. (Last week, Microsoft issued another big security rollup for stable versions of IE, but not IE9 which didn't need the patch.)
So IE9 has a big target painted on itself, and it doesn't help that its scores are going nowhere during the beta period. Meanwhile, there's a new element of Firefox 4 that could make it one of the fastest, if not the fastest and most efficient, graphics rendering browser in the pack. I say "could" because, even in the latest betas, the superb performance scores posted in HTML 5 Canvas rendering tests, which are supposed to be accelerated using Microsoft's Direct2D library in Windows, are only intermittent. Scores flip between "good" and "incredible," and just when it appears to be time to declare Firefox 4 the latest phoenix to rise from the ashes, those Canvas scores flip back to just "good," even on a pristine and unencumbered test platform.
This past month, both Firefox 4 and Google's newly renumbered Chrome 8 have been trying to achieve the same efficiency gains, in the same way: Their focus is the HTML 5 Canvas element, which threatens Adobe Flash's leadership in rich Web graphics. Today, many developers rely on Flash to produce simple things such as pie charts and org charts; and when they want their Web site to do more than just sit there and show links, many choose to abandon HTML altogether and relocate their entire sites to the Flash platform. If Canvas were not only just as smooth as Flash but easy to learn, ubiquitous, and even better, free, one huge selling point for Flash could be wiped off the map.
When Mozilla first made public betas of Firefox 4 available last month, the organization announced it was experimenting with GPU acceleration - using graphics processors to process (. . . wait for it) graphics, specifically HTML 5 Canvas. For a platform that's essentially balanced atop a JavaScript interpreter, this is actually not such an easy thing. JavaScript is, by nature, single-threaded; and offloading processes to other available processors implies a level of parallelism with which JavaScript usually isn't very comfortable.
Since last week, I began seeing Firefox 4's acceleration capability off and on - quite literally. One moment, Canvas graphics would go zing with minimal CPU load, and the next moment they wouldn't. In the most recent browser performance tests conducted Monday morning by Ingenus LLC, it appears the later beta builds are less fickle than before about when the GPU co-opting benefits kick in. The browser still needs to be run in near-pristine conditions (for example, another heavyweight browser like Firefox 3 or IE8 cannot have been run previously during the same session). But once the test is run under the most fair conditions, Firefox 4's "next gear" of acceleration appears to kick in.
The latest private build of Beta 8 runs the 25, 100, and 500-sphere Man-in-Blue Canvas rendering test faster than any other browser in the field, stable or developing. And at 25 and 100 spheres - again, when the planets are all properly aligned and breathing is kept to a minimum - Firefox 4's drain on CPU resources is nearly imperceptible. Ingenus tests confirm no other CPU-delegated process has been picking up the slack, indicating that the GPU (which cannot be tracked using the CPU alone) is doing its job well.
Here's exactly what's going on: In the Man-in-Blue tests created by Web developer Cameron Adams, a given number of animated circles ("spheres") are plotted as though they've been dumped into the center of a floor and made to scatter, with no gravity or other complex physics to slow them down. Each test accounts for the average number of frames per second (FPS) that the interpreter makes available for plotting to the screen; the number of times the program allows the test runner to sample the FPS (the more spheres are on-screen; the fewer samples there can be) and the standard deviation among all samples, which is an indicator of stability. Ingenus modified the Man-in-Blue tests just slightly so that they'd run for exactly 15 seconds, and report their statistics at the end of each run. The tests were created to compare current HTML, Scalable Vector Graphics, HTML 5 Canvas, and Adobe Flash; Ingenus omits the Flash test since it's independent of browser performance.
The current stable Firefox 3.6.10 (not the beta) runs the 100-sphere Canvas test with an average FPS of 54.582, with 820 samples taken over the 15-second interval. Quad-core CPU utilization during that interval was 93.6% (where full quad-core utilization is registered as 400%, so about 23.4% of actual capacity). Private memory allocation during the test was about 40.4 MB.
By comparison, the current public Beta 6 of Firefox 4 allows 1480 samples over the same period (an indicator of better workload performance), with average FPS of 99.11. Yet CPU utilization dropped to 45.2% (about 11% of actual capacity). Weighing down on Beta 6's efficiency gains is a private memory allocation number of 91.1 MB. If memory were infinite, you could always improve speed by allocating a boundless field of RAM, so more than doubling memory allocation is not indicative of better efficiency.
It's with Beta 8 where Mozilla has found the "on" switch for efficiency, and has applied sufficient duct tape to keep it "on." On Saturday, Ingenus tests for 100 spheres on Canvas revealed the latest build pumping out an FPS of 99.344, with 1489 samples over 15 seconds. But CPU utilization drops to an almost imperceptible 1.17%, with private memory allocation dropping to 73.4 MB.
Click the chart above for a complete categorical breakdown
Those scores were good enough for a time to propel Firefox 4 above the 4.000 overall performance mark, well ahead of even IE9. Monday morning's overall scores - which include eight other test suites - dropped, even with the "on" switch kicking in, and now FF4 is behind IE9 by just a tick. In the 100 sphere test Monday morning, FF4 posted 1494 samples with a frame rate of 99.647. CPU utilization stayed fairly low at 1.853%, and private memory allocation was 72.2 MB. But lower text rendering and SlickSpeed scores dragged FF4's overall score down to 3.666, behind IE9's 3.692.
Here's an important note, which will play a big role a bit later: The frames per second number that each browser cranks out are not necessarily shown on-screen that fast. Imagine an animation studio operating in real-time where the animator is faster than the photographer, and the projector can only show a certain maximum FPS anyway. The animator can draw as many sheets as he's able to, but the photographer will only capture a picture of the sheet that's on top of the pile when he gets around to that pile. That's an analogy for how FPS tests work in Web browsers, and that's probably true for any such test you'll come across.
Next: Google tries the same play, but with different results...
This article originally appeared in Net1News.
©Copyright 2010 Ingenus, LLC.
©Copyright 2010 BetaNews, Inc.
On the other side of the pond, Google has been ramping up Chrome to enable GPU acceleration as well. Late last week, Google premiered its first acceleration code with build number 552.0 - which the company has decided to dub, for the first time, Chrome 8. (The distinctions between Chrome version numbers aren't always as discrete as everyone else's; Chrome 8 is not at all an overhaul of Chrome 7, which itself has yet to be released in a stable edition.)
Google operates two development channels, dubbed "beta" for more general public participation, and "dev" for more experimental code. Chrome has often been the fastest brand of browser in Ingenus' tests, but it achieves many of its speed gains by distributing its tasks over two processes. (I've seen it fork a third process but not use it yet.) As a result, CPU and memory utilization are often quite high, just like for any sports car that achieves greater horsepower by lowering its gas mileage.
The latest beta build of Chrome 7 cranks out 84 FPS in the 100-sphere Canvas test, with 1260 samples in the 15-second interval. Because Chrome uses two processes when the going gets tough, quad-core CPU utilization ramps all the way up to 152.4% on Performance Monitor, or about 38% of real capacity. But compared to Firefox, memory utilization for Chrome is actually quite low, with 36.4 MB distributed over both processes.
With the dev build of Chrome 8, Google begins its own acceleration experiments. The early results. . . are mixed. For now, frames per second actually slows down versus Chrome 7 beta, to 56 FPS with 839 samples. CPU utilization for process #1 definitely declines to an imperceptible 1.366%, showing GPU offloading is clearly working. But that's just for process #1. Process #2 drops utilization only to 57.2%, leaving Chrome overall with a noticeable footprint, and memory consumption that's still slightly higher than for Chrome 7 beta (41.6 MB).
This is why Chrome's efficiency score on this test increases only marginally, from 0.803 (relative to Firefox 3.0.19 in the same test) to 1.083; whereas Firefox 4's efficiency catapults from 0.977 for Beta 6 to 20.666 for Beta 8. Keep in mind, that number is but one factor in a wide-ranging test suite of nine different batteries, accounting for computational speed, rendering speed for text and graphics, "scalability" (the ability to improve with higher workloads), and resource efficiency.
It was IE9 that increased developers' focus on GPU acceleration in the first place; and until this week, IE9 had the highest efficiency scores on the 100-sphere test. IE9 cranks out 59 FPS with 885 samples, with average quad-core CPU utilization of 24.6%, and memory utilization of 36.9 MB, for a relative score of 1.957. (IE also splits its workload over two processes, but tends not to use process #2 unless in extreme conditions.)
For now, Firefox's GPU offloading efficiency gains only show up during the 25- and 100-sphere tests; at 500 and 2500, the latest Beta 8 build appears to revert to standard CPU utilization above the 90% mark, comparable to Beta 6. There appears to be a kind of virtual rocker switch somewhere, at least in the current beta builds, limiting offloaded workloads to lighter amounts only. Theoretically, if the GPU offloading code is stabilized and the same performance gains end up applying proportionally to the 500 and 2500 tests, Firefox 4's overall score could surge forward all the way to the 5.0 mark, potentially eclipsing Opera and Chrome.
There's a reason that Firefox 4's score now threatens that of Apple Safari 5 as well, and it has to do with a scoring change decision. Both the stable Safari and the test build with the WebKit development snapshot, are capable of cranking out huge FPS numbers - for instance, 239 FPS in the 100-sphere Canvas test. At least, that's what the statistics read. But Safari gets away with this by turning off screen refreshes during the high-processing period, often during the entire 15-second interval. Often the final screen with the FPS and deviation count, is the first moment you'll actually see the spheres.
Now, being able to calculate that fast is awfully nice, but if the task is graphics and you're not seeing the graphics, it all means nothing. Imagine if this task were instead a complex game being rendered on the Canvas element; how could the player be expected to win without a touch of clairvoyance? So I made a decision: I would treat each test run where Safari (or any other browser) turned off the screen as a fail, just like a teacher who realized his student turned in empty pages for homework. That's why I've scored the latest stable Safari 5.0.2 down overall to 3.520, and 3.666 for the latest Safari with WebKit dev build 69699.
Opera's snapshot build remains the overall performance leader, but not by much. Chrome 8's marginal efficiency gains, coupled with very impressive computational speed gains, have narrowed the gap to just over a hundredth of a point: 4.992 for Chrome 10.70 build 9071, versus 4.980 for Chrome 8.0.552.0. The overall computational speed leader in Ingenus' tests is now Chrome 8, with 9.760 versus 7.965 for Opera 10.70 (the future Opera 11), 7.870 for Firefox 4 Beta 8, and 7.645 for IE9. Chrome 8 is currently the undisputed champion of the JSBenchmark battery, and staying in the top three for computational speed elsewhere in the suite.
Here's another important note: With multi-test batteries like SunSpider and SlickSpeed, Ingenus compares the scores in each individual test against those of the index browser (Firefox 3.0.19), and then computes the average of those scores (or, in the case of SlickSpeed, the geometric mean since each test in that battery is exactly the same job, just using different libraries). Most testing focuses on the final time elapsed, especially for SunSpider. But there are individual elements of SunSpider where IE9 blows away the field - for example, the CORDIC algorithm test, where IE9 scores 1 ms versus 8 ms for Opera 10.70 and 10 ms for Chrome 8. If we were examining cumulative final speed only, that 1 ms posting would get completely overlooked. That's one big reason why IE9's SunSpider score remains so high (14.268 versus 12.118 for Opera 10.70 and 11.177 for Chrome 8).
Where Opera 10.70 excels is in the rendering department, with a 5.223 score in that department, and its nearest competitor - the Chrome 7 beta build - only at 4.314.
Efficiency has been the great equalizer in Ingenus' latest test suite; it's where the current stable Internet Explorer 8 sets the high watermark at 3.899 (compared against the old, low-efficiency Firefox 3.0.19), with everyone else's scores tending to go down from there. Today, the browser with the most comparable efficiency score to IE8 is the latest stable release of Opera 10.63, with a 3.844. The last build of Opera 10.70 before the numbers change, follows up in the efficiency department at 3.670. And then in third, thanks to its successful Canvas experiments (when they are successful), is Firefox 4 Beta 8 with a 3.146. That's a huge jump over Beta 6's efficiency score of 1.728.
This article originally appeared in Net1News.
©Copyright 2010 Ingenus, LLC.
©Copyright 2010 BetaNews, Inc.
Copyright Betanews, Inc. 2010
