By Scott M. Fulton, III, Net1News
The latest wave of upcoming changes to the world's two most used Web browsers, jointly responsible for easily three-fourths of the Internet's HTTP requests, has nearly everyone in the business rethinking the meaning of "quality" as it pertains to browser architecture. While their arguments start with the usual reminders that folks just want to see their pages load faster, before too long, they wander into dissertations about the methods architects use to achieve the appearance of loading faster. . . especially when they actually don't.
In preparing to test Microsoft's first Internet Explorer 9 public beta, released last week, and Mozilla's public Firefox 4 beta, released late last month, the advice I received most often fell into two departments: 1) Pay more attention to graphics rendering, since new browsers will be spending more time processing Web apps than just displaying pages; 2) start paying attention to how browsers utilize memory and CPU cycles. Since my smarter readers are typically right, that's what I've done in crafting my all-new browser performance test suite.
In the first round of total scores for the latest beta builds, the first IE9 beta has squeaked by the latest FF4 beta by about 8%. These scores take into account four major departments: computational ability (30%), rendering ability (30%), scalability (losing less processing ability over time as processing jobs scale up, 20%), and now, hardware utilization efficiency (20%). In past tests, as readers will recall, stable Firefox builds have typically scored around nine times the current IE8 score.
With last week's release, Microsoft has made up all of that ground, and then some. If you consider the general performance of the old Firefox version 3.0.19 (stable) as scoring a 1.000 in all categories (my new index browser), then by comparison, IE9 scores a 3.652 - better than three-and-a-half times the performance of what Firefox users were typically experiencing as recently as 2008. Firefox 4 Beta 5 posts a score of 3.377 in the same suite of tests.
What the numbers tell us
The computational engine in IE9, part of the new JavaScript engine dubbed "Chakra" (a Hindu term referring to the seven centers of human energy), is tremendously more capable than its predecessor. In computational scores alone, where IE8 (overall score: 1.145) scored an abysmal 0.502 (about half as powerful as the old Firefox 3.0), the first IE9 beta scored an impressive 7.645 - an improvement of more than 1500%. IE9 beta not only scores better than FF4 beta on the well-respected SunSpider benchmark (relative score: 14.268 versus 13.827, average time elapsed 417.8 ms versus 517.8 ms), but better than the latest stable version of Google Chrome 6 (relative score for build 472.62: 10.672). Although Chrome's cumulative time was better than IE9's (317.2 versus 417.8), my tests compare the relative times of each heat and average the results. Thus the fact that IE9 computed CORDIC variations (an alternate method for computing arctangents) in 1 ms versus Chrome's 10.4 ms and FF4's 29.6 ms, is reflected in this scoring system rather than washed out in the sum.
IE9 was not computationally faster across the board. It only scored a 2.077 on the JSBenchmark battery, which is even below the score for the latest stable version of Firefox (2.161 for version 3.6.10). A recent blog post by IE9 lead program manager Jason Weber, written in preparation for the IE9 beta's release, tries to spin testers' opinions away from certain brands of publicly available tests, describing and even diagraming them as incomplete. But Weber's description matched that of the original Celtic Kane benchmark, not the present test which is a complete replacement, and by the author's own admission, far more applicable to real-world workloads.
The JSBenchmark score, coupled with the blog post, gave credence to some Firefox users and other naysayers who argued that Microsoft has merely been advancing IE's scoring ability on the SunSpider test, and neglecting other departments. Is this true? To find out the answer for myself, I used a series of benchmarks that I wrote myself, for which neither Microsoft nor Mozilla have the source code. They're blistering problem-solving tests that are executed in varying workload sizes, to test for both performance and scalability - the ability to kick in more processing power when it's needed.
The news for the naysayers. . . is very bad indeed. IE9 absolutely flattened FF4 on my own DFScale battery, with a speed score of 6.590 versus 3.539. Maybe IE9 isn't the best performer in every test, but it's not because Microsoft is too focused on test numbers.
Next: Better performance, but at what cost?...
This article originally appeared in Net1News.
©Copyright 2010 Ingenus, LLC.
©Copyright 2010 BetaNews, Inc.
But at what cost?
As is the case with any other kind of machine ever built throughout history, when you build it to perform better, there's a cost in terms of efficiency. The Chevy Chevette may have been the single ugliest product to emerge from Detroit (way uglier than the Edsel), but it did have that fuel economy thing going for it.
So one of the most important questions we can ask when testing Web browsers, is this: In making Internet Explorer 9 a far better performer than its predecessors, did its architects sacrifice "fuel economy?" Or more specifically, does IE9 chew up more memory and processor cycles?
Say what you will about IE's performance through the years, but one advantage of working next door to the Windows architects that its own architects can use to their advantage, is building a more efficient browser. Even Firefox's most fervent supporters have historically conceded that it's a memory hog. Indeed, Firefox 3.0 (my new index browser) was a virtual black hole into which memory appeared to fall with no chance of escape.
If we score that black hole a 1.000 for CPU and memory efficiency, then by comparison, IE8 - which in several respects is more like a Chevette than a Corvette - scores a 2.845, judging from the SunSpider battery, SlickSpeed CSS selector processing test battery, and DFScale battery combined. Against most other browsers, including the stable versions of Firefox, that's actually very high indeed. This is the one scoring department where we can expect the latest, high-performance betas to fall, not rise.
Overall, IE9 beta does pay a price for its computational superiority, scoring a 2.014 in the efficiency department. But that average doesn't tell the complete story: The current stable IE8 sips memory cycles at low workloads. My latest addition to the complete test suite, adapted from The Man in Blue's animation benchmarks, coupled with processor cycle and memory consumption monitoring using Microsoft's Windows 7 Performance Monitor, shows that in low-workload animation processing, IE8 can consume less than 0.2% of available processor cycles, while Firefox 3.6.10 chomps away at 21%. The IE9 beta does consume more CPU cycles at low workloads than its predecessor, maxing out at about 4%, but that's still impressively low.
As workload increases, however, IE8 opens its throttle and starts chomping down gas like it was water. The Firefox 4 beta posts some very impressive numbers in middleweight workloads, with efficiency scores on individual Man-in-Blue heats approaching 14 (reflecting, for example, about 0.7% overall CPU cycle consumption when plotting 100 simultaneously moving objects on the HTML 5 Canvas element). But at high workloads, where IE8 can actually crash (I had to repeat some tests several times until it didn't crash) and Firefox 4 falls through the floor, the IE9 beta moves steadily along, posting numbers that are still above that 1.000 index mark.
Google Chrome's recent claim to fame has been its slick graphics and high frames-per-second numbers. My latest test confirms that claim, but it also shows the price Google pays: In the "beauty" department, to borrow Microsoft's new term for it, IE9 beta scored a 3.122 in performance and 1.832 in processing efficiency. Google Chrome 6, meanwhile, scored only 2.892 in performance and 1.461 in efficiency. So think of IE9 as a more fuel-efficient performance engine. If you do the math, this suggests that IE9 makes about 26% better use of its CPU cycles and allocated memory for overall graphics production (which, for the first time, includes SVG) than does Chrome 6.
However, Firefox 4 Beta 5 did beat the IE9 beta in the efficiency department, scoring a 2.295 overall. Memory and CPU utilization efficiency have never been Firefox's strong suits, so FF4's high scores here are very encouraging. Particularly, FF4 is learning to scale down its CPU cycle consumption with lower workloads; for example, reducing its low-workload consumption averages from 9.1% to under 1%. Firefox 4 could very well be the best behaving Firefox that Mozilla has produced to date. The problem is, it'll have more company.
The complete test suite results, including an extensive analysis of every browser in the field including Opera and Safari, is the subject of the first Ingenus Report, available soon.
This article originally appeared in Net1News.
©Copyright 2010 Ingenus, LLC.
©Copyright 2010 BetaNews, Inc.
Copyright Betanews, Inc. 2010
SunSpider -
Google Chrome -
Internet Explorer -
JavaScript engine -
Firefox 4 beta
