Performance and conformance testing smackdown.
Round 2: Testing Firefox 2 against Internet Explorer 7.
Archive of Network World tests
Subscribe to the Network Product Test Results newsletter
In our testing of markup conformance, we found that no browser supports perfectly even older markup specifications, such as HTML 4. If you poke around the HTML and XHTML specifications hard enough, you will find many little and occasionally some big problems in the browsers.
The markup omissions were more notable with IE. For example, IE 7 does not support a number of character entities, including useful math symbols. It does not support the tag, nor does it support all aspects of how the
When it comes to markup conformance, neither browser offers perfect support, but many developers will deem both passable in their HTML support. Yet upon deep inspection, we found Firefox 2 to be more complete in addressing specification details.
The case of Cascading Style Sheets (CSS) support is similar. At a basic level, you may not see much of a difference between the browsers. In basic tests we noticed mostly subtle visual differences, such as the interpretation of keywords for border types, the fantasy font family or font-sizing keywords. However, all these differences are within the realm of being correct interpretations of the CSS specification.
| |||||||||||||||||||||||||||||||||||||
When looking at more complex CSS examples, we saw almost three dozen significant failures or lack of support in IE 7 and nearly the same number of cosmetic differences that some might construe as failures. Firefox 2 completely failed only nine times in this round of testing; in four of those, all browsers failed. Our control browser Opera 9 was notable in that it failed only five tests, and it uniquely supported a few interesting features, such as CSS2 list counters.
Note that IE 7 is substantially better than IE 6 in its CSS support. Numerous bugs have been fixed and many features added, including new selectors and important properties integral to page layout, such as fixed positioning and maximum and minimum heights and widths.
In the case of JavaScript conformance, Firefox 2 is the browser to beat. Admittedly, what exactly constitutes JavaScript conformance in a browser is a bit hard to pin down, because JavaScript to a developer is a collection of three things: the core languages defined by the ECMAScript specification; various commonly supported objects, such as Window and Navigator, that are not covered by any set standard; and the various objects related to document manipulation covered under the W3C Document Object Model (DOM) specifications.
With that definition in mind, both browsers do support core ECMAScript 3. However, in terms of core language, Firefox is further along in its support of emerging JavaScript versions, with its support now at 1.7. IE 7 reports to be at Version 1.3 and makes no known changes to language-feature support outside of making XMLHTTPRequest object used in AJAX a native feature rather than one implemented in ActiveX.
Firefox reigns supreme in DOM standards support. We found 96 problems in IE 7 for DOM Level 1 and 2 Core tests combined, compared with 31 problems in Firefox 2. Looking at the DOM Level 2 HTML bindings, we found 74 problems in IE 7 compared with only nine in Firefox 2.
When talking about JavaScript performance, the test results are unclear. Our initial test showed JavaScript performance to be slower in IE 7, but further testing showed that depending on the types of test case we used, we got very different answers. For example, running numerous varied mathematical functions as benchmarks, Firefox 2 was 75% faster than IE 7. Yet benchmarking with string manipulation functions, we found IE 7.0 nearly three times faster than Firefox. When testing array access and manipulation, we see IE 7 nearly 10 times faster than Firefox 7, but doing a simple benchmark with document.write, we always found Firefox 2 faster.
Varying test types did little to clarify script performance. For example, writing benchmarks consisting of DOM functions that would be heavily used in modern Web applications, we saw widely divergent results. For example, IE 7 was about 20% faster than Firefox 2 for creating tables, yet when accessing all the rows of this table, IE was 10 times slower. Testing page manipulation using the nonstandard innerHTML construct, Firefox 2 was about 20% faster than IE 7. Further tests continued to show varying winners depending on the features exercised, with the control browsers just as erratic as the tested browsers.
Declaring a scripting-performance winner is not reasonable from our observations. It was quite interesting that in a number of cases older browser versions were faster. When it comes to script benchmarking, performance may vary dramatically.
Because it is closely related to script performance, we spent quite a bit of time exploring the memory use of the two browsers when running various tests. It has been noted by many developers that IE 6 has significant memory-leak issues related to JavaScript. We verified that IE 7 fixes the most serious problems, such as cleaning up the circular object references. In IE 6 this problem caused significant memory leaks and poor performance.
To illustrate this, we made a test script that created 10,000 objects through the DOM with circular references. In IE 6, in a mere 10 test runs, memory use increased from 30MB to 115MB, and the execution time went from 2.3 seconds to 13.8 seconds. IE 7 showed dramatic improvement, with execution time staying around 500 millisec and memory use hovering around 28MB. We did note some minor memory leakages in JavaScript tests in both browsers, but nothing as worrisome as what IE 6 exhibits.
To deduce page-rendering speeds, we found that defendable numbers are hard to come by. At first, we used simple examples of large tables, randomly generated HTML structures in a valid form and other baseline benchmarks. The results were highly variable. In a test with random text in various-sized paragraphs, we saw IE 7 to be nearly 50% slower than Firefox 2. However, when doing a similar test with large tables, Firefox was much slower.
When looking at real-world pages, we saw that rendering the results were almost just as variable. Performance rankings changed depending on the site tested, though in all cases either IE 6 or Opera 9 was the top displayer. Very notable is that almost always IE 7 and Firefox 2 were slower than their previous versions.
In our sample of real-world pages, IE 7 did slightly edge Firefox 2 more times than not. However, the differences were very thin regardless of which came out on top, and some sites that did not hold to that trend. For example the MSN homepage sample rendered in 1200 millisec on average in IE 7, but only around 1110 millisec in Firefox 2. The one trend that was fairly constant was that when Opera rendered faster, it rendered noticeably faster, as in this case, where it came in at
781 millisec.
In our standards-conformance analysis between the two new browsers, Firefox 2 is the clear winner. But we cannot say the same for its performance. Internet Explorer trends faster than Firefox in our testing. However, we are not convinced we can significantly defend any page-rendering performance test between the two newer browsers, because they are so close in our measurements, and there is plenty of variability both in baseline and real-world tests.
Powell is the founder of PINT, a San Diego Web development and consulting firm. He can be reached at tpowell@pint.com.
Powell is also a member of the Network World Lab Alliance, a cooperative of the premier reviewers in the network industry, each bringing to bear years of practical experience on every review. For more Lab Alliance information, including what it takes to become a member, go to www.networkworld.com/alliance.
