Apple vs Adobe: Vetting Steve Jobs Flash Assertions

Apple and CEO Steve Jobs have at various time in the past 6 months complained about Adobe Flash performance on both OS4 iDevices [iPhone and iPad] and Mac OS X. This will post will vet those remarks with publicly available benchmarks and other reports. They will be confined to Mac OS X since results of Flash on iOS3 and 4 are rather hard to obtain. And in fact, it is hard to get good performance data across the board [speed benchmarks, bug/reliability reports, security assessments, battery drain results] as  many software and hardware licenses prohibit users from doing benchmarks or simply reporting straight forward performance observations.

Nonetheless the checks on  Steve Jobs assertions use comparative data wherever available. All of the sources cited are provided with links to the primary data and readers are encouraged to check up on the complete data to verify conclusions for themselves. The preferred benchmarks would use the same hardware and just change the OS on that hardware to conduct the tests. This is not trivial. Likewise lining up identical software versions on both OS can have one pulling at hairs. But surprisingly there have been a number of   apples to apples test results which are cited here.

Speed/Performance of Flash

Keep an Open Eye has already cited a study that suggests that there is a pretty wide gap between Windows and Mac basic graphics performance with  Windows beating Mac by hefty margins.  So Keep an Open Eye sought to confirm these results and did find additional benchmarks which appear to confirm Windows superior graphics performance vis a vis Macs on identically the same hardware.  All of these results show that Flash [and other graphics software as well] runs faster on Windows using identically the same hardware  Other sources corraborate this. First consider the result from ArsTechnica from a year and a half ago:

ArsTechnica Oct 2008
Mac OS X 10.5	Windows Vista
Flash 9 GUIMark-9.5fps, 104%cpu Flash 10 GUIMark-28fps, 140%cpu Flash 9Hulu-63%cpu Flash 10 Hulu-56%cpu Flash 9YouTube-45%cpu Flash 10 You Tube -40%cpu	Flash 9 GUIMark-45FPS, 54%cpu Flash 10 GUIMark-46fps, 54%cpu Flash 9 Hulu-9%cpu Flash 10 Hulu-7%cpu Flash 9YouTube-8%cpu Flash 10 You Tube6%cpu
Same Quad 2.66 MacPro 6GB of RAM

Now for the lastime, please note that these tests are conducted on the same Apple hardware, the only difference is the operating system being used. What is interesting about these results is that again the benchmarking has been done on Apple Hardware using Mac OS X and Windows Vista and therefore an opportunity for Mac to shine given Vista’s poor performance benchmarks.But the benchmarks indicate the opposite, Windows Vista outperforms Mac OS X in every case. Faster frame rate rate in Flash 9 and 10 by at least 100% and lower CPU usage by half again in the GUIMark tests and CPU usage is a factor of 7 better than Mac OS X for Hulu and YouTube tests. Also it is significant that the Flash 9 to 10 versions show a large improvement in the frame rate from 9.5fps to 28fps and at least 10% less cpu required with the notable exception of the GUIMark where cpu usage went up by 40%.
These results are confirmed by:
CNET – Windows 7 beats OSX Snow Leopard by 7% in Cinebench  and 24% in gaming tests

In sum one can conclude that Adobe has been acting in good faith making notable improvements on Mac OS for at least 3 years as indicated  in the change from Flash 9 to Flash 10 [and just shortly below more on 10.1 results]. Also one has to concede that even Windows Vista seems to easily outperform Mac OS X on these graphics tests. But the complaint about Flash consuming a lot of CPU power on Mac Graphics appears justified especially in the case of the GUIMarks.

Finally it would have been very helpful to have Quicktime performance as a point of comparison as to how Flash is improving versus Apple’s own  video-primarily product. But here is a  recent set of benchmarks by AnandTech on the new 10.1 beta release published Nov 19th 2009. This benchmark shows the improvements when Flash taps directly into the GPU to accelerate operations :

Windowed Average CPU Utilization	Flash 10.0.32.18	Flash 10.1.51.45
Hulu Desktop – The Office – Murder	70%	30%
Hulu HD 720p – Legend of the Seeker Ep1	75%	52%
Hulu 480p – The Office – Murder	40%	23%
Hulu 360p – The Office – Murder	20%	16%
YouTube HD 720p – Prince of Persia Trailer	60%	12%
YouTube – Prince of Persia Trailer	14%	7%

The improvements shown, which are pretty dramatic but  are for Windows only in these results. AnandTech explains why:

I took the same Office clip I’d been using for all of the other tests and ran it on my Mac Pro at full screen (2560 x 1600). Using Activity Monitor I looked at the CPU utilization of the Flash Player plug-in. I compared both versions of Flash and saw a significant drop in CPU utilization:

Hulu Full Screen (2560 x 1600) Average CPU Utilization	Flash 10.0.32.18	Flash 10.1.51.45
Hulu 480p – The Office – Murder	450%	190%

Going from roughly 450% down to 190% (or a bit over 10% of total CPU utilization across 16 threads) made full-screen Hulu playable on my machine. In the past I always had to run it in a smaller window, but thanks to Flash 10.1 I don’t have to any longer.

With actual GPU-accelerated H.264 decoding I’m guessing those CPU utilization numbers could drop to a remotely reasonable value. But it’s up to Apple to expose the appropriate hooks to allow Adobe to (eventually) enable that functionality.

The last statement which I have underlined seems to lend credence to the arguments being made by Adobe that Apple has not been forthcoming in providing the GPU Accelerator APIs necessary for the Flash Player to take advantage fully of a Mac’s CPU+GPU combo[finally fully released in May 2010]. Just as striking is the continuing superiority of Windows over Mac in handling graphics – all of the benchmarks here confirm the Phoronix results – Windows appears to dominate Macs in Graphics performance on test using identically the same hardware. See here just one of several test results where both Windows and Linux dominate MacOS/X using identically the same Apple hardware and the same gaming software. Since these tests are using graphics software using no Adobe elements they confirm that it is Apple’s OS software or graphic drivers that are at fault in Apple’s poor graphics performance:

Here is another test from CNET using OpenGL Cinebench software tests which confirms the Phoronix results:

It appears that Apple is adopting a redirection strategy – “the best defense is to blame others for your own systems shortcomings .”

Flash’s Effect on Battery Life

A second point that Apple has made is that Flash drains the battery life of various Mac systems. The ArsTechnica performance benchmarks show that CPU utilization is high for Flash on Mac, much more so than on Windows. So one would suspect that battery drain using Flash would be greater on Macs OS X than on Windows. Fortunately, ZDNet did a series of benchmarks that measure battery life explicitly:

ZDNet – June 2010
Mac OS X 10.6.3	Windows 7
Flash 36% avg cpu 43% batlrm Flash GPU 31%avg cpu 39% batlrm Qtime 8% avg cpu 59% batlrm Wmv 13% avg cpu 62% batlrm	Flash 18% avg cpu, 62% batlrm Qtime 24% avg cpu, 57% batlrm Wmv 14% avg cpu, 55% batlrm
batlrm = battery life remaining Windows1 Thinkpad Core2Duo 2.4Ghz 4GB Ram, Intel x4500HD, Win 7 64bit Apple Macbook Pro, Core 2 duo 2.53 GHz 4GB Ram Nvidia MacOSX 10.6.3

The results of these tests confirm the notion that Flash does consume more battery life on a Mac. Note each test started with the battery reading 100%; so after playing the same complete movie the results were read as batlrm=battery life remaining. No other programs were running including turning off TSR-Terminate and Stay Residents. Note that Flash at 43% battery life remaining is 16% behind Apple’s Quicktime and 19% behind Microsofts WMV. These results are large but not ovewhelming battery drain problems. However, even using a beta of the new GPU powered Flash, battery life got worse falling 4% to 39% . What is really interesting is that Flash had the best performance under Windows beating both Quicktime and WMV with 62% battery life remaining.

However, there were some strange results in these tests. Note that WMV had the best performance on the Mac despite giving away 5% avg cpu to Quicktime. Likewise on the Windows side Quicktime had the worst avg cpu usage % at 24 but turned in the second best battery life remaining figure at 57%. Clearly the driver development wizards at Adobe, Apple and Microsoft are doing more than just minimizing CPU usage to deliver such battery life figures. Although not atrocious, Flash on Mac delivers about 1/3 less battery life than Quicktime and WMV. Question for Adobe engineers – how can it do so well on Windows and drop by 1/3 on Mac?

Flash is Buggy

Keep an Open Eye was hoping to find that JD Power or some other reputable firm kept software reliability reports. One suspects that Microsoft, Apple, and Mozilla and the other browser vendors keep crash reports. However, they aint telling. So it is very difficult to get a reading on the reliability of Flash or other major video plugins like Quicktime or WMV. This in turn means commenting on “Flash is buggy” is quite subjective. In that nonsensical spirit, Keep an Open Eye went to the Bing search engine and put in “Flash crash browser” and turned up 5.3 million hits while “Quicktime crash browser” turned up only 3.8 million. However, even on the first page one could see references to the recent “Flash Crash” on the New York Stock Exchange…..

Yes this is nonsense. So any readers that do have some serious bug/reliability data sources on Flash versus Apple Quicktime, please send Keep an Open Eye the associated links and references in the comments to this posting below. Your reward will be the choice of a free copy of your favorite IT book or the opportunity to write up  a posting analyzing the data on Keep an Open Eye with payment to you of the $100 posting fee.

Flash Poses a High Security Risk

Unlike in the case of reliability data, there are public security data. Secunia provides a public security advisories database which allows users to check out literally hundreds of software programs on many OS platforms. However to make a valid comparison of security risk, one needs to find a program/plugin comparable to Flash.This  is not easy as Flash goes well beyond delivering video as it also delivers programmed animations with vector graphics, bitmaps, customized audio, and customized text and effects. The closest plugin appropriately enough is Apple’s own Quicktime which has some customizing features for audio, graphics, and text but not to the same extent as Flash.

So let us look at what Secunia has to say about Flash and Quicktime in terms of security advisories for the past 4 years. Secunia ranks the various security advisories on a 5 point scale: not critical, less critical, moderately critical, highly critical and extremely critical.:
Flash has had 7 advisories all highly critical but one which was moderately critical in the past 4 years. All have been patched with the latest advisory cleared within 19 days.
Quicktime has had 21 advisories 19 highly critical, 1 extremely critical and 1 less critical in the same 4 year period. All have been patched with the latest advisory cleared within 14 days.
Symantec’s Norton Security has a similar database of vulnerabilities but it is not so readily searched. Nonetheless, the Secunia advisories were confirmed there. So the data on Security appears solid and is revealing. On one hand Apple and Steve Jobs upbraid Adobe’s Flash plugin for security risks and then tolerate Quicktime delivering 3 times as many advisories as Flash despite the fact that Quicktime’s functionality does not approach Flash. There is a little Cognitive Dissonance here [ aka the kettle calling the pot black ass]. This in turn brings Apple’s Flash bug recriminations up onto the question line.

Flash is being replaced by HTML5

The ace in the hole for Apple and Steve Jobs is the declaration that Flash functionality is largely and rapidly being replaced by HTML5 which Apple is committed to. Now Keep an Open Eye has been eyeing this latter subject very carefully and will have a number of notes on this topic in the coming weeks. But the bottom line is that many but not nearly all Flash+Air features will have HTML5 equivalents and many will and can argue  vociferously about which features are better in specific instances – Flash or HTML5. But there a number of other issues cropping up on the speed and openness with which HTML5 will reach the marketplace:
1)the W3C  HTML5 standard is still being worked on and consensus on key topics such as touch and gestures syntax and implementation, video codecs, and DOM integration among others have some very divergent proposals. These key topics could take months if not years to work out. Just consider that  tabs which are used in every browser are still unavailable to standard  HTML or JavaScript control.
2)There is also the very contentious issue of  proprietary, hardware assisted/accelerated operations for HTML5 on different  OS platforms. This is a very dangerous area as that can and often is used to defeat openness and cross platform uniformity. Suddenly additional options only available on one or a few  designated platforms  creep into the standard. This is  followed shortly thereafter by special options settable/modifiable only on one platform. Just take a look at Microsoft JScript variant of JavaScript and ASP on Windows versus its hobbled Linux subset.
3)Not all the browser vendors have agreed to implement all of the HTML5, CSS3, DOM and other standards. And don’t be fooled. Apple, Google, Opera and Microsoft have still not implemented JavaScript’s very useful and powerful E4X standard. Microsoft is still way behind on old CSS, DOM, and JavaScript standards from the 1998 era that are vital and not deprecated. And many vendors are using Red Herrings like the Acid 3 test as stand-ins for completing full Web compatibility standards.
4)There is virtually no HTML5 development software available yet. This is important because JavaScript is brought right into HTML5. For example,  SVG and Canvas tags  allow direct JavaScripting such  that many Web Graphics Designers a)are not going to be pleased; b)are going to need a lot of clever wizards and help and c) are going to need assistance that goes well  beyond Intellisense. Adobe just released an Adobe Labs beta set of HTML5 extensions  for Dreamweaver CS5. But it is not vigorous or complete. Netbeans will have HTML5 hints and color coding in its upcoming 7.0 version tentatively. Activestate Komodo has nothing as yet. Nor does Aptana [but Aptana Studio 3 with HTML5 goodies has just gone public beta]. Visual Studio 2008 and 2010 just added only HTML5 Intellisense. Sencha released a HTML5 screen designer but it only works for Webkit while touch screen and gestures only work for  mobile applications. In short, there is not a lot of HTML5 development tools despite the huge changes to HTML  and what is available is limited in scope.
5)For the HTML5 that is out there, Flash is beating it in performance in many cases and by wide margins.
Let us consider the last case with a look at some recent performance tests of Flash vs HTML5 on Mac and Windows platforms. These are a series of GUIMark 2.0 tests and produce some very interesting results. The following is typical

Vector Graphic Charting Test	HTML5	Flash 10
Windows 7
Internet Explorer 8.0.7600	N/A	30.7
Firefox 3.6.3	15.73	29.65
Chrome 4.1.249	6.41	26
Opera 10.53	24.77	29.9
Safari 4.0.5	Safari*	29.5
	Avg (15.64) fps	Avg (29.15) fps
Snow Leopard
Safari 4.0.5	4.04	20.55
Firefox 3.6.3	3	23.92
Chrome 5.0.342	2.86	25.48
Opera 10.10	12.22	15.24
	Avg (5.53) fps	Avg (21.29) fps
Bitmap Gaming Test	HTML5	Flash 10
Windows 7
Internet Explorer 8.0.7600	N/A	17.34
Firefox 3.6.3	5.78	17.7
Chrome 4.1.249	10.1	15.98
Opera 10.53	13.59	17.23
Safari 4.0.5	Error*	17.29
	Avg (9.82) fps	Avg (17.1) fps
Snow Leopard
Safari 4.0.5	11.76	13.21
Firefox 3.6.3	7.5	14.09
Chrome 5.0.342	7.4	19.96
Opera 10.10	5.86	14.53
	Avg (8.13) fps	Avg (15.44) fps

Now I strongly encourage readers to goto the Craftminds site where there is a complete posting with 7 tests plus a compelling analysis of why Flash has a speed advantage over HTML5 in many of these tests. Note in only one, the Text Column Test, did HTML5 outperform Flash. And as you can see above, Flash has a 2 to 1 speed advantage in most tests. Also note another familiar trend,  Windows consistently beats Mac in all the tests – be it HTML5 or Flash.

The above arguments would suggest a good deal of caution on planning to replace Flash with HTML5. There are some really tough battles ahead in the standards process let alone development as Apple, Google, Microsoft, Palm/HP, RIM, and Nokia/Intel maneuver for advantage in the rapidly expanding smartphone and tablet marketspaces as those devices collide with the traditional PC and laptops. If you think the collision is going to be gentle and all web standards compliant, meet me at the Bridge in Brooklyn that Keep an Open Eye has for sale.

Why Did Apple and Steve Jobs Attack Adobe Flash

What a lot of readers may have not noticed is that not only did Steve Jobs ban Flash from iOS4 and all the iDevices [iPad, iPod, and iPhone] but he also outlawed Java and program generators. Why he might do this is part of the conclusion of this posting. However, there have been several eloquent postings as to why Apple would do such things. Daring Fireball has a clever one here but omits to consider why Java and program generators were banned in Bost… IDevices. But for the sake of brevity I quote Magnum here:

Flash became  software that has been related to Virus and a source of exploitation on your PC. It introduces serious security problems which cannot be fixed by the operating system’s developer, Apple. It introduces bugs which cannot be fixed by Apple. It forces Apple to be dependent on Adobe for fixes and updates. It forces Apple to live with crashes, security breaches, and bugs. It makes the operating system unstable. It seriously DRAINS the batteries.
Apple definitely doesn’t want to have those problems. That’s why they are shifting to HTML5, CSS Animation, H.264 — Apple can do almost all of what Flash can do without the problems that Flash creates.

This is in a nutshell the crux of why Steve Jobs ruled out not just Flash but by implication Java and program generators from his iOS4 and associated iDevices. Maximum Apple control over what appears on Apple iDevices.

Now more sinister viewers might accuse Apple and Steve Jobs of trying to create an Apple “monopoly” equivalent to the Microsoft PC monopoly. This is the dominant monopoly in IT where even in China Windows has a 90% ++  market share.  But Apple is after no such thing! Steve knows that even with the wildest success Windows PCs like the IE browser will just survive. So Apple will likely never achieve even a 60%++ share of client computing. So that means  that the DOJ’s antitrust division will never be able to accuse Apple of having a monopoly [the crossing point is 60 to 65% market share].

Meanwhile Apple will have very tight control on what appears on iDevices and a handsome cut of all  the revenue streams from them. Apple approves the software. Apple sets the price categories. Apple takes a very generous cut of all program and media sales. And no iDevice App can easily be ported to other platforms because all the cross platform paths – Flash, Java, and program generators have been cut out of iDevices development.

But wait – you say HTML5 is available for cross platform development. Uhhh maybe not! Remember those platform and hardware specific accelerators and APIs. And remember the crucial touch and gesture HTML5 standards have not been agreed to. And Apple has a patent pot to call upon. Apple can be confident that HTML5 iDevice apps running on other devices or PCs via HTML5  is maybe 2 to 3 years away if ever at all.

So, has Flash in particular but also Java and program generator vendors been treated shabbily? Yes, most definitely. And in the case of Flash the evidence above would seem to indicate a possible  courtroom remedy for Adobe. But Apple holds an ace card – Adobe does not want to offend those fanatic Apple users [after all they appear to have paid  double if not triple price for Mac systems that short change them where it counts the most – graphic performance]. Those graphic users make up such a large chunk of Adobe graphic suite sales, and so Adobe dares not sue Apple at the risk of provoking the Apple fanbase ire. So after this bit of  “business finesse”,  Ayn Rand would say to Steve Jobs – “now you have finally done it right”.