Here is the Gold Standard for a Vista review: updated Feb 02, 2007
First, because Microsoft is claiming that after 20 years Windows has finally reached Enterprise caliber levels of security, reliability and availability while retaining all of its good virtues – most applications, usability, most applications, new GUI features, most applications, etc – this should be put to the test. There should be independent benchmarks by the reviewers establishing by tests and performance reports that indeed the security, reliability, and availability of Vista is improved over Windows XP and Server 2003 – not the truthiness and limited scope of Redmonds own Get the Facts ad campaigns.
Second, because desktop Apple OS/X and Linux represent the OS with the best record for Enterprise caliber delivery of security, reliability and availability both operating systems should be the primary points of comparison. The IT community already have had such a review from Information Week (and remarkably the review is getting major coverage over at Infoworld). But again, comparative feature comparisons (tabular if you please) are essential here. The public suspects there is great equity among the major OS desktops – show this or show where they diverge – how and why.
Third, because bloat and performance are issues with Vista, benchmarks should be done on equivalent dual loaded machines (possible with all of MacOs/X, Linux and Windows XP) comparing how basic OS features such as program startup, response time, and basic operations compare among major applications like Office, file explorer, browser, IM, Photoshop, Dreamweaver, etc, etc. And there are dozens of certified test suites like BAPCO etc available to help with the comparisons.
Now given that the IT trade press should be exempt from various benchmarking restrictions in Vista, XP and Apple EULAs, these results would be of keen interest to the IT community and would definitely “sell” well to the various readerships. We shall try to keep this table updated for what major IT and other publications are doing in this regard.
| Reviewer | Security Reliability | Apple or Linux Diff | Performance Tests |
| Jim Allchin | Deferred | Praises Mac! | Implies the worst |
| CNET | Nothing | Very little | Nothing |
| Computerworld | broad brush | Says switch to Mac | Nothing |
| DesktopLinux | Good so far | Best Linux v Vista | Will see |
| Information Week | “en passant” | Best coverage so far | Nothing |
| MIT Tech Review | Little | Apple only | Nothing |
| Paul Thurrott | Little | Only in passing | Littler |
| PCMag | So very little | XP versus Vista | Best so far |
| PCWorld | Nothing | Nothing | Nothing |
Finally note the absence of eWeek and Infoworld from the Vista reviews – lots of chatter and news stories but no comprehensive review from either. eWeek is promoting the excellent Desktop-Linux review; but really ZiffDavis and Steven Vaughan-Nicholls at DesktopLinux is the power behind this. But despite the Desktop Linux Information Week reviews, what is surprising is how such a good subject gets such casual, even short-changed treatment by most of the major IT Trade press sources. These are the big IT media that can get past the EULA restrictions on benchmarking and testing that is so important in the case of Vista.
Supposedly Vista is bloated – but maybe MacOS is too. Vista is supposed to be a big step up from Windows XP – but we have from the IT Trade Press only barebones benchmark results to give us only an inkling of an idea of what the trade-offs are. And Linux is supposedly lean and trim – does that produce any performance and/or reliability advantages ? And all the major IT trade press reviewers have had a version of Vista for at least a month – tell us about reliability, security versus XP, Mac, Linux. Why should we buy into Windows Vista? Greater security often brings trade offs sacrificing ease of use and fast performance – what is happening here? It is almost as if the IT Trade Press has caught a bad case of Thou Shalt Not Benchmark.
Even on the What is Vista – the IT Trade Press is so mixed in quality. Why would some one buy Vista Ultimate versus Vista Premium Business ? What are the clear trade-offs going with Vista now. Many missing drivers, big learning curve, big hardware investment are the givens – now what are the top 5 things I am getting in return that I cant get from Mac/OS, Linux or even Windows XP? So many questions and good story angles left uncovered by the IT Trade Press. And then the IT Trade Press wonder why they are losing readership to the bloggers, business mags (they at least tell us what the big picture and price performance trade-offs are) and the IT video rags. The biggest story of the past five years in IT and its as if all the IT Trade Press sent either Homer Simpson or reporters with strait-jackets and blinders out to cover the story. A story crying out for simple comparative test and benchmarks among the OS for features, reliability, and performance. Such a missed opportunity.
Anatomy of the Reviews
Jim Allchin – these are candid “Id buy a Mac” memo remarks made to Bill Gates, Steve Ballmer and top level Microsoft team late in the Vista development cycle by THE Windows Executive.
CNET – unfortunately this is the median review, lots of glitz and flashlight shines on this or that great feature, very little perspective or test and benchmark coverage on what should be the best IT story of the decade – how good in real tested and performance benchmark results is Vista compared to Apple, Linux and even Windows XP.
Computerworld – this is another big-viewpoint review by Mike Elgan(former editor of Windows Magazine) and he is echoing the Gartner line: Wait to buy Vista and look seriously at the Mac.
Computerworld – Scott Finnie switches from Windows to Mac
Information Week – also comes down strongly in favor of the Mac; but at the expense of not covering Linux or the very revealing Vista versus XP, MacOS, and Linux performance benchmarks.
Desktop Linux – Finally, a very good Vista review in the comparative Linux versus Vista – a candid must-read
MIT Technology Review – Another Windows to Mac conversion story with surprisingly little meat to back up the switch – run Windows and Mac on the same machine and tell us how they perform. Same machine, same memory, same disk drives which one screams and which coughs and where. And just for amusement add Linux to the mix. Then tell me your choice.
Paul Thurrotts Supersite Vista Review – this is really part of writing of a book on Vista with Brian Livingston; so take it with a block of salt. It is detailed to the extreme but it also is kid glove. Again because most of this is based on beta and RC code there is very little testing and benchmark results.
PC magazine – this is sprawling review but only one to do comparative benchmarks between XP and Vista marred by no/poor summary of benchmark results.
PCWorld – this is a “Visual Tour”=Advertising Brochure of Vista and a cop out on all three tests: no security and reliability benchmarks, no solid comparison with Apple, Linux alternatives; no benchmarks of performance vis a vis XP let alone Mac and Linux. This is close to useless. Dont they understand – blogs and others are taking their business away because they are so docile and minimally informative.
(c)JBSurveyer 2007
If you liked this, let others know: 
Slashdot 
Digg 
del.icio.us 
reddit 
newsvine 
Y! MyWeb