SOA+ESB:Better but not Best

Information Weeks Charles Babcock in the IWeek Aug 12th isssue praises SOA-Service Oriented Architecture in no uncertain terms: “Service Oriented Architectures let companies lay the foundation for software that is fast to write, easy to integrate, and runs on a range of platforms….SOAs provide a way of converting existing software into a set pattern of software components and networking protocols that can create a Web Service. This kind of architecture can also provide a foundation for future apps built that way. When implemented right, SOA software can speed up application development, let companies re-use more code and provide more standardized, less-proprietary means of integration among apps.”

Rigggghttttttt. And we have heard these same words for Object Oriented Analysis, CASE-Computer Aided Software Engineering, 4GLs-Fourth Generation Languages, and Structured Analysis and Design in preceeding silver-lined programming tidal waves occurring about once every eight years. The problem is that when selling the new technology we get sold the upside but not the full set of trade-offs which in effect is the downside.

Structured Analysis taught programming how to make copies of itself by dividing processes and tasks into subroutines and modules that are logically self-contained – they balanced high internal cohesiveness with loose(read simplest possible) coupling between modules/routines. The problem – Structured Analysis was more art than science because it had not resolved the facets or objects issue – how to handle all those slightly different cases that encompassed many common processing tasks and attributes but had a few distinctive and uniquely essential and different attributes and tasks. In short, Structured Analysis was missing the objects and inheritance vision and behaviors.

4GLs taught programming how much could be accomplished with simple, declarative approaches to programming and system design. Declare the problem space and then let transparent, well designed core routines transparently solve the problems and dispatch the results. The core patterns of MVC-Model-Viewer-Controller, Software Factory, and Delegation got almost unconsciously defined and refined with 4GLs. But severe performance and robustness of design frameworks – 4GLs being applied where they had no right to go brought a new wave of CASE tools that would apply data models, templates and customizable code generators to the problem of producing reliable systems rapidly.

CASE was the epitome of my University of Michigan and Professor Teory days – “get the data structure and data model right and your whole system falls into place” driving some pretty sophisticated top down design and then generator systems. Problems: creaking at cross platform, messaging, backward integration, proprietary flavor, and way way oversold and over priced.

OOAD-Object Oriented Analysis and Design was to bring the benefits of reuse, fast and easy development and integration plus cross platform portability and range. Ohhh …. thats what SOA just promised us. What happened to OOAD ?

Well perhaps its short ascendency could be attributed to bad timing. Microsoft was not ready with .NET, the W3C had not yet produced XML and the underlying XML, SOAP and WSDL of Web Services. Also the industry stalwarts had not had the experience of rejecting/spitting out DCE, OpenDoc, DCOM, CORBA, and several other distributed processing architectures. Finally, Open Source was not in full bloom to prevent proprietary from becoming too reckless and closed.

But despite all the naysaying about OO, contemplate this: Huge pre-built OO libraries and component frameworks underlie the two most popular OO development frameworks: Java JVM and .NETs CLR. Reuse is already being applied on a huge scale. What SOA and ESB imply is huge scale reuse of legacy systems. Backward compatibility, interoperability and system integration have finally won fulltime support and allegiance. RIP to rip and replace.

The Web Services Trade-offs

Before examining the limitations and trade-offs of SOA and web Services let me recommend two fairly recent sources of high quality evangelism concerning the same .
Service-Oriented Architectures by Thomas Erl, Prentice Hall – $40US
Enterprise Service Bus by David Chappell, OReilly – $40US
If after perusing over (if not reading) either of these two tomes you do not come to the conclusion that system design is:
a)being seen in the context of a full network of IT Systems and resources;
b)being planned for not only its own internal processing capability but also how the new system fits in and uses existing resources;
c)being designed in a framework where messaging and basic workflow paths between systems can be explicitly seen, described and hopefully drilled down into;
d)being considered in the context of change: where mergers, divestitures, new partners, suppliers can significantly modify the current map of system reliance/interdepenency and data interchange;
-if at least 2 or 3 of these points dont scream out at you – check again.

I think these two books and others on Enterprise IT Architecture signify that IT has finally reached Professor Ashbys level of problem solving competence called the Law of Requisite Complexity (or Variety). In short, the Law looks for the completeness in the statement of a problem. The Law says repeated failures in solving a problem (shades of the Standish CHAOS reports that show 25-35% percent absolute IT project failure rates) may be attributed to the fact that not all the essential dimensions with associated properties and processes have been identified in a problem set.

But having identified all the dimensions and attributes of a problem does not mean at all that a correct description let alone solution of the problem has been achieved. This is where we are at with SOA, ESB and Web Services. The problem domain is reasonably well delineated; the problem definition needs to be much better refined and the solution sets (note the plural – there will be no One Best Way) proposed and tested. Here are some of the major trade-offs and problems:

1)SOA Web Services become much more heavily committed to messaging, workflows and very long transactions. These very long transactions (from minutes to days and months)present severe problems in term of transaction integrity. The current atomic commit or rollback strategy simply breaks down in the long world. There are other approaches – but suddenly business rules, legal requirements/constraints, and basic concepts of fairness and equity come strongly (and diversely interpreted) into play;
2)SOA Web Services puts a premium on trust; especially concerning issues of personal and more tellingly corporate privacy and spying. Web Service messaging and transaction brokers could easily skim off buying and other committment intelligence within closed black boxes and users would not know at all that their info had been “X-rayed and/or aggregated”. Interestingly Web Services measure in the areas of compression, encryption, and non-tamper signaturing are still in the proposal and draft stages in the various standard bodies. Also both Web Services books, particularly Chappells ESB, are light on these issues;
3)SOA Web Services lacks a complete set of tools. Most notably, SOAP, WSDL, UDDI, BPEL, and other XML and Web Service dialects are largely declarative and lack programmatic robustness. Now some would say that is the very strength of SOA and Web Services – they are programming language agnostic. But I am not talking about a programming delivery language to be used by Web Service-based systems but rather a macro command or scripting language used to support construction and delivery of Web Services and operations. Gluing together the pieces of the SOA and Web Services is going to require a new breed of scripting tool. One that is cross platform; easy to use and develop in; readily talks GUI, SQL, XML, SNMP, HTTP, LDAP, RPC, etc; and can be deployed in multiple modes – interpreter, server-side compiled code, and JVM/CLR bytecode. There are a lot of scripting pretenders (JavaScript, JudoScript, Ruby, NetRexx, Xen, etc) but no clear cut winner. And this only addresses scripting, with others calling for a Web Services Metadata facility and still others calling for WebDO uniting SDO, ADO, JDBC and other object data serializing methods and “standards”. In sum, the Web Services tools space is very much in flux and in need of completion;
4)The Web Services political situation is very much out of whack. The standards organizations can hardly control and police their members. Witness W3C and other standards bodies powerless to control Microsoft and others from taking out patents on what has been declared a patent free zone in the case of most Web standards. As well, the major IT revenue players like IBM, Microsoft, HP, Sun and others wield enormous clout relative to some major Open Souce delivery players like Apache, Sourceforge, Mozilla, W3C, etc. Somehow this imbalance will have to be corrected for;
5)SOA and Web Services can quickly go from dead simple to extraordinairly complex. This is true all the way down the pike from design through development to delivery and ongoing operation. Its sort of like “getting to Baghdad is relatively easy” but securing operations and making things work well is no trivial matter by any measure.

So, yes, SOA and Web Services are finally delivering to IT a good/complete definition of its problem space. Make no mistake this is a major accomplishment. The whole nature of wicked problems is getting them delineated and defined properly. And yes, Web Services is delivering robust legacy system inclusive solutions right now – particularly in read-primarily, non complex transaction processing. But dont let anyone sell you the Surefire SOA & Web Services ticket; because its probably a wrapper for a brick “guaranteed from the Brooklyn Bridge”.