The Wave of New Tools and Paradigms

Three significant trends will continue to impact developers substantially throughout 2004 and 2005. First, Open Source software development tools are not only becoming main stream but also are evolving their licensing terms to define a new rival standard to EULA-based commercial software. This will impact profoundly how software will be developed, priced and supported. Open Source software is rapidly becoming a major player, particularly in software development tools. And commercial vendors are already adjusting to that new reality.

Second, due to the complexity of current n-tier design and deployment, development software tools are splitting out into three new development paradigms: i)Backend Services, ii)Presentation Services, and iii)Interprocess Control. This split out is necessitated by the fact that the major IDEs either cannot deliver solutions or do so inefficiently for the new requirements: information at your fingertips, anytime, anywhere on any device. These requirements place conflicting needs on systems. Dynamic, runtime adjustable programs that are also highly secure and reliable. Or thin client apps that get most of theirstarting and ongoing instructions from a central server but that must be able to run offline and and then return to the central fold.. Fortunately, these new specialized tools can still be used effectively with the big IDE-Integrated Development Environs (think Microsoft Visual Studio, IBM Websphere Studio, Borland JBuilder, etc) because the specialized tools deliberately back integrate to the IDEs or have XML, Web Services, or other integrating connections. Also the role of the IDEs remains intact for a substantial swath of development tasks and roles. But don't be fooled, there is also a growing percentage of IT projects that are best handled by the new development services.

The third major trend is that a huge wave of new software will be appearing over the next year as recession and technically delayed software plus the new specialized toolkits reach the market in a huge tsunami of new development tools. For example, the major vendors will have big improvements of their already giant IDEs: BEA Weblogic Server, Borland JBuilder, IBM WebSphere , Oracle's JDeveloper, Sun's Java Studio among others can't fail to have major updates given the new J2SE 1.5 with its syntax improvements and J2EE 1.4 with its new extensions as well. Likewise, Microsoft's Visual Studio has already seen a major update with the release of BizTalk 2004 whose routines have been largely incorporated into the Studio. In addition, Whidbey or Visual Studio 2005 will launch in the first half of next year with further .NET based implementations.

As well there is a wave of new software coming from the Open Source communities. Already, MySQL has turned heads with its new cluster capable database. PostrgeSQL has new replication and .NET interfaces. PHP by early summer and Perl within the year will have significant new versions of their very popular web scripting languages. And NetBeans (beta is looking awfully good) and Eclipse(has a big lead in popular plugins) will be duking it out for fairest of Open Source Java IDEs with major upgrades. Ditto for category leading JBoss in J2EE servers. And this list just skims the surface of what is happening in Open Source tools.

Finally, the previously mentioned specialized tools are emerging on the market. This line up of new tools follows the MVC pattern. They include tools for M-Model tasks associated primarily with Backend Services such as MDD-Model Driven Development tools or SOA-Service Oriented Architecture tools. Next there are tools for V-Viewer tasks associated with Presentation Services such as RIA-Rich Internet Applications, PSI-Portal Server Interfaces or new Smart Graphics. Finally there are the C-Control tasks represented by InterProcess Control services lead by development tools for BPM-Business Process Management, EAI-Enterprise Application Integration and ESB-Enterprise Service Bus. These new tools help meet the new requirements of distributed processing.

In essence the problem boils down to this: delivering anytime, anywhere, and on any device information at your fingertips is simply not possible with the existing giant IDEs and 2-tier architectures. Neither CORBA nor Java nor .NET as delivered in the major IDEs is up to the task.. Some tools have to be supplemented by one or more of the specialized development tiers and associated servers. The classic example is the adding of a Presentation server to beyond current thin client development. The big IDEs can handle thin clients. Java IDEs use JSP to dynamically generate final server pages while Visual Studio has ASP.NET doing the same. But now end users want both thick and thin services; online, offline and rich media operations from the same application which the big IDEs are hard pressed to deliver. So it is up to the new Presentation Servers like Macromedia Flex or Plumtree Suite to provide more robust interfaces. But it is important to note that the big IDEs still can be used to setup backend and/or collaboration services because the new Presentation Servers will have the hooks and connectors to integrate withthe big IDEs.

The Problems of The Big IDEs

Some developers don't like big IDEs from the get go they prefer to work with Visual Notepad and do their own makes and debugging. And that is fine as long as they can remain productive. But the pace of change and complexity of systems find that big IDEs can contribute positive benefits by automating tedious and/or repetitive design, coding, and debug tasks. And most of the big comprehensive IDEs can deliver either standalone desktop .exes or 2nd generation client server applications.

These second generation client server application are nothing to sniff at - they take advantage of the Web interface, use application servers for business rules and smart load balancing and help guarantee application integrity with underlying database systems. There is even accommodation for asynchronous processing of some tasks so suppliers and customers can be integrated to some extent in to a larger system flow. Finally, many IDEs provide developer productivity aids as simple as color coding and code completion to automatic testing with refactoring hints plus build and project conveniences like Ant to sophisticated forward and reverse engineering of code snippets to drag and drop visual design.

It is almost a case of what more could you ask for ? But end users are asking for more especially now that computing has gone mobile and connected through WiFi and high bandwidth network connections. They want output to adapt to many devices: tablets, PDAs, mobile phones, smart kiosks, etc. And they want the devices to work whether connected or offline or occasionally connected. This puts enormous demands on developers. They now have to master input/output onto many and diverse devices. Also developers cannot rely on a server to be able to control and manage the session as ASP or JSP do because the device maybe offline, yet the user expects to get some work done with cached data. This puts pressure not just on Presentation Services but also on the Backend Services.

Finally, many developers and system architects argue that systems cannot afford to be developed in isolation. All the problems of isolated islands of information, silo applications and EAI-Enterprise Application Integration can be traced to a)lack of basic interoperability standards for data and programming and b)lack of planning for cross system interactions. The big IDEs are great for single projects but weaker at integrating into legacy apps and weaker still at adapting to and setting up systems with multi-step workflows or business processes that step across organizational or system boundaries.

We shall look at the new MVC-tier of tools and Servers that extend development into new paradigms. For example, almost all of the MVC tools utilize existing technologies in new fashion so developers are, in most cases, not confronted with learning a new language or methodology. Second, almost all of the MVC tools bend over backwards to back integrate into existing tools and legacy systems. Not just XML or Web Services but broad interoperability, open and defacto standards like Adobe PDF, Apple Quicktime, Macromedia SWF, W3C XMI, CORBA, EDI plus cross platform functionality are prevailing. So we start our tour of new tools with the M-Model=Backend Servers.

New Backend Server Tools

Developers have had a love-hate relationship with automated programming toolkits that expedite development by generating application code as in CASE-Computer Aided Software Engineering tools of the late 80's and early 90's or the 4GL-4^th Generation Language tools of the 70's and 80's. Well they are back and primarily because backend development has become both complex and full of lots of mind-numbingly tedious work to do. Plus now there is the need to deliver code in both on-the-fly and expedited fashion as computing pervades all aspects of business. Finally, the concept of dynamic generation of code from templates has taken root with 2-tier presentation systems like ASP, JSP, and PHP using code-generation extensively. For example, displayed web pages are dynamically programmed or generated back at the server. So now developers are more conceptually disposed to solving problems with generated software.

So the problems of Backend Services are threefold:
1)Development of n-tier distributed backends with requisite reliability, availability, scalability and security is non-trivial. Changing and modifying a system is sometimes fraught with problems as mergers and acquisitions or changes in supplier/client relationships either create or remove some critical requirements of the system. The ability to modify enterprise systems, particularly at the backend, is a major challenge.
2)Asynchronous operations with possibly disconnected clients and extended transactions lead to a hornets nest of transaction integrity and long-duration commit or rollback conditions. This will require automated, manual, or some tiered conflict resolution/ reconciliation methods. As well, the best methodologies for doing this are still being worked out.
3)Web Services are competing for the first time against established CORBA, pure RPC and other EDI-inspired service methodologies. Given that Web Services still has not worked out all its security, reliability and scalability technologies it is no wonder they are being adapted cautiously.

To meet these challenges, SOA-Service Oriented Architecture helps to solve some of these backend problems because it is a framework for defining any functions or services using a standards-based descriptor language which provide consistent APIs or methods/interfaces used to call business processes. . As well SOA describes processes and interfaces that are platform independent: this targets the anytime,anywhere and any device client seeking services that are OS and programming language agnostic. In effect SOA seeks to create platform independent, reusable components. For skeptics, backend services sound like the Holy Grail of software.

But in fact as basic backend services are evolving so too are methods of provisioning services for them. Because systems are operating over many tiers, there are constant data mapping and binding tasks plus other rote operations that are conducive to automation. As well the advance of object design methods with strong pattern templates and refactoring methodologies plus XML exchange and mappings puts both services and auto-generation systems on a much more solid base playing into the strengths of the new MDD-Model Driven Development and SOA-Service Oriented Architectures. Finally there is not just the appeal but the realization that using a constantly updated design model to drive auto-generation(MDD) or refactoring of large subsystems(SOA) is within practical reach. See the Backend Services references on MDD and SOA below.

      Figure 1 - Popular Developer tools like Oracle JDeveloper now deliver SOA Toolkits
The problem with the technologies is that each has had a history of mixed success. MDD's previous incarnations foundered on proprietary and expensive CASE tools while SOA contends with ghosts of DCE and CORBA which some major vendors refused to support. But each technology is taking a more flexible approach this time and has the benefit of more and better cross platform standards. MDD has them in cross platform objects with Java in a major way and at least standard syntax and some cross platform libraries in the case of C++, JavaScript, PHP, Perl and Python.

          Figure 2 - MDD tools like iO's Arcstyler use UML as a Point of Departure
Both MDD and SOA can look to XML not just for data interchange but also XML Web Services. Part of their flexibility is also revealed in a measured approach: development and deployment of MDD tools has concentrated on basic backend services while SOA is taking a vertical applications approach with Rosetta.Net and other specialized business applications (HR, supply chain, CRM, etc). Finally, expect SOA methods and tools to merge over time with BPM/EAI/ESB technologies (see below) as the mandates and underlying technologies converge.

The Current Presentation Challenges

Currently Presentation Services present an even more daunting challenge for developers. The Web interface has gradually won a preeminent role in development. However, the Web interface has a number of problems:
1-the Web UI can be difficult to code to for lack of uniformly enforced standards (Microsoft is now the most remiss vendor on JavaScript, CSS, and DOM standards) plus the sheer number of technologies employed: HTML, CSS, JavaScript, Flash, Java, PHP, Perl,XML in many forms:XHTML, XSD, XPath, XSLT, XSFO, XFORMs, etc;
2-this coding is subject to increased security and reliability risks;
3-current technologies also require too much to and fro-ing between client and server with performance and usability penalties;
4-Web UI's controls are getting long in tooth or are weak in rich media support;
5-mapping to new, mobile/PDA devices is often in different and incompatible code;
6-multi-view portals with collaboration require hard to do dynamic integration;
7-Web interface does not readily support both online and offline operations;
8-portability across browsers and PC platforms is starting to weaken.

None of these problems are absolute show-stoppers for the Web interface. And some problems are being addressed by current software. For example, ASP/ASP.NET, JSP/JSF(Java Server Faces)+Struts, Cold Fusion and other markup languages start to address issues 2, 4, 5, and 6. But it is the new generation of RIA-Rich Internet Applications and Portal Servers that effectively take on the Web Interface issues. The second generation of Presentation Services
 
                    Figure 3- The Macromedia Flex IDE - Built on familiar standards
attack a broad spectrum of the Web Interface problems particularly improving runtime speed, better choice of components and rich media, plus ability to deliver multiple applications or portlets which may be intercommunicating in the same workspace.

In essence, rich clients are a reaction against too thin clients. Typically, RIAs and Portals bring back to the client-side such operations as validations, simple business logic, UI control and event handling, even database interactions. The portal client also addresses Web interface issues. Notably 6)because they are designed to be multi-views but also 4) with a broader range of component support and 8)by running on a wide range of PC clients.

In fact, it is encouraging from a developers point of view to see that most vendors are adapting known technologies rather than re-inventing the wheel. XML is being heavily used in many cases with little departure from standards. The one major exception is the proliferation of different XML UI component description dialects: Laszlo Lzl, Macromedia MXML, Mozilla/Nexaweb's XUL, and Microsoft's XAML being examples. Also JavaScript usage is coming closer to standards as Macromedia makes ActionScript 2 adhere much more closely to ECMAScript 262 version 3(the JavaScript 1.5 base standard) while Adobe Designer and Microsoft InfoPath uses JavaScript in standard fashion throughout.

     Figure 4 - RIA Tools certainly deliver richer Web interfaces to the Browser
With the notable exception of Microsoft, all the vendors provide many options for their Presentation Services to connect to a broad selections of backend servers. For example, IBM's Websphere Portal and Sybase's Portal can automatically generate portlets from database queries or from defined XML files or static HTML content. Plumtree Web Suite does the same and adds integration services to link in other content management, search services, plus J2EE and .NET application services.

However, backend connectivity can create issues for 7) offline operation and 6) multi-view or portal operations since those environs become more challenging for the backend interface. In effect the client can become too big and unwieldly again. Also as the client becomes thicker to allow for these stand-alone and/or multi-views, security and performance start to become more problematic. Security is an issue because dynamic code creation and offline/online transitions create points of exposure. Performance can suffer because as the client becomes fatter and transactions longer or reconciliation takes longer there is slowing of overall transaction rates.

The split of development into specialized Presentation Services appears to be a win-win situation. Because most RIA-Rich Internet Applications and Portal Servers and their associated studios a)provide easy to use visual designers speeding development while b)including strong connections to backend services to integrate well with existing systems and development tools. And from the programmer's point of view, most vendors are using a blend of existing technologies and supplying complete, developer editions for free download so getting up to speed on some of the new technologies is easier than ever before. In sum look for Presentation Services to be split off from the big IDEs until those tools can offer comparable online/offline, anywhere, anytime on any device user interfaces.

InterProcess Control System Problems

InterProcess Control Systems have long been a part of software development. Message Queuing Middleware, Peer-to-Peer systems, Multi-user Agenting, Workflow Management Systems are a small sampling in this arena. There have been many good systems from a variety of vendors: Staffware, Lotus Notes, Jini, Groove, etc. But many of these software tools have been held hapless to contend against two major software disabilities. First, in computing there has been a very low level of data interoperability. For example, it has taken about 45 years of development to arrive at an adequate and universal character coding scheme, Unicode. In the 1980's, technologies and data interchange processes such as SGML, SQL, EDI and various ETL schemes emerged but also fell victim to proprietary extensions and simple non-standards compliance - close but no cigar. Only with the launch of the Web has the overwhelming need for a standard data interchange mechanism allowed XML to resist the centripetal forces of proprietary extensions and become a nearly universal and robust standard.

A similar pattern of proprietary pull apart has afflicted programming interoperability. Languages such as ADA, Algol, C, C++, Cobol, Fortran, Pascal, PL/I and Smalltalk to just name a few all became balkanized by proprietary language extensions or by vast but differing function libraries for the same basic tasks. Java so far has been able to resist that trend; JavaScript likely will not be as lucky.

Finally, Web Services with its standard XML packaging has been able to supply the last magic ingredient a standard mechanism for making InterProcess Service Calls between systems on the same or different platforms. Not only that to package the request message but also sophisticated conditional response messages as well. Finally, messages and requests for service can be synchronous (I am waiting for your immediate reply) or asynchronous (signal when you are ready to serve). Actually IT community had at least two previous standards for InterProcess Service Calls but some major software players refused to embrace them.

Web Services changes all that. W3C, OASIS, WSI and others are now at work defining the authentication, encryption, identity, integrity, security and other process standards needed to consolidate the strong base put in place with SOAP, UDDI, WSDL. This will put InterProcess Control solidly back into Agenting, BPM-Business Process Management, EAI-Enterprise Application Integration, P2P-Peer-to-Peer processing, and Workflow Management Systems on a much more solid foundation. InterProcess Control has earned a name - the Enterprise Service Bus or ESB (see Figure below). The question is not "will ESB-like services happen?" (both Gartner and IDC see major adoption by 2005 year end). The question is how and what InterProcess Control services will be provided.


            Figure 5 - What Services should be on ESB is subject to debate

And the industry has responded with rapid developments in the past 2-3 years in all of above areas as eliminating island of information and application silos has stayed at the top of IT executives priorities. In addition, business motivators for InterProcess Control now include government compliance decrees such as Sarbanes-Oxley, SEC's rule 17a-4 on Financial Institution retention of info and client transactions or HIPAA employee health insurance compliance and other process-verification regulations such as NASDAQ 3010 and NYSE 342. As well there are organization's own process improvements as envisioned in Six Sigma and other process methodologies. Table 3 below lists some of the major players and the type of InterProcess Control system they are selling.

        Figure 6 - Both BPM and SOA vendors provide much better Process Monitoring Consoles
The problem in InterProcess Control arena is the lack of standards for design, control syntax (BPMN vs BPEL) and interoperability among the major players tools except at the major API or Web Service level if at all. However, do expect to see advances in reporting as better Presentation Services are invoked.Also see BPM and SOA converge on so called composite applications - that start to share components for say query and search, monitoring/reporting or backup and recovery etc. The whole idea will be to orchestrate common usage in BPM and find and re-use common components in the SOA approach. Look for all the InterProcess Control tools to contribute to better configuration and change management on an enterprise level.

Forrester is reporting that BPM is doing well despite the economy just emerging from a down turn. Trial of new BPM is up by 300% over 2002 with technology and telecom firms leading the way. So expect to see the trend in system development to follow project management. Five years ago project management tools were used primarily on single, large projects; now they are used on many projects and contribute their results into portfolio of all project results. These data are then used by the project portfolio programs to not only rate the projects but also help to decide the sharing and movement of people and other resources between projects. BPM, EAI, ESB are starting (along with SOA) to foster that same global view of system applications and the running assets they represent so these components can be shared across the enterprise.

Why the Tsunami of Tools

It should now be fairly obvious why there will be a wave of tools which will be coming to market through 2004-2005. First it is the usual rate of change in software development exaggerated by the 3 year slowdown in the economy. But don't underestimate the effect of finally having open standards for data interchange and lightweight persistence with XML plus a highly interoperable programming language in Java. This has tripped off a lot of innovation in novel solutions to the tough problems of n-tier, heterogeneous, distributed processing. Also, don't discount the influence of Open Source. As we have seen Open Source is not only contributing to the new specialized Server categories but also is helping to curb proprietary extravagances. Finally, for developers it is a mixed bag. Yes, the new MVC Servers may mean smearing an application over 3-5 tiers with all the complexity that entails. But the good news is that much of the technology used is well integrated with the novel application of old familiars. So this may be more a story of not the new but the rediscovery of Paradigms Lost.

Jacques Surveyer is a consultant & photographer, see the latter at thephotofinishes.com

References:

Presentation Services

RIA usability practices a good analysis of what makes a rich interface richer - Boxesandarrows
Another Designer/Architect View - Sitepoint
RIA examples with Flash-based tools - Webvertising.com
Techweb Enterprise Portal Suites Review- NetworkWorld
ZapFlash: You can never be too rich or too thin by Richard Schmelzer- ZapThink
Techweb: Altio Revs Presentation Layer Server - InternetWeek
Techweb: Dr. Dean Gets Rich Client - Serverpipeline

Open Source Links

-Valid Open Source licensing Terms List contained at Open Source Institute - OSI
-The changing face of Open Source in InternetNews.com by Susan Kuchinakias describes new 2^nd gen Open Source projects where lead developers are permanent hires. - InternetNews
-Open Source Part 2 in Information Week by Charles Babcock describes new licensing, support - InformationWeek
-Open-source MySQL DBMS adds clustering tech in ADTmag.com by John K. Waters looks at clustering and other advances from MySQL and other dual-licensing database vendors - ADTmag
-Prospering in the Open Source Software Era by Karen Schaeffer her website. Enunciates the changes in licensing and support in Open Source Software and its implications - Neuralscape
-Jboss' Fleury Seeking for-profit Open Source incubator by Vance McCarthy at IT Managers Journal new VC capitalized, for-profit trend in Open Source explained IT Managers
-A Not Quite Open Source - Open Source Project in World Oil another variation - Worldoil
- Paradigm Shift by Tim O'Reilly from personal website June 2003 reflections on Open Source Software's sweep into whole software realm - tim.O'Reilly.com

InterProcess Control Links

New vision of EAI/BAM/BPM- Overview by Yankee Group for InterSystems Ensemble - Yankee Group
-Overview of key players and dominating trends in BPM by Upside Research @ Upside Research
-Gartner magic quadrant on BPM for Metastorm @ Gartner
-ESB overview @ Network World

Backend Services

-MDA FAQ from OMG @ OMG
-MDSD-Model Driven Software Development Patterns agile base for Model driven SW @
www.voelter.de
-MDSD Model Driven Software Development conceptual base for MDSD by Jorn Bitte @ Softmetaware
-Does Code Generation Matter to Java Developers by Norman Richards practical view -
theServerSide.com
-Service Oriented Architecture Introduction Part 1 by Michael Stevens @ developer.com
-Making SOA A Reality An Appeal To Software Vendors by Beth Gold-Bernstein - ebizq.net
-Forrester says SOA Tools slow to arrive by Rich Seeley @ ADTmag