« Perfect or sloppy - RDF, Shirky and Wittgenstein | Main

Web Services and the Innovators Dilemma

Web 2.0 is a vision of the web where content and functions can be remixed and reused to create new content or new applications.  Web services and the semantic web are two of the key enablers for this vision but there appears to be dual approaches to both web services and the semantic web emerging. Why is that? Which is best?


Web Services

SOAP & WSDL - opens up new vista of possibilities by solving some of the real hard problems (WS-this that and the other), requires expertise and new infrastructure e.g. toolkits app servers to manage complexity. Unsurprisingly the app server vendors are driving the new standards in enterprise software.

REST - open up a new vista of possibilities by making it very easy to use web application APIs, so new audiences can get involved and doesn't require much in the way of changes to existing software stack. This is largely being driven by a very different community from the enterprise web services lot.


Semantic Web

RDF & OWL - open up new possibilities by solving some really hard problems. Requires expertise and therefore tooling and new infrastructure like a new query language, data storage, parsers etc. Driven by standards bodies like the W3C .

XHTML & Microformats - opens up new possibilities by lowering the barrier to participation for producers and consumers, uses existing technology, can be hand crafted i.e. disintermediates the expert.


It seems to me that the difference in complexity and cost between the approaches is actually a symptom of something deeper.

SOAP Webservices are trying to go beyond what expert developers could already do with RMI, DCOM etc.

By its nature it must compete with what is already possible which is mission critical software systems that are Trusted, secure, reliable, accountable, and typically have a high cost of failure. Most of these developers could not buy into a new way of working if it mean going backwards in any of those critical areas.

Similarly, RDF & OWL are trying to go beyond what expert developers can do with semantics in XML today.

If you are familiar with the work "The Innovators Dilemma" by Clayton M. Christensen, you may recognise this as the classic description of sustaining innovation. It must be better than what went before because it competes along the same dimensions with the same audience.

Clayton also describes what he terms as "Disruptive Innovation" of which one type is the low-end disruption. This is where a technically inferior innovation radically reduces the barrier(be that skill, cost or location) to entry thereby allowing an audience that was previously excluded to participate.  This competes on new dimensions with a new audience.

This massive new audience is currently excluded from the traditional solution so the disruptive innovation only competes against being better than nothing for this audience.

So disruptive innovation allows a new, less skilled community to participate and do new kinds of things. Almost by definition this community is larger than the community of experts i.e. it is the long tail.

If we consider both REST and Microformats we see that neither are technically as good as Web Services and RDF. But both are significantly easier with lower skill and cost barriers for both producer and consumer. And sure enough Amazon are finding that the vast majority of the users of their platform are using the REST APIs.

Software standards have always had a massive network effect. What good is a standard if nobody else uses it. This makes the size of the community around any standard or approach hugely important. The pace of innovation is also deeply linked to the size of the community that can innovate. Consider the number of web authors(including bloggers) who can probably get their heads around REST and Microformats. It is vastly larger than the community hard core software developers on the planet.

Clayton describes, with many examples, how low-end disruptions rapidly become better and better until the complex high end solutions are pushed off the map.

It wouldn't be the first time that innovation become de facto outside the corporate firewall but eventually become good enough to be adopted by the enterprise.

Am I saying that Webservices and RDF are doomed. The truth is I have no idea, but I doubt it. The reason that experts create these new solutions is because they are needed to solve those difficult problems. But, on the other hand, vastly more innovation is likely when ordinary people gain the ability to do what is a "solved problem" for the expert. I would put money on Web2.0 emerging first from the ordinary web user rather than the software experts.





Posted on Thursday, July 28, 2005 at 06:00PM by Registered CommenterJustin Leavesley in , | Comments2 Comments

Reader Comments (2)

Are the "heavier" web services like SOAP really just addressing a different market? There is value for them in mission critical transaction systems.

REST and microformats, in particular, are fulfilling largely informational needs (e.g., "show me a review"). They do outcompete things like SOAP for that. But, SOAP was never really adopted for informational needs.
August 2, 2005 | Unregistered CommenterBud Gibson
I agree with Bud, regarding, say SOAP vs REST. Take the example of the various eBay APIs. eBay has for quite some time had an XML API which is now transitioning to SOAP, for the sake of simplicity let's just refer to it as "the SOAP API".

It was designed from the ground-up to allow 3rd-parties to perform secure transactions on behalf of eBay members. In here, i'm referring to "transactions" in more of a business sense. I would not want to execute a bid, add an item to my watch list over an API built on-top of HTTP GET and REST. With SOAP you can validate integrity of XML payloads against well-formedness, XML Schemas, define complex data types, etc.

But there clearly was a need to make it easier for 3rd parties to proxy simple searches. Their "affiliates" get revenue from driving traffic to eBay, without necessarily proxying any transaction. A REST API to allow non-transactional operations such as catalog searches, item look-up, offers a more effective alternative to SOAP in such specific use cases. Which is why many of us eBay developers were very happy to see eBay toe-in their REST API in late 2004 to perform GetSearchResults operations. It gives us more flexibility in our presentation of search results, giving us the opportunity to pit them against other marketplaces.

In short, it's fairly reasonable to see different more appropriate use cases for SOAP and REST, and it ought hopefully be a matter of developers picking the "best" tool for the "job" at hand.

... keeping in mind that the definitions of "job" and "best" can easily be moving targets, heh!
August 3, 2005 | Unregistered Commenterchris holland

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
All HTML will be escaped. Hyperlinks will be created for URLs automatically.