Web Things, by Mark Baker

Just stumbled upon a requirement to use REST for an EU software project;

   System to system interoperability

   Should use:

      1. Roy Fielding's REST principles

Of course, in the same list they also mention “XML-RPC”, which cannot be used RESTfully (unlike SOAP, also in the list), but it’s a start. I’ve talked with folks from other government projects, and they wanted to use REST because they wanted to use the principles that made the Web work; they just weren’t exactly sure how to go about it.

Hmm, somehow I’m just not too excited about things this year. Perhaps this has something to do with me making the most intriguing prediction – the inevitable death of Web services on the Internet – three years ahead, two years ago.

But first, let’s see how I did with last years predictions…

As alluded to above, I said that Web services would continue to struggle for adoption on the Internet, and lo and behold, that’s the case. Ho hum. +1

I also claimed that another high profile public Web service would be released. I can’t think of any really high profile announcements this past year except for eBay I guess, but theirs seems SOAP-only, and isn’t public. Bloglines announced their services, but AFAICT, they’re REST-inspired only with no SOA side. So I guess I flubbed that one.

Ok, so this year, hmmm… As I see it, things have really got to start hitting the fan in the SOA space. To that end, I make the following two predictions;

• at least one prominent second tier Web services ISV will move away from WS-* and towards an architectural style which adopts a constrained interface, such as REST, MEST, MOM, or one of the Grid styles.
• a prominent Web services architect will have a Gestalt moment and realize that the Web is actually what Web services have been trying to become since they started down the “document orientation” path in 2001. This person will embrace the Web and REST, though it will be done in a manner which saves face for them and their employer, thereby making it difficult to determine that this is, in fact, what has happened.

A great post from Clay;

And this is the ur-message of Flickr use at ITP – this is what web services looks like when it’s not “Web Services.” No SOAP, no UDDI, no BPML4WS, just good old REST-alicious modeling of resources, and an adopting population that wants to get things done. This is an easier and more labile version of web services than anything being hawked by Big Iron or (Big Binary) vendors.

An interesting post from Dave. A few comments…

I’ve been saying for a while now that I think it’s a shame that SOAP 1.2 didn’t define a general SOAP to HTTP binding that used HTTP as a transfer protocol, for the previous 2 reasons.

It does, Dave. The default binding is a transfer binding; I made sure of that. I think you’re confusing how people use it with how it’s defined. Web services proponents generally think that a SOAP envelope is a SOAP message, yet that interpretation is not licensed anywhere in the spec, and is even explicitly rejected in the HTTP binding where the state transition table clearly shows HTTP response codes affecting SOAP message semantics. It’s also alluded to in the glossary where the definition of the two terms differ (you think this was accidental? Hah! 8-).

I would love it if there was a reasonable way to bridge the SOAP/WS-Addressing world and the HTTP Transfer protocol world, but I just don’t see that each side really want the features of the other side. The SOAP/WSA folks want the SOAP processing model for Asynch, and don’t care about the underlying protocol. The Web folks want their constrained verbs and URIs and don’t care about SOAP processing model.

Avert ye eyes! False dichotomy alert!! You can get the SOAP processing model, and HTTP as transfer protocol (including asynch, which HTTP handles just fine despite insistance from many that it doesn’t) simply by using SOAP in the manner prescribed in the SOAP 1.2 spec and default HTTP binding. In order to do so though, you need to give up on the idea of a new (non-URI) identifier syntax. This is really not a big deal!. We are, after all, primarily talking about syntactical differences here. What EPRs are trying to do is comparable to inventing a new alphabet for the english language; perhaps there are benefits, but I think the phoenician alphabet has a, ahem, rather large and insurmountable head start in deployment, making those benefits – if they exist at all – completely inconsequential.

Dave then makes a really interesting statement of the “protocol independent” variety;

Here’s a test case: Would the Atom protocol switch to using WS-Addressing and then use the HTTP as Transport binding(s) and HTTP as Transfer binding? Seems to me not likely. The Atom folks that want to use HTTP as Transfer have baked the verbs into their protocol, and they won’t want to switch away from being HTTP-centric. And same as I don’t see the SOAP centric folks wanting to “pollute” their operations and bindings with HTTP-isms.

Emphasis on “baked the verbs into their protocol”. Seriously – no matter how you slice it you’re always baking verbs into a “protocol”, because an application developer has to know what verbs they’re using. The problem as I see it, again, is one of nomenclature; that Web services proponents have a very narrow RPC-inspired definition of “protocol” (transport), and their mental models built around this definition simply can’t fully absorb the implications of the broader definition used in the IETF and W3C (transfer). They simply can’t conceive of something called a “protocol” playing such an enormously significant role in a distributed system, yet this is precisely how all existing Internet scale systems are built, and precisely why Web services proponents haven’t yet realized that the Web is what they’ve been trying to build, at least since the quest for “document oriented” services began in 2001/2002.

One might also look at Dave’s statements and ask themselves, well, if they’re going to be dependent on a protocol, then it might as well be the most successful one ever developed rather than one which has struggled for deployment anywhere except behind the firewall. And somebody please remind me; why is it desirable to be independent of a transfer protocol, but dependent on SOAP the protocol?

Tim Ewald’s asking all the right questions;

If all of this makes your head spinning, it should, because there is a lack of consistency here. If I’m designing a Web service, where do semantics exist? Are they in the message body, where they started? Are they in one or both of the action URIs? If they aren’t in SOAPAction because we don’t want to count on a header, should they be in wsa:Action, which is also a header? Can portType/operation define semantics (as the default values of various action headers suggest) or not?

Sounds eerily similar to some previous comments of mine.

I’ve used (and studied) many distributed computing platforms, and I’ve never seen anything quite so thoroughly fouled up as Web services has fouled up the the essence of the contract. It tried to be everything to everybody, and in the end is nothing to anybody. Constrain or die.

Yes, it’s protocol independence theme month!

A gem of an exchange between the WSD and XMLP WGs.

It turns out that WSD wanted to be able to send a SOAP request via HTTP, but get the response back on some other channel. Fortunately, HTTP supports the 202 response code which permits the server to indicate exactly that. But unfortunately, the default SOAP 1.2 HTTP binding explicitly does not support 202. It actually used to, but the protocol-independence promoters had it removed because they feared, IIRC, that too much of HTTP was exposed to the application.

That made my week. 8-)

P.S. I hope to see lots of people out at XML 2004 next week. I drive down tomorrow. It’ll be my third road trip to D.C. in as many years; about 1000 kms each way, but a pleasure in my ride (though not as nice as a trip to Boston through Vermont and New Hampshire!). I’ll be announcing what I’ve been up to for the past little while too, with my latest client.

Patrick reminds me how similar the “XML is a universal solution” position is to the that of SOAP as similarly universal. In both cases, the premise is that by being independent of data models (for XML) and application interfaces (for SOAP), broad applicability is achieved.

Now contrast this with the Web/Semantic-Web position, that by constructing a generic data model and generic application interface, broad applicability is achieved.

Two interesting hypotheses, for sure. If only we had some objective manner to evaluate them!

The real issue with protocol independence, I believe, is the word “protocol”; that the two camps in this debate – the Web/Internet folks, and the Web services folks – each have their own, quite different, definition of the word. For Web services proponents, “protocol” means one thing and one thing only; a spec whose job it is to move bits from point A to point B over a network.

Meanwhile, for the Web/Internet crowd, “protocol” has a much broader definition. In common use, it encompasses the “bit moving” specs, but also others which do a lot more than simply move bits (more below). Some even (properly, IMO) refer to the data formats, such as HTML, as protocols, though you don’t see that too often any more.

As if this disconnect wasn’t bad enough, another – interface constraints – compounds the problem. Specifically, Internet based efforts (including the Web, of course) always start with an interface constraint. This is simply for the reason that they’re (usually) always focused on a single task – for example, email exchange, mail folder access and synchronization, file transfer – and pay little to no attention to what it means to define interoperability between those applications, since that’s tangential to their primary objective. A consequence of this approach is that there becomes little value in using a common sub-layer-7 protocol (like BEEP, IIOP, or how most people use SOAP). This has enormous benefit, with the big one being it permits the mechanics of mapping onto the network to be optimized for the semantics; consider that without GET, HTTP wouldn’t have needed a bunch of features, in particular caching. When semantics are detached from the “wire format” (as with BEEP et al, as mentioned previously), it’s optimized for no particular application, thereby resulting in poor performance for practically all applications.

I’ve drawn a diagram that I hope helps explain these two different views;

Hopefully you’ll at least see the fundamentally different visions of the stack in play here, and perhaps better appreciate my concern about “protocol independence”. Protocols play a much more important role in the stack on the right than they do in the one on the left!

Steve promptly follows up. In response to my comments about self-description, he writes;

Reading over Marks comments, I’m having a hard time determining if this is a technical criticism or more of a complaint about the division of responsibility between the standards.

I was just pointing out that the envelope he described would not, in fact, be self-descriptive. There are many possible implications of this depending upon your POV, but yes, in (my view of) Steve’s view of the stack, SOAP should have a dependency on WS-Addressing, and should have defined the equivalent of wsa:Action. In my view of the stack, it should not have, and in fact there should be no WS-Addressing because application protocols already provide their own addressing mechanism.

And a slight historical correction; “TCP” circa 1974 was in fact the functional equivalent of TCP/IP, with the two layers munged into one. “The Split” into separate specs and layers occurred in 1978.

I absolutely would consider being independent of WS-Transfer a good thing, if my applications hard dependency on the protocol was making it really hard to accommodate the new requirements I wanted to satisfy.

How?!?!?! WS-Transfer provides the application semantics that an app is hardcoded to use, just as a stock quote app would be hard coded to use getStockQuote. You can’t just swap in WS-Notification, IMAP, POP3, or any other application protocol and expect the app to work the same, because they provide different application semantics!!

You can’t escape it; protocol dependence is a necessity, since protocols are the basis of all interoperability. Show me a working Web service, and I’ll show you where it’s protocol dependent.

This is the fundamental misunderstanding of Web services, and IMO what’s preventing a majority of Web services proponents from realizing that the Web is what they’ve been looking for all along.

I think that’s where we are with HTTP right now – it works in the simple case, but there’s all this other stuff we want to be able to do (security, transactions, duplex communication, end-to-end reliability, etc). Mapping all that stuff into HTTP is turning out to be counterproductive and quite hard. However, those problems are tractable if we assume an infoset-centric view of the world. When the status quo doesn’t work any more and something better is available, it’s time to change the status quo.

The status quo works just fine. I hear folks make the claim to the contrary all the time, but have yet to see an example of a Web service that couldn’t be done in an all ’round superior manner using HTTP and URIs. I’m not claiming that an example can’t be found, only that there just aren’t that many, at least once you’ve chosen to use coarse grained, document based messaging.

… to my latest. He writes;

Given that in Mark’s world a SOAP envelope is not a SOAP message, then I’m assuming that a SOAP message must logically be {SOAP envelope + other stuff}. Considering Mark’s position with respect to REST, I’m assuming that the magical “other stuff” is a resource URI and an HTTP protocol verb. I don’t want to put words in Mark’s mouth, but that’s my interpretation of his view of the world.

Pretty close, yes, but the “other stuff” is more than just the HTTP envelope of URI/method/headers, it’s also the stuff contributed by TCP/IP, Ethernet, or whatever else frames the SOAP envelope.

If my assumptions about Mark’s view are correct, I have a hard time seeing why it’s so important that this “other stuff” be externalized from the envelope. I don’t see why including the destination URI and protocol verb as headers is such an offensive concept. This is, after all, exactly what WS-Addressing does – it puts all the information necessary to process an envelope inside of the envelope itself, so the envelope is totally decoupled from its delivery mechanism. What am I missing here – why is having the envelope stand on its own such a terrible thing? It’s still a “document wrapper” at that point – one that happens to be fully self-describing, in fact.

Actually, it’s not in fact, self-descriptive, which is a large part of my complaint and concern.

If you worked your way up the stack, outside-in through the message, each frame tells you how to process the next; the ethernet frame includes a field which says that it contains an IP packet, the IP packet contains a field which says it’s a TCP packet, the TCP packet contains a port number which says it’s an HTTP message, and the HTTP message contains a header which says it’s a SOAP message. Along the way, of course, the semantics of the message are determined, including the application semantics provided by HTTP; the operation, message endpoints (the request URI), and the various headers. But there is nothing in the Ethernet, IP, TCP, HTTP, or even SOAP specs which says to expect that there are actually new operations, and new endpoint references which override those of HTTP, within the SOAP envelope itself.

Some more good comments and questions …

I’m curious as to what’s most important to Mark – constraining the number of verbs or using HTTP as a universal protocol. Hypothetically, what if WS-Addressing was written such that the wsa:Action URI was constrained to 4 verbs, each of which had a direct 1:1 correspondence with an HTTP protocol verb? That’s sort of the thought experiment I see in the WS-Transfer protocol. That’s a compromise that offers REST-style universal semantics while still keeping all the information required to process an envelope inside of the envelope itself. Along with that comes the ability to use any mechanism of moving bits from point A to point B and removes the strict reliance on HTTP.

What’s most important, IMO, is the constrained interface. So yah, I’d be all for stuffing more things into the SOAP envelope if wsa:Action were constrained to uniform interface semantics. But such an envelope, for all the same reasons of self-description above, would only be suitable to be used on top of transport protocols (like TCP), not application protocols.

What you describe in that last sentence is “protocol independence”, and I consider it the root of all WS-Evil (so to speak). HTTP isn’t a bit moving protocol, TCP is. HTTP provides the application semantics, just like WS-Transfer does. Would you consider it a selling point that your WS-Transfer-using app was made independent of WS-Transfer? I didn’t think so. That’s how absurd I see the requirement that apps which use HTTP should do so independent of HTTP.

I think he’s incorrect about his first statement – to be technically correct, you’d have to say that WS-Transfer reinvents HTTP on top of SOAP on top of any number of network protocols, one of which happens to coincidentally be HTTP

Yes, true. I just like saying “HTTP over SOAP over HTTP”. 8-)