Joel's SharePoint Architect Blog

Mon, 12 Nov 2007 18:22:35 GMT

It’s been a while since I last saw Pat presenting on SOA. He hasn’t lost the magic. But he has lost a ton of weight. Or gained a lot of height or something! Pat – well done mate!

Continuing his Metropolis series of SOA talks, Pat’s theme this time is around composability and interchangeability of Services.

A Short History Lesson

The European System of Manufacturing – before 1800s

The build of complicated objects required each part to be manually lathed or turned in such a way that no two craftsmen produced the same precise part; no two (otherwise identical) parts produced by the same craftsmen were identical enough to be interchanged. To scale up, more craftsmen were required. When things broke down with the goods being built, replacement parts would require "manual fitting" to make them work. Variance in the manufacture of parts caused unscalable systems.

The American System of Manufacturing – circa 1800s

Due to a lack of manual labour and engineers, the Americans adopted a mechanised approach of manufacture. Heavy (steam) power tooling was used, which saved labour. However, these were not automated, and variation in parts produced during manufacture was inevitable. Manual fitting was still required; machinery just got us to that point quicker.

The US Armoury System of Manufacturing – mid 1800s

Le Blanc, Jefferson, Whitney hypothesised a system of interchangeable parts, which depended upon the manufacture of specific parts being reproducible, predictable, and with tight tolerances to variation. Notably, Whitney won a contract with the US military to manufacture arms using this method; he failed because he was unable to eliminate the requirement for manual fitting – i.e. the methods were too hard to produce identical, interchangeable parts to their rifles.

John H. Hall, circa 1820s

John H. Hall developed accurate machines to build interchangeable parts in gun manufacture. His machines were expensively hand crafted to accurately reproduce, time and again, the same parts down to a very fine tolerance in variation. Ultimately Hall was to fail as he lacked the funds to realise economies of scale.

Sewing Machines – circa 1870s

Wheeler and Wheeler sewing machines produced 25,000 machines in 1860. By 1876 this figure rose to 108,000. They achieved this by using machines to make their products. Brown and Sharp also adopted this method.

The Pope Bicycle – circa 1890

Parts built by Sharps Rifle Company using machines that worked from reusable, accurate templates.

Why is this important?

This takes the narrative from European, hand-finished manufacture, all the way through to the Model T Ford, via the US Armoury and the Pope bicycle.

Model T Ford

Ford was inspired by mass production of meat. Butchers – working in what Pat calls "disassembly plants" – would stand in front of a conveyor of meat carcasses, and apply the meat cleaver to each one without having to move from carcass to carcass. What impressed Ford was the efficiency with which the production line operated. The "operator" never had to move; the parts were brought to the operator. From this, it’s a relatively short journey to Ford’s Assembly Line of Model Ts.

The Assembly Line’s purpose was to eliminate custom fitting altogether. From this he derived the ability to repeatedly, repeatably, indefinitely minimise variation and produce tens of thousands of identical parts. Ford’s machines and buildings were tailor made to the design of a specific car. To change the design required buildings to be dramatically changed, and machines to be retooled. For 20 years the Model T Ford did not change at all in design. A new one was virtually indistinguishable from one 10 years its senior.

General Motors

GM introduced the concept of the recurring annual new model. Arguably the first concrete example of agile manufacturing. GM provided heavy duty, multi-puropse, multi-site production processes and equipment, taking the assembly line approach from Ford and extending to allow reuse of key components – engine, chassis, transmission, etc – whilst allowing evolution of other components – e.g. bodywork, gearing.

This first example of flexible mass production owes everything to mechanised production and complete interchangeability of parts. This allowed GM to rapidly (over a number of years) change to new models.

From History to Transactions

American System of Transaction Processing

Pat then draws the comparison between early American manufacture to traditional transactional processing.

Any two appilcations performing similar tasks are allowed by the technology an immense amount of flexibility and variance in implementation. Consider the operation to update an accounting and an invoicing system. This could be implemented in a massive variety of different ways; at the simplest level, updating accounting tables first, and invoicing tables last. Alternatively the other way round is equally logical.

It is precisely this flexibility of the toolset – in the case of manufacture we’re talking about lathes, planes and so on; in the case of transactions we’re talking about software languages, programming environments, etc – which gives the problem. Of course, this is only a problem when you come to a point that you need to swap out the accounting or invoicing system for a new one.

So – flexiblity of this scale is a problem when we cross boundaries of trust or service. Instead, we should try and aim for a more appropriate model of…

Transactional Machine Tools

Applying the General Motors model to application development, we need to consider:

• Applications are Independent, Standalone – although no program is an island, eh Don?
• Interconnectivity with Independent Services – services share Schema (format), Contract (sequence) and VERSIONED Reference Data.
• Optimistic Concurrency does not stretch across Trust Boundaries – Interactions should be based on business functionality, not data exchanges!

That last point is particularly poignent. Consider, if you will, Optimistic Concurrency and your bank. You can’t just give your bank a set of new values you’d like your balance to be. You have to invoke business operations; Deposit, Withdraw. Incidentally, banks do not do atomic transactions as such. Deposit and Withdraw are examples of potentially long running, cancellable transactions; which brings us to….

Armoury System of Transactions

We can go one better right now. If the US Armoury had been building transactional systems, they would have probably built them to use:

• Compensating Operations
• Tentative Operations
• No Transactions Shared Across Boundaries

Pat then quoted one of his previous discourses. "Two phase commit is the anti-availability protocol." Which makes a lot of sense. 2 Phase Commit locks resources; the more contention, the worse the locking. For services, that should be composable, this is not acceptable.

Key to deriving the same economies of scale with services are Tentative Operations (Cancellability), Composability and Nesting of Cancellation. At all costs, avoid Variance, Option and Choice. These are bad for interchangeability.

A Word about Data

Consider two kinds of data – Activity Data and Resource Data. These data have very different requirements during transactional handling.

Activity data, is what you might find in activity-related operations. Your Amazon shopping basket; deposits to your bank balance. These are things t
hat can be rolled back very easily if anything should go awry. Compensating actions are easy to devise and apply.

Resource data is altogether more complex. Consider booking hotel rooms for your stay during Tech Ed. If the conference were cancelled before the event, each night of your potential stay would have to be compensated. The night of Nov 5th. The night of Nov 6th etc. All get one extra spare hotel room added back to the appropriate bucket. Back to the "Reservation Pool" if you will.

Essential in the handling of resource data – in Reservation Pool Management – is that all access and changes should be contained within, encapsulated in, a suitable Reservation business operation.

To handle these various scenarios, your operations need to be able to cope with

• Long Running Transactions
• The Ability to Place the Same Order Again
• Tentative Operations

In order to achieve these behaviours, remember: Specificity GOOD, Variety BAD.

Domain Specific Languages

DSLs are the heavy duty, multi-purpose machine tools of the General Motors model. Just as GM was able to swap out chassis, transmission, engine parts, DSLs let you swap out logon services, retrieve customer services, accounting services.

DSLs get us close to flexible mass production. Harping back to the World of Warcraft in Visual Studio demo of the Keynote – wasn’t that an example of a Domain Specific Language that churned out Lua code? Retooling made easy, without having to restructure your factory (your IDE) or your craftsmen (your dev team).

Finish on  a Song

OK. He didn’t. Not this time, anyway.

But he did finish with some philosophy.

Atomic Transactions are like Singularities. Singular points in space and time. Two phase commit makes time problems go away. Changes in our industry, moving away from mainframes to minis, from monoliths to services, introduce distance into the equation. Interchangeability of parts makes space and time effects surmountable.

Summary

Services offer us a different paradigm to traditional transactional processing. Always consider:

• Explicit Boundaries – autonomy of services
• Acceptance is Different – Confirm and Cancel, not Commit
• Semantics – Intricacies should be avoided
• Interchangeability – Variance prevents interchangeable operations

TechEd-Developers

Technorati Profile

Technorati Tags: Microsoft, SOA, TechEd 2007