Contents	5
	Preface	17
	Introduction	19
	A User’s Guide to This Book	29
	Trademarks	32
	PART I Basic Distributed Computing Technologies	33
	1 Fundamentals	35
	1.1 Introduction	35
	1.2 Components of a Reliable Distributed Computing System	39
	1.2.1 Communication Technology	44
	1.2.2 Basic Transport and Network Services	46
	1.2.3 Reliable Transport Software and Communication Support	47
	1.2.4 Middleware: Software Tools, Utilities, and Programming Languages	48
	1.2.5 Distributed Computing Environments	49
	1.2.6 End-User Applications	51
	1.3 Critical Dependencies	52
	1.4 Next Steps	54
	1.5 Related Reading	55
	2 Basic Communication Services	57
	2.1 Communication Standards	57
	2.2 Addressing	59
	2.3 Network Address Translation	63
	2.4 IP Tunnelling	65
	2.5 Internet Protocols	65
	2.5.1 Internet Protocol: IP layer	65
	2.5.2 Transmission Control Protocol: TCP	66
	2.5.3 User Datagram Protocol: UDP	66
	2.5.4 Multicast Protocol	67
	2.6 Routing	68
	2.7 End-to-End Argument	69
	2.8 OS Architecture Issues: Buffering and Fragmentation	71
	2.9 Next Steps	73
	2.10 Related Reading	75
	3 High Assurance Communication	77
	3.1 Notions of Correctness and High Assurance Distributed Communication	77
	3.2 The Many Dimensions of Reliability	77
	3.3 Scalability and Performance Goals	81
	3.4 Security Considerations	82
	3.5 Next Steps	83
	3.6 Related Reading	84
	4 Remote Procedure Calls and the Client/ Server Model	85
	4.1 The Client/Server Model	85
	4.2 RPC Protocols and Concepts	89
	4.3 Writing an RPC-based Client or Server Program	92
	4.4 The RPC Binding Problem	95
	4.5 Marshalling and Data Types	97
	4.6 Associated Services	99
	4.6.1 Naming Services	99
	4.6.2 Time Services	101
	4.6.3 Security Services	102
	4.6.4 Threads packages	103
	4.6.5 Transactions	106
	4.7 The RPC Protocol	107
	4.8 Using RPC in Reliable Distributed Systems	110
	4.9 Layering RPC over TCP	113
	4.10 Related Reading	115
	5 Styles of Client/Server Computing	117
	5.1 Stateless and Stateful Client/Server Interactions	117
	5.2 Major Uses of the Client/Server Paradigm	117
	5.3 Distributed File Systems	124
	5.4 Stateful File Servers	131
	5.5 Distributed Database Systems	138
	5.6 Applying Transactions to File Servers	145
	5.7 Message-Queuing Systems	148
	5.8 Related Topics	148
	5.9 Related Reading	149
	6 CORBA: The Common Object Request Broker Architecture	151
	6.1 The ANSA Project	151
	6.2 Beyond ANSA to CORBA	154
	6.3 Web Services	156
	6.4 The CORBA Reference Model	156
	6.5 IDL and ODL	163
	6.6 ORB	164
	6.7 Naming Service	165
	6.8 ENS—The CORBA Event Noti.cation Service	165
	6.9 Life-Cycle Service	167
	6.10 Persistent Object Service	167
	6.11 Transaction Service	167
	6.12 Interobject Broker Protocol	168
	6.13 Properties of CORBA Solutions	168
	6.14 Performance of CORBA and Related Technologies	169
	6.15 Related Reading	172
	7 System Support for Fast Client/Server Communication	173
	7.1 Lightweight RPC	175
	7.2 fbufs and the x-Kernel Project	178
	7.3 Active Messages	179
	7.4 Beyond Active Messages: U-Net	181
	7.5 Protocol Compilation Techniques	186
	7.6 Related Reading	187
	PART II Web Technologies	189
	8 TheWorld WideWeb	191
	8.1 TheWorld WideWeb	191
	8.2 TheWeb Services Vision	193
	8.1 Web Applications	194
	8.3 Web Security and Reliability	196
	8.4 Computing Platforms	201
	8.5 Related Reading	201
	9 MajorWeb Technologies	203
	9.1 Components of theWeb	203
	9.2 HyperText Markup Language	205
	9.3 Extensible Markup Language	206
	9.4 Uniform Resource Locators	206
	9.5 HyperText Transport Protocol	207
	9.6 Representations of Image Data	211
	9.7 Authorization and Privacy Issues	212
	9.8 Web Proxy Servers	219
	9.9 Web Search Engines andWeb Crawlers	220
	9.10 Browser Extensibility Features: Plug-in Technologies	221
	9.11 Future Challenges for theWeb Community	222
	9.12 Consistency and theWeb	224
	9.13 Related Reading	224
	10 Web Services	225
	10.1 What is aWeb Service?	225
	10.2 Web Service Description Language: WSDL	229
	10.3 Simple Object Access Protocol: SOAP	230
	10.4 Talking to aWeb Service: HTTP over TCP	233
	10.5 Universal Description, Discovery and Integration Language: UDDI	234
	10.6 Other Current and ProposedWeb Services Standards	235
	10.6.1 WS- RELIABILITY	235
	10.6.2 WS- TRANSACTIONS	236
	10.6.3 WS- RELIABILITY	238
	10.6.4 WS- MEMBERSHIP	240
	10.7 HowWeb Services Deal with Failure	240
	10.8 The Future ofWeb Services	242
	10.9 Grid Computing: A MajorWeb Services Application	243
	10.10 Autonomic Computing: Technologies to ImproveWeb Services Con . guration Management	244
	10.11 Related Readings	245
	11 Related Internet Technologies	247
	11.1 File Transfer Tools	247
	11.2 Electronic Mail	248
	11.3 Network Bulletin Boards (Newsgroups)	249
	11.4 Instant Messaging Systems	251
	11.5 Message-Oriented Middleware Systems (MOMS)	252
	11.6 Publish-Subscribe and Message Bus Architectures	254
	11.7 Internet Firewalls and Network Address Translators	257
	11.8 Related Reading	258
	12 Platform Technologies	259
	12.1 Microsoft’s .NET Platform	260
	12.1.1 .NET Framework	260
	12.1.2 XMLWeb Services	261
	12.1.3 Language Enhancements	262
	12.1.4 Tools for Developing for Devices	262
	12.1.5 Integrated Development Environment	262
	12.2 Java Enterprise Edition	263
	12.2.1 J2EE Framework	263
	12.2.2 Java Application Veri.cation Kit (AVK)	264
	12.2.3 Enterprise JavaBeans Speci.cation	264
	12.2.4 J2EE Connectors	265
	12.2.5 Web Services	265
	12.2.6 Other Java Platforms	265
	12.3 .NET and J2EE Comparison	265
	12.4 Further Reading	266
	PART III Reliable Distributed Computing	267
	13 How and Why Computer Systems Fail	269
	13.1 Hardware Reliability and Trends	269
	13.2 Software Reliability and Trends	270
	13.3 Other Sources of Downtime	272
	13.4 Complexity	273
	13.5 Detecting Failures	274
	13.6 Hostile Environments	275
	13.7 Related Reading	278
	14 Overcoming Failures in a Distributed System	279
	14.1 Consistent Distributed Behavior	279
	14.1.1 Static Membership	281
	14.1.2 Dynamic Membership	283
	14.2 Formalizing Distributed Problem Speci.cations	285
	14.3 Time in Distributed Systems	286
	14.4 Failure Models and Reliability Goals	293
	14.5 The Distributed Commit Problem	294
	14.5.1 Two-Phase Commit	296
	14.5.2 Three-Phase Commit	303
	14.5.3 Quorum update revisited	306
	14.6 Related Reading	307
	15 Dynamic Membership	309
	15.1 Dynamic Group Membership	309
	15.1.1 GMS and Other System Processes	310
	15.1.2 Protocol Used to Track GMS Membership	314
	15.1.3 GMS Protocol to Handle Client Add and Join Events	316
	15.1.4 GMS Noti.cations With Bounded Delay	318
	15.1.5 Extending the GMS to Allow Partition and Merge Events	320
	15.2 Replicated Data with Malicious Failures	322
	15.3 The Impossibility of Asynchronous Consensus (FLP)	326
	15.3.1 Three-Phase Commit and Consensus	329
	15.4 Extending our Protocol into the Full GMS	332
	15.5 Related Reading	334
	16 Group Communication Systems	335
	16.1 Group Communication	335
	16.2 A Closer Look at Delivery Ordering Options	339
	16.2.1 Nonuniform Failure-Atomic Group Multicast	343
	16.2.2 Dynamically Uniform Failure-Atomic Group Multicast	345
	16.2.3 Dynamic Process Groups	346
	16.2.4 View-Synchronous Failure Atomicity	348
	16.2.5 Summary of GMS Properties	350
	16.2.6 Ordered Multicast	352
	16.3 Communication from Nonmembers to a Group	365
	16.3.1 Scalability	367
	16.4 Communication from a Group to a Nonmember	368
	16.5 Summary of Multicast Properties	368
	16.6 Related Reading	370
	17 Point to Point and Multi-group Considerations	373
	17.1 Causal Communication Outside of a Process Group	374
	17.2 Extending Causal Order to Multigroup Settings	376
	17.3 Extending Total Order to Multigroup Settings	378
	17.4 Causal and Total Ordering Domains	380
	17.5 Multicasts to Multiple Groups	381
	17.6 Multigroup View Management Protocols	381
	17.7 Related Reading	382
	18 The Virtual Synchrony Execution Model	383
	18.1 Virtual Synchrony	383
	18.2 Extended Virtual Synchrony	388
	18.3 Virtually Synchronous Algorithms and Tools	394
	18.3.1 Replicated Data and Synchronization	394
	18.3.2 State Transfer to a Joining Process	399
	18.3.3 Load-Balancing	401
	18.3.4 Primary-Backup Fault Tolerance	403
	18.3.5 Coordinator-Cohort Fault Tolerance	404
	18.4 Related Reading	406
	19 Consistency in Distributed Systems	407
	19.1 Consistency in the Static and Dynamic Membership Models	408
	19.2 Practical Options for Coping with Total Failure	416
	19.3 General Remarks Concerning Causal andTotal Ordering	417
	19.4 Summary and Conclusion	421
	19.5 Related Reading	422
	PART IV Applications of Reliability Techniques	423
	20 Retrofitting Reliability into Complex Systems	425
	20.1 Wrappers and Toolkits	426
	20.1.1 Wrapper Technologies	428
	20.1.2 Introducing Robustness in Wrapped Applications	434
	20.1.3 Toolkit Technologies	437
	20.1.4 Distributed Programming Languages	439
	20.2 Wrapping a Simple RPC server	440
	20.3 Wrapping aWeb Site	442
	20.4 Hardening Other Aspects of theWeb	443
	20.5 Unbreakable Stream Connections	447
	20.5.1 Reliability Options for Stream Communication	448
	20.5.2 An Unbreakable Stream That Mimics TCP	449
	20.5.3 Nondeterminism and Its Consequences	451
	20.5.4 Dealing with Arbitrary Nondeterminism	452
	20.5.5 Replicating the IP Address	452
	20.5.6 Maximizing Concurrency by Relaxing Multicast Ordering	453
	20.5.7 State Transfer Issues	456
	20.5.8 Discussion	456
	20.6 Reliable Distributed Shared Memory	457
	20.6.1 The Shared Memory Wrapper Abstraction	458
	20.6.2 Memory Coherency Options for Distributed Shared Memory	460
	20.6.3 False Sharing	463
	20.6.4 Demand Paging and Intelligent Prefetching	463
	20.6.5 Fault Tolerance Issues	464
	20.6.6 Security and Protection Considerations	465
	20.6.7 Summary and Discussion	465
	20.7 Related Reading	466
	21 Software Architectures for Group Communication	467
	21.1 Architectural Considerations in Reliable Systems	468
	21.2 Horus: A Flexible Group Communication System	471
	21.2.1 A Layered Process Group Architecture	472
	21.3 Protocol stacks	475
	21.4 Using Horus to Build a Publish-Subscribe Platform and a Robust Groupware Application	477
	21.5 Using Electra to Harden CORBA Applications	480
	21.6 Basic Performance of Horus	482
	21.7 Masking the Overhead of Protocol Layering	486
	21.7.1 Reducing Header Overhead	487
	21.7.2 Eliminating Layered Protocol Processing Overhead	489
	21.7.3 Message Packing	490
	21.7.4 Performance of Horus with the Protocol Accelerator	490
	21.8 Scalability	491
	21.9 Performance and Scalability of the Spread Toolkit	493
	21.10 Related Reading	496
	PART V Related Technologies	497
	22 Security Options for Distributed Settings	499
	22.1 Security Options for Distributed Settings	499
	22.2 Perimeter Defense Technologies	503
	22.3 Access Control Technologies	507
	22.4 Authentication Schemes, Kerberos, and SSL	509
	22.4.1 RSA and DES	510
	22.4.2 Kerberos	512
	22.4.3 ONC Security and NFS	515
	22.4.4 SSL Security	516
	22.5 Security Policy Languages	519
	22.6 On-The-Fly Security	521
	22.7 Availability and Security	522
	22.8 Related Reading	524
	23 Clock Synchronization and Synchronous Systems	525
	23.1 Clock Synchronization	525
	23.2 Timed-Asynchronous Protocols	530
	23.3 Adapting Virtual Synchrony for Real-Time Settings	537
	23.4 Related Reading	540
	24 Transactional Systems	541
	24.1 Review of the Transactional Model	541
	24.2 Implementation of a Transactional Storage System	543
	24.2.1 Write-Ahead Logging	543
	24.2.2 Persistent Data Seen Through an Updates List	544
	24.2.3 Nondistributed Commit Actions	545
	24.3 Distributed Transactions and Multiphase Commit	546
	24.4 Transactions on Replicated Data	547
	24.5 Nested Transactions	548
	24.5.1 Comments on the Nested Transaction Model	550
	24.6 Weak Consistency Models	553
	24.6.1 Epsilon Serializability	554
	24.6.2 Weak and Strong Consistency in Partitioned Database Systems	555
	24.6.3 Transactions on Multidatabase Systems	556
	24.6.4 Linearizability	556
	24.6.5 Transactions in Real-Time Systems	557
	24.7 Advanced Replication Techniques	557
	24.8 Related Reading	560
	25 Peer-to-Peer Systems and Probabilistic Protocols	561
	25.1 Peer-to-Peer File Sharing	563
	25.1.1 Napster	564
	25.1.2 Gnutella and Kazaa	565
	25.1.3 CAN	566
	25.1.4 CFS on Chord and PAST on Pastry	568
	25.1.5 OceanStore	568
	25.2 Peer-to-Peer Distributed Indexing	569
	25.2.1 Chord	570
	25.2.2 Pastry	573
	25.2.3 Tapestry and Brocade	575
	25.2.4 Kelips	575
	25.3 Bimodal Multicast Protocol	578
	25.3.1 Bimodal Multicast	581
	25.3.2 Unordered pbcast Protocol	583
	25.3.3 Adding CASD-style Temporal Properties and Total Ordering	584
	25.3.4 Scalable Virtual Synchrony Layered Over Pbcast	586
	25.3.5 Probabilistic Reliability and the Bimodal Delivery Distribution	586
	25.3.6 Evaluation and Scalability	589
	25.3.7 Experimental Results	590
	25.4 Astrolabe	590
	25.4.1 How it works	592
	25.4.2 Peer-to-Peer Data Fusion and Data Mining	596
	25.5 Other Applications of Peer-to-Peer Protocols	600
	25.6 Related Reading	600
	26 Prospects for Building Highly Assured Web Services	603
	26.1 Web Services and Their Assurance Properties	603
	26.2 High Assurance for Back-End Servers	610
	26.3 High Assurance forWeb Server Front-Ends	614
	26.4 Issues Encountered on the Client Side	615
	26.5 Highly AssuredWeb Services Need Autonomic Tools!	616
	26.6 Summary	619
	26.7 Related Reading	620
	27 Other Distributed and Transactional Systems	621
	27.1 RelatedWork in Distributed Computing	621
	27.1.1 Amoeba	622
	27.1.2 BASE	622
	27.1.3 Chorus	622
	27.1.4 Delta-4	623
	27.1.5 Ensemble	623
	27.1.6 Harp	623
	27.1.7 The Highly Available System (HAS)	624
	27.1.8 The Horus System	625
	27.1.9 The Isis Toolkit	625
	27.1.10 Locus	626
	27.1.11 Manetho	626
	27.1.12 NavTech	627
	27.1.13 Paxos	627
	27.1.14 Phalanx	628
	27.1.15 Phoenix	628
	27.1.16 Psync	628
	27.1.17 Rampart	629
	27.1.18 Relacs	629
	27.1.19 RMP	629
	27.1.20 Spread	630
	27.1.21 StormCast	630
	27.1.22 Totem	631
	27.1.23 Transis	632
	27.1.24 The V System	632
	27.2 Peer-to-Peer Systems	633
	27.2.1 Astrolabe	633
	27.2.2 Bimodal Multicast	634
	27.2.3 Chord/CFS	634
	27.2.4 Gnutella/Kazaa	634
	27.2.5 Kelips	634
	27.2.6 Pastry/PAST and Scribe	634
	27.2.7 QuickSilver	635
	27.2.8 Tapestry/Brocade	635
	27.3 Systems That Implement Transactions	635
	27.3.1 Argus	635
	27.3.2 Arjuna	636
	27.3.3 Avalon	636
	27.3.4 Bayou	637
	27.3.5 Camelot and Encina	637
	27.3.6 Thor	638
	Appendix: Problems	639
	Bibliography	661
	Index	693
	More eBooks at www.ciando.com	0