C# 4.0编程(第6版) Programming C#4.0

Programming C# 4.0 SIXTH EDITION Programming C# 4.0 Ian Griffiths, Matthew Adams, and Jesse Liberty Beijing • Cambridge • Farnham • Köln • Sebastopol • Taipei • Tokyo Programming C# 4.0, Sixth Edition by Ian Griffiths, Matthew Adams, and Jesse Liberty Copyright © 2010 Ian Griffiths and Matthew Adams. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Editors: Mike Hendrickson and Laurel Ruma Production Editor: Adam Zaremba Copyeditor: Audrey Doyle Proofreader: Stacie Arellano Indexer: Jay Marchand Cover Designer: Karen Montgomery Interior Designer: David Futato Illustrator: Robert Romano Printing History: July 2001: First Edition. February 2002: Second Edition. May 2003: Third Edition. February 2005: Fourth Edition. December 2007: Fifth Edition. August 2010: Sixth Edition. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Programming C# 4.0, the image of an African crowned crane, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information con- tained herein. ISBN: 978-0-596-15983-2 [M] 1280338225 Table of Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv 1. Introducing C# .......................................................... 1 Why C#? Why .NET? 1 The .NET Framework Class Library 2 Language Style 3 Composability 4 Managed Code 5 Continuity and the Windows Ecosystem 6 C# 4.0, .NET 4, and Visual Studio 2010 7 Summary 9 2. Basic Programming Techniques . . . ....................................... 11 Getting Started 11 Namespaces and Types 14 Projects and Solutions 19 Comments, Regions, and Readability 24 Bad Comments 26 XML Documentation Comments 26 Variables 28 Variable Types 28 Expressions and Statements 35 Assignment Statements 38 Increment and Decrement Operators 38 Flow Control with Selection Statements 39 if Statements 40 switch and case Statements 45 Iteration Statements 47 foreach Statements 48 for Statements 50 while and do Statements 52 v Breaking Out of a Loop 53 Methods 55 Summary 58 3. Abstracting Ideas with Classes and Structs ................................. 59 Divide and Conquer 59 Abstracting Ideas with Methods 59 Abstracting Ideas with Objects and Classes 62 Defining Classes 64 Representing State with Properties 64 Protection Levels 66 Initializing with a Constructor 68 Fields: A Place to Put Data 72 Fields Can Be Fickle, but const Is Forever 75 Read-only Fields and Properties 76 Related Constants with enum 79 Value Types and Reference Types 82 Too Many Constructors, Mr. Mozart 88 Overloading 88 Overloaded Methods and Default Named Parameters 89 Object Initializers 92 Defining Methods 95 Declaring Static Methods 98 Static Fields and Properties 99 Static Constructors 101 Summary 102 4. Extensibility and Polymorphism . . . ...................................... 103 Association Through Composition and Aggregation 104 Inheritance and Polymorphism 106 Replacing Methods in Derived Classes 109 Hiding Base Members with new 109 Replacing Methods with virtual and override 112 Inheritance and Protection 114 Calling Base Class Methods 116 Thus Far and No Farther: sealed 118 Requiring Overrides with abstract 121 All Types Are Derived from Object 127 Boxing and Unboxing Value Types 127 C# Does Not Support Multiple Inheritance of Implementation 132 C# Supports Multiple Inheritance of Interface 132 Deriving Interfaces from Other Interfaces 135 Explicit Interface Implementation 136 vi | Table of Contents The Last Resort: Checking Types at Runtime 141 Summary 142 5. Composability and Extensibility with Delegates .... . . . . . . . . . . . . . . . . . . . . . . . . 143 Functional Composition with delegate 150 Generic Actions with Action 156 Generic Predicates with Predicate 160 Using Anonymous Methods 162 Creating Delegates with Lambda Expressions 163 Delegates in Properties 165 Generic Delegates for Functions 167 Notifying Clients with Events 171 Exposing Large Numbers of Events 180 Summary 183 6. Dealing with Errors . . . ................................................. 185 When and How to Fail 191 Returning Error Values 194 Debugging with Return Values 200 Exceptions 201 Handling Exceptions 207 When Do finally Blocks Run? 214 Deciding What to Catch 215 Custom Exceptions 218 Summary 220 7. Arrays and Lists . . . .................................................... 221 Arrays 221 Construction and Initialization 222 Custom Types in Arrays 225 Array Members 230 Array Size 236 List 243 Custom Indexers 247 Finding and Sorting 253 Collections and Polymorphism 254 Creating Your Own IEnumerable 258 Summary 264 8. LINQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 Query Expressions 265 Query Expressions Versus Method Calls 267 Extension Methods and LINQ 268 Table of Contents | vii let Clauses 271 LINQ Concepts and Techniques 271 Delegates and Lambdas 271 Functional Style and Composition 273 Deferred Execution 274 LINQ Operators 275 Filtering 275 Ordering 276 Concatenation 279 Grouping 280 Projections 282 Zipping 288 Getting Selective 289 Testing the Whole Collection 291 Aggregation 292 Set Operations 294 Joining 295 Conversions 296 Summary 297 9. Collection Classes ... .................................................. 299 Dictionaries 299 Common Dictionary Uses 301 IDictionary 308 Dictionaries and LINQ 309 HashSet and SortedSet 310 Queues 311 Linked Lists 312 Stacks 313 Summary 314 10. Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 What Is a String? 316 The String and Char Types 317 Literal Strings and Chars 318 Escaping Special Characters 319 Formatting Data for Output 322 Standard Numeric Format Strings 323 Custom Numeric Format Strings 329 Dates and Times 332 Going the Other Way: Converting Strings to Other Types 336 Composite Formatting with String.Format 337 Culture Sensitivity 338 viii | Table of Contents Exploring Formatting Rules 340 Accessing Characters by Index 341 Strings Are Immutable 341 Getting a Range of Characters 343 Composing Strings 344 Splitting It Up Again 346 Upper- and Lowercase 347 Manipulating Text 348 Mutable Strings with StringBuilder 349 Finding and Replacing Content 353 All Sorts of “Empty” Strings 355 Trimming Whitespace 357 Checking Character Types 360 Encoding Characters 360 Why Encodings Matter 362 Encoding and Decoding 363 Why Represent Strings As Byte Sequences? 370 Summary 370 11. Files and Streams ... .................................................. 371 Inspecting Directories and Files 371 Examining Directories 374 Manipulating File Paths 375 Path and the Current Working Directory 376 Examining File Information 377 Creating Temporary Files 381 Deleting Files 381 Well-Known Folders 383 Concatenating Path Elements Safely 387 Creating and Securing Directory Hierarchies 388 Deleting a Directory 394 Writing Text Files 396 Writing a Whole Text File at Once 396 Writing Text with a StreamWriter 397 When Files Go Bad: Dealing with Exceptions 400 Finding and Modifying Permissions 404 Reading Files into Memory 409 Streams 413 Moving Around in a Stream 419 Writing Data with Streams 421 Reading, Writing, and Locking Files 422 FileStream Constructors 423 Stream Buffers 423 Table of Contents | ix Setting Permissions During Construction 424 Setting Advanced Options 425 Asynchronous File Operations 425 Isolated Storage 428 Stores 429 Reading and Writing Text 430 Defining “Isolated” 431 Managing User Storage with Quotas 436 Managing Isolated Storage 436 Streams That Aren’t Files 439 An Adapting Stream: CryptoStream 443 In Memory Alone: The MemoryStream 444 Representing Binary As Text with Base64 Encoding 444 Summary 447 12. XML ................................................................ 449 XML Basics (A Quick Review) 449 Elements 450 XHTML 451 X Stands for eXtensible 452 Creating XML Documents 452 XML Elements 455 XML Attributes 456 Putting the LINQ in LINQ to XML 459 Searching in XML with LINQ 461 Searching for a Single Node 465 Search Axes 466 Where Clauses 466 XML Serialization 467 Customizing XML Serialization Using Attributes 469 Summary 471 13. Networking .......................................................... 473 Choosing a Networking Technology 473 Web Application with Client-Side Code 474 .NET Client and .NET Server 477 .NET Client and External Party Web Service 479 External Client and .NET Web Service 480 WCF 481 Creating a WCF Project 481 WCF Contracts 482 WCF Test Client and Host 483 Hosting a WCF Service 486 x | Table of Contents Writing a WCF Client 493 Bidirectional Communication with Duplex Contracts 501 HTTP 511 WebClient 512 WebRequest and WebResponse 516 Sockets 522 IP, IPv6, and TCP 523 Connecting to Services with the Socket Class 528 Implementing Services with the Socket Class 531 Other Networking Features 536 Summary 537 14. Databases ........................................................... 539 The .NET Data Access Landscape 539 Classic ADO.NET 540 LINQ and Databases 544 Non-Microsoft Data Access Technologies 545 WCF Data Services 546 Silverlight and Data Access 546 Databases 547 The Entity Data Model 548 Generated Code 551 Changing the Mapping 554 Relationships 555 Inheritance 562 Queries 563 LINQ to Entities 563 Entity SQL 568 Mixing ESQL and LINQ 570 The EntityClient ADO.NET Provider 571 Object Context 571 Connection Handling 571 Creating, Updating, and Deleting 574 Transactions 576 Optimistic Concurrency 581 Context and Entity Lifetime 583 WCF Data Services 584 Summary 588 15. Assemblies .......................................................... 589 .NET Components: Assemblies 589 References 590 Writing Libraries 593 Table of Contents | xi Protection 595 Naming 598 Signing and Strong Names 599 Loading 601 Loading from the Application Folder 602 Loading from the GAC 603 Loading from a Silverlight .xap File 603 Explicit Loading 604 Summary 605 16. Threads and Asynchronous Code ......................................... 607 Threads 609 Threads and the OS Scheduler 611 The Stack 613 The Thread Pool 620 Thread Affinity and Context 622 Common Thread Misconceptions 623 Multithreaded Coding Is Hard 629 Multithreading Survival Strategies 632 Synchronization Primitives 634 Monitor 634 Other Lock Types 645 Other Coordination Mechanisms 649 Events 649 Countdown 650 BlockingCollection 650 Asynchronous Programming 651 The Asynchronous Programming Model 652 The Event-Based Asynchronous Pattern 655 Ad Hoc Asynchrony 656 The Task Parallel Library 656 Tasks 657 Cancellation 663 Error Handling 665 Data Parallelism 666 Parallel For and ForEach 667 PLINQ: Parallel LINQ 669 Summary 670 17. Attributes and Reflection . . . ........................................... 671 Attributes 671 Types of Attributes 672 Custom Attributes 673 xii | Table of Contents Reflection 677 Inspecting Metadata 678 Type Discovery 679 Reflecting on a Specific Type 681 Late Binding 683 Summary 686 18. Dynamic ............................................................. 687 Static Versus Dynamic 687 The Dynamic Style and COM Automation 689 The dynamic Type 690 Object Types and dynamic 693 dynamic in Noninterop Scenarios? 703 Summary 706 19. Interop with COM and Win32 ........................................... 707 Importing ActiveX Controls 707 Importing a Control in .NET 708 Interop Assemblies 711 No PIA 712 64-bit Versus 32-bit 713 P/Invoke 716 Pointers 720 C# 4.0 Interop Syntax Enhancements 725 Indexed Properties 725 Optional ref 726 Summary 727 20. WPF and Silverlight . . . ................................................ 729 Xaml and Code Behind 731 Xaml and Objects 735 Elements and Controls 738 Layout Panels 739 Graphical Elements 748 Controls 755 User Controls 760 Control Templates 761 Styles 764 The Visual State Manager 766 Data Binding 767 Data Templates 769 Summary 773 Table of Contents | xiii 21. Programming ASP.NET Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 775 Web Forms Fundamentals 775 Web Forms Events 776 Web Forms Life Cycle 778 Creating a Web Application 779 Code-Behind Files 780 Adding Controls 781 Server Controls 783 Data Binding 784 Examining the Code 789 Adding Controls and Events 790 Summary 794 22. Windows Forms . . . ................................................... 795 Creating the Application 796 Adding a Binding Source 797 Controls 800 Docking and Anchoring 805 Data Binding 806 Event Handling 811 Summary 813 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 815 xiv | Table of Contents Preface Microsoft unveiled the .NET Framework in 2000, and in the decade that followed, it became an extremely popular choice for developing software for Windows. While .NET supports many programming languages, it is most strongly associated with the language designed specifically for the platform: C#. C# has grown considerably since its launch. Each new version enabled new program- ming techniques—C# 2.0 added generics and enhanced functional programming ca- pabilities, then integrated query features and yet more powerful functional capabilities arrived in C# 3.0, and now C# 4.0 adds new dynamic language capabilities. The .NET Framework has grown with the language. Back in .NET 1.0, the class libraries offered relatively patchy coverage of the underlying Windows capabilities. Moreover, the library features that were unique to .NET, rather than being wrappers for something else, were relatively modest. Now, as well as more comprehensive platform coverage we have a GUI framework (WPF), much stronger database capabilities, powerful sup- port for concurrent execution, and an extensive set of communication services (WCF), to name just a few of the available features. And the features that have been there since version 1.0, such as web support (ASP.NET), have been fleshed out substantially. .NET is no longer limited to running just on Windows. Some people recognized its potential for platform independence early on, but for years, Microsoft supported C# just on Windows, leaving open source projects to offer the only way to run C# on other systems. But in 2008, the release of Silverlight 2 saw C# code running with Microsoft’s full support on non-Windows platforms such as the Mac for the first time. The C# language has come a long way since 2000, in both reach and size. Our goal with Programming C# 4.0 is to show how to use C#. How This Book Is Organized The book begins by looking at the details of the C# language that you will use in everyday programming. We then look at the most common parts of the .NET Frame- work class library that you will also use very regularly. Next, we move into some more xv specialized areas of the framework. Finally, we look at some of the application frame- works for building Windows and web applications in .NET. Chapter 1, Introducing C# This chapter talks about the nature of C# and its relationship with the .NET Framework. Chapter 2, Basic Programming Techniques In this chapter, we show the core elements of C# code—the steps required to get up and running, and fundamental features such as variables, flow control, loops, and methods. Chapter 3, Abstracting Ideas with Classes and Structs C# supports object-oriented programming, and this chapter describes the lan- guage features dedicated to these techniques. Chapter 4, Extensibility and Polymorphism This chapter continues the discussion from the preceding chapter, illustrating how C# supports inheritance, interfaces, and related concepts. Chapter 5, Composability and Extensibility with Delegates C# isn’t limited to object-oriented programming—it also supports some very powerful functional programming idioms. This chapter shows how these can sometimes be more flexible and also simpler than OO techniques. Chapter 6, Dealing with Errors All programs encounter failures, whether due to programming errors, unexpected input, network failures, or a host of other eventualities. This chapter shows the options for detecting and responding robustly to errors. Chapter 7, Arrays and Lists This chapter shows the tools C# offers for representing simple collections of information. Chapter 8, LINQ It’s not enough merely to be able to represent collections, so this chapter shows how you can use the integrated query features in C# to process your collections of data. Chapter 9, Collection Classes This chapter shows some of the more specialized classes for working with collec- tions in particular ways. Chapter 10, Strings Text is a particularly important data type for most applications, so this chapter shows how text is represented, and how you can format data into textual form. Chapter 11, Files and Streams This chapter shows how to store information on disk and read it back in, and how to perform other filesystem operations. It also shows how some of the abstractions used when working with files can be applied in other scenarios. xvi | Preface Chapter 12, XML This chapter shows the classes offered by the .NET Framework for processing XML, and how these can work in conjunction with the LINQ features in C#. Chapter 13, Networking In this chapter, we look at the various techniques for communicating over a network. Chapter 14, Databases This chapter shows how to access a database from C#. Chapter 15, Assemblies In this chapter, we show how to compile code into libraries for reuse, and how programs made up from multiple components work. Chapter 16, Threads and Asynchronous Code Many programs need to deal with concurrency, and this chapter shows the tools and techniques available. Chapter 17, Attributes and Reflection C# has the ability to inspect the structure of code, which makes it easier to auto- mate certain kinds of tasks. This chapter shows the API for doing this, and how you can extend the structural information through attributes. Chapter 18, Dynamic One of the new features in C# 4.0 is support for dynamic binding. This is partic- ularly useful in certain interop scenarios, as we discuss in this chapter. Chapter 19, Interop with COM and Win32 Sometimes it’s necessary for C# code to communicate with components not de- signed to be used from .NET. This chapter shows how to do this with both COM components and Win32-style DLLs. Chapter 20, WPF and Silverlight WPF and Silverlight offer very similar programming models for building user in- terfaces. This chapter shows how to use that model from C#. Chapter 21, Programming ASP.NET Applications This chapter shows how to use ASP.NET, the part of the .NET Framework de- signed for building web applications. Chapter 22, Windows Forms This chapter shows how to use Windows Forms, which is a wrapper around the classic Windows user interface mechanisms. While it is less flexible than WPF, it can offer an easier way to integrate with old components such as ActiveX controls. Where to Find Features New in C# 4.0 and .NET 4 Although this book is written to be read as a whole, we expect that some readers will want to look for the features new to C# 4.0, and also to .NET 4. Since our goal is to show how the C# language is used today, we have avoided structuring the book around Preface | xvii the history of the language, because you will use language features of varying ages in combination. As it happens, one of the new features in C# 4.0 serves a very specific purpose, so it gets its own chapter, but for the most part, new language features are spread throughout the book, because we aim to mention them where you need to know about them. We cannot point you at a particular set of chapters, so instead, here’s a quick guide to where we discuss these features. Chapter 1 talks about the broad goals behind the new features in C# 4.0. Chapter 3 shows the use of default values and named arguments (and these come up again very briefly in Chapters 11 and 17). Chapter 7 describes variance, a rather technical feature of the type system that has some useful implications for collection types. Chapter 16 talks about the extensive new multithreading support added in .NET 4. Chapter 18 is dedicated entirely to a new language feature: support for dynamic programming. Chapter 19 describes the new no-PIA feature, and some features that allow more elegant code in some interop scenarios. Who This Book Is For If you have some basic knowledge of C# but want to brush up your skills, or if you are proficient in another programming language such as C++ or Java, or even if C# is your first programming language, this book is for you. What You Need to Use This Book To make the best use of this book, please obtain the latest release of Visual Studio 2010. Any edition will do, including the free Express edition for C#, which can be downloaded from http://www.microsoft.com/express/. For Chapter 14 you will need a copy of SQL Server or SQL Server Express. Some editions of Visual Studio will install SQL Server Express for you by default, so you may already have this. The example source code for this book is available through the O’Reilly site at http:// oreilly.com/catalog/9780596159832/. Conventions Used in This Book The following font conventions are used in this book: Italic is used for: • Pathnames, filenames, and program names • Internet addresses, such as domain names and URLs • New terms where they are defined xviii | Preface Constant Width is used for: • Command lines and options that should be typed verbatim • Names and keywords in program examples, including method names, variable names, and class names Constant Width Italic is used for: • Replaceable items, such as variables or optional elements, within syntax lines or code Constant Width Bold is used for: • Emphasis within program code Pay special attention to notes set apart from the text with the following icons: This is a tip. It contains useful supplementary information about the topic at hand. This is a warning. It helps you solve and avoid annoying problems. Using Code Examples This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: "Programming C# 4.0, Sixth Edition, by Ian Griffiths, Matthew Adams, and Jesse Liberty. Copyright 2010 Ian Griffiths and Matthew Adams, 978-0-596-15983-2.” Preface | xix Safari® Books Online Safari Books Online is an on-demand digital library that lets you easily search over 7,500 technology and creative reference books and videos to find the answers you need quickly. With a subscription, you can read any page and watch any video from our library online. Read books on your cell phone and mobile devices. Access new titles before they are available for print, and get exclusive access to manuscripts in development and post feedback for the authors. Copy and paste code samples, organize your favorites, down- load chapters, bookmark key sections, create notes, print out pages, and benefit from tons of other time-saving features. O’Reilly Media has uploaded this book to the Safari Books Online service. To have full digital access to this book and others on similar topics from O’Reilly and other pub- lishers, sign up for free at http://my.safaribooksonline.com/?portal=oreilly. Acknowledgments From Ian Griffiths I want to thank the technical reviewers, whose feedback helped to improve this book: Nicholas Paldino, Chris Smith, Chris Williams, Michael Eaton, Brian Peek, and Stephen Toub. Everyone at O’Reilly has provided a great deal of support and patience throughout the project, so many thanks to Mike Hendrickson, Laurel Ruma, Audrey Doyle, and Sumita Mukherji. Thanks also to John Osborn for getting things started in the early days of this project, and for getting Matthew and me on board as O’Reilly authors in the first place, all those years ago. Thank you to my coauthor for not learning his lesson from the last book and agreeing to write another with me. And finally, thank you to Jesse Liberty for asking us to take over his book. From Matthew Adams I’d like to add my thanks to those of my coauthor to all those at O’Reilly whose patience, help, and support have made this book possible, and to all our reviewers whose feed- back has been invaluable. In addition, I’d like to add a nod to Karolina Lemiesz, coffee wizard at the Starbucks where most of my text was written, for the constant supply of ristretto, and an education in coffee tasting when work got too much. xx | Preface As always, my partner Una provided the necessary foundation of love and support (despite her own book deadlines). And finally, anyone who tells you that squeezing a book out of an author is a breeze is clearly deluded, but my coauthor makes it look easy. My thanks go to him especially for his forbearance, wit, and friendship. And good dinners. Preface | xxi CHAPTER 1 Introducing C# C#—pronounced “See Sharp”—is a programming language designed for Micro- soft’s .NET platform. Since its first release in 2002, C# has found many roles. It is widely used on the server side of websites, and also on both the client and server in line-of-business Windows desktop applications. You can write smartphone user inter- faces and Xbox 360 games in C#. More recently, Microsoft’s Silverlight platform has made C# an option for writing Rich Internet Applications that run in a web browser. But what kind of language is C#? To understand a language well enough to use it effectively, it’s not enough to focus purely on the details and mechanisms, although we’ll be spending plenty of time on those in this book. It is equally important to un- derstand the thinking behind the details. So in this chapter, we’ll look at what problems C# was built to solve. Then we’ll explore the style of the language, through aspects that distinguish it from other languages. And we’ll finish the chapter with a look at the latest step in the evolution of C#, its fourth version. Why C#? Why .NET? Programming languages exist to help developers be more productive. Many successful languages simplify or automate tedious tasks that previously had to be done by hand. Some offer new techniques that allow old problems to be tackled more effectively, or on a larger scale than before. How much difference C# can make to you will depend on your programming background, of course, so it’s worth considering what sorts of people the language designers had in mind when they created C#. C# is aimed at developers working on the Windows platform, and its syntax is instantly familiar to users of C or C++, or other languages that draw from the same tradition, such as JavaScript and Java. Fundamental language elements such as statements, ex- pressions, function declarations, and flow control are modeled as closely as possible on their equivalents in C family languages. 1 A familiar syntax is not enough of a reason to pick a language, of course, so C# offers productivity-enhancing features not found in some of its predecessors. Garbage col- lection frees developers from the tyranny of common memory management problems such as memory leaks and circular references. Verifiable type safety of compiled code rules out a wide range of bugs and potential security flaws. While C or C++ Windows developers may not be accustomed to those features, they will seem old hat to Java veterans, but Java has nothing to compete with the “LINQ” features C# offers for working with collections of information, whether in object models, XML documents, or databases. Integrating code from external components is remarkably painless, even those written in other languages. C# also incorporates support for functional pro- gramming, a powerful feature previously most commonly seen in academic languages. Many of the most useful features available to C# developers come from the .NET Framework, which provides the runtime environment and libraries for C#, and all other .NET languages, such as VB.NET. C# was designed for .NET, and one of the main benefits of its close relationship with the .NET Framework is that working with framework features such as the class library feels very natural. The .NET Framework Class Library Working in C# means more than using just the language—the classes offered by the .NET Framework are an extremely important part of the C# developer’s everyday experience (and they account for a lot of this book’s content). Most of the library func- tionality falls into one of three categories: utility features written in .NET, wrappers around Windows functionality, and frameworks. The first group comprises utility types such as dictionaries, lists, and other collection classes, as well as string manipulation facilities such as a regular expression engine. There are also features that operate on a slightly larger scale, such as the object models for representing XML documents. Some library features are wrappers around underlying OS functionality. For example, there are classes for accessing the filesystem, and for using network features such as sockets. And there are classes for writing output to the console, which we can illustrate with the obligatory first example of any programming language book, shown in Example 1-1. Example 1-1. The inevitable “Hello, world” example class Program { static void Main() { System.Console.WriteLine("Hello, world"); } } 2 | Chapter 1: Introducing C# We’ll examine all the pieces shown here in due course, but for now, note that even this simplest of examples depends on a class from the library—the System.Console class in this case—to do its job. Finally, the class library offers whole frameworks to support building certain kinds of applications. For example, Windows Presentation Foundation (WPF) is a framework for building Windows desktop software; ASP.NET (which is not an acronym, despite appearances) is a framework for building web applications. Not all frameworks are about user interfaces—Windows Communication Foundation (WCF) is designed for building services accessed over the network by other computer systems, for instance. These three categories are not strict, as quite a few classes fit into two. For example, the parts of the class library that provide access to the filesystem are not just thin wrap- pers around existing Win32 APIs. They add new object-oriented abstractions, provid- ing significant functionality beyond the basic file I/O services, so these types fit into both the first and second categories. Likewise, frameworks usually need to integrate with underlying services to some extent—for example, although the Windows Forms UI framework has a distinctive API of its own, a lot of the underlying functionality is provided by Win32 components. So the three categories here are not strict. They just offer a useful idea of what sorts of things you can find in the class libraries. Language Style C# is not the only language that runs on the .NET Framework. Indeed, support for multiple languages has always been a key feature of .NET, reflected in the name of its runtime engine, the CLR or Common Language Runtime. As this name implies, .NET is not just for one language—numerous languages have access to the services of the .NET Framework class library. Why might you choose C# over the others? We already mentioned one important reason: C# was designed specifically for .NET. If you are working with .NET technologies such as WPF or ASP.NET, you’ll be speaking their language if you work in C#. Compare this with C++, which supports .NET through extensions to the original language. The extensions are carefully thought out and work well, but code that uses .NET libraries just looks different from normal C++, so programs that bridge the worlds of .NET and standard C++ never feel com- pletely coherent. And the dual personality often presents dilemmas—should you use standard C++ collection classes or the ones in the .NET class library, for example? In native .NET languages such as C#, such questions do not emerge. But C# is not unique in this respect. Visual Studio 2010 ships with three languages designed for .NET: C#, VB.NET, and F#. (Although VB.NET follows on from its non-.NET Visual Basic predecessors, it was radically different in some important ways. It is a native .NET language with a VB-like syntax rather than VB 6 with .NET capa- bilities bolted on.) The choice between these languages comes down to what style of language you prefer. Language Style | 3 F# is the odd one out here. It’s a functional programming language, heavily influenced by a language called ML. Back in 1991, when your authors were first-year students, our university’s computer science course chose ML for the first programming language lectures in part because it was so academic that none of the students would previously have come across anything like it. F# is still at the academic end of the spectrum despite having climbed far enough down the ivory tower to be a standard part of a mainstream development environment. It excels at complicated calculations and algorithms, and has some characteristics that can help with parallel execution. However, as with many functional languages, the cost of making some hard problems easier is that a lot of things that are easy in more traditional languages are remarkably hard in F#— functional languages are adept at complex problems, but can be clumsy with simple ones. It seems likely that F# will mostly be used in scientific or financial applications where the complexity of the computation to be performed dwarfs the complexity of the code that needs to act on the results of those calculations. While F# feels distinctly other, VB.NET and C# have a lot of similarities. The most obvious factor in choosing between these is that VB.NET is easier to learn for someone familiar with Visual Basic syntax, while C# will be easier for someone familiar with a C-like language. However, there is a subtler difference in language philosophy that goes beyond the syntax. Composability A consistent theme in the design of the C# programming language is that its creators tend to prefer general-purpose features over specialized ones. The most obvious ex- ample of this is LINQ, the Language INtegrated Query feature added in C# 3.0. Su- perficially, this appears to add SQL-like query features to the language, providing a natural way to integrate database access into your code. Example 1-2 shows a simple query. Example 1-2. Data access with LINQ var californianAuthors = from author in pubs.authors where author.state == "CA" select new { author.au_fname, author.au_lname }; foreach (var author in californianAuthors) { Console.WriteLine(author); } Despite appearances, C# doesn’t know anything about SQL or databases. To enable this syntax, C# 3.0 added a raft of language features which, in combination, allow code of this sort to be used not just for database access, but also for XML parsing, or working 4 | Chapter 1: Introducing C# with object models. Moreover, many of the individual features can be used in other contexts, as we’ll see in later chapters. C# prefers small, composable, general-purpose features over monolithic, specialized ones. A striking example of this philosophy is a feature that was demonstrated in prototype form in C#, but which eventually got left out: XML literals. This experimental syntax allowed inline XML, which compiled into code that built an object model representing that XML. The C# team’s decision to omit this feature illustrates a stylistic preference for generality over highly specialized features—while the LINQ syntax has many ap- plications, XML literal syntax cannot be used for anything other than XML, and this degree of specialization would feel out of place in C#.* Managed Code The .NET Framework provides more than just a class library. It also provides services in subtler ways that are not accessed explicitly through library calls. For example, earlier we mentioned that C# can automate some aspects of memory management, a notori- ous source of bugs in C++ code. Abandoning heap-allocated objects once you’re done with them is a coding error in C++, but it’s the normal way to free them in .NET. This service is provided by the CLR—the .NET Framework’s runtime environment. Al- though the C# compiler works closely with the runtime to make this possible, provid- ing the necessary information about how your code uses objects and data, it’s ultimately the runtime that does the work of garbage collection. Depending on what sorts of languages you may have worked with before, the idea that the language depends heavily on the runtime might seem either completely natural or somewhat disconcerting. It’s certainly different from how C and C++ work—with those languages, the compiler’s output can be executed directly by the computer, and although those languages have some runtime services, it’s possible to write code that can run without them. But C# code cannot even execute without the help of the run- time. Code that depends entirely on the runtime is called managed code. Managed compilers do not produce raw executable code. Instead, they produce an intermediate form of code called IL, the Intermediate Language.† The runtime decides exactly how to convert it into something executable. One practical upshot of managed code is that a compiled C# program can run on both 32-bit and 64-bit systems without modification, and can even run on different processor architectures—it’s often possible * VB.NET supports XML literals. Since C# 2.0 shipped, the C# and VB.NET teams have operated a policy of keeping the feature sets of the two languages similar, so the fact that VB.NET picked up a feature that C# abandoned shows a clear difference in language philosophy. † Depending on whether you read Microsoft’s documentation, or the ECMA CLI (Common Language Infrastructure) specifications that define the standardized parts of .NET and C#, IL’s proper name is either MSIL (Microsoft IL) or CIL (Common IL), respectively. The unofficial name, IL, seems more popular in practice. Language Style | 5 for code that runs on an ARM-based handheld device to run unmodified on Intel-based PCs, or on the PowerPC architecture found in the Xbox 360 game console. As interesting as CPU independence may be, in practice the most useful aspect of man- aged code and IL is that the .NET runtime can provide useful services that are very hard for traditional compilation systems to implement well. In other words, the point is to make developers more productive. The memory management mentioned earlier is just one example. Others include a security model that takes the origin of code into account rather than merely the identity of the user who happens to be running the code; flexible mechanisms for loading shared components with robust support for servicing and ver- sioning; runtime code optimization based on how the code is being used in practice rather than how the compiler guesses it might be used; and as already mentioned, the CLR’s ability to verify that code conforms to type safety rules before executing it, ruling out whole classes of security and stability bugs. If you’re a Java developer, all of this will sound rather familiar—just substitute byte- code for IL and the story is very similar. Indeed, a popular but somewhat ignorant “joke” among the less thoughtful members of the Java community is to describe C# as a poor imitation of Java. When the first version of C# appeared, the differences were subtle, but the fact that Java went on to copy several features from C# illustrates that C# was always more than a mere clone. The languages have grown more obviously different with each new version, but one difference, present from the start, is particularly im- portant for Windows developers: C# has always made it easy to get at the features of the underlying Windows platform. Continuity and the Windows Ecosystem Software development platforms do not succeed purely on their own merits—context matters. For example, widespread availability of third-party components and tools can make a platform significantly more compelling. Windows is perhaps the most striking example of this phenomenon. Any new programming system attempting to gain ac- ceptance has a considerable advantage if it can plug into some existing ecosystem, and one of the biggest differences between C# and Java is that C# and the .NET Framework positively embrace the Windows platform, while Java goes out of its way to insulate developers from the underlying OS. If you’re writing code to run on a specific operating system, it’s not especially helpful for a language to cut you off from the tools and components unique to your chosen platform. Rather than requiring developers to break with the past, .NET offers con- tinuity by making it possible to work directly with components and services either built into or built for Windows. Most of the time, you won’t need to use this—the class library provides wrappers for a lot of the underlying platform’s functionality. However, if you need to use a third-party component or a feature of the operating system that doesn’t yet have a .NET wrapper, the ability to work with such unmanaged features directly from managed code is invaluable. 6 | Chapter 1: Introducing C# While .NET offers features to ease integration with the underlying plat- form, there is still support for non-Windows systems. Microsoft’s Sil- verlight can run C# and VB.NET code on Mac OS X as well as Windows. There’s an open source project called Mono which enables .NET code to run on Linux, and the related Moonlight project is an open source version of Silverlight. So the presence of local platform integration fea- tures doesn’t stop C# from being useful on multiple platforms—if you want to target multiple operating systems, you would just choose not to use any platform-specific features. So the biggest philosophical difference between C# and Java is that C# provides equal support for direct use of operating-system-specific features and for platform independence. Java makes the former dispro- portionately harder than the latter. The latest version of C# contains features that enhance this capability further. Several of the new C# 4.0 features make it easier to interact with Office and other Windows applications that use COM automation—this was a weak spot in C# 3.0. The relative ease with which developers can reach outside the boundaries of managed code makes C# an attractive choice—it offers all the benefits of managed execution, but retains the ability to work with any code in the Windows environment, managed or not. C# 4.0, .NET 4, and Visual Studio 2010 Since C# favors general-purpose language features designed to be composed with one another, it often doesn’t make sense to describe individual new features on their own. So rather than devoting sections or whole chapters to new features, we cover them in context, integrated appropriately with other, older language features. The section you’re reading right now is an exception, of course, and the main reason is that we expect people already familiar with C# 3.0 to browse through this book in bookstores looking for our coverage of the new features. If that’s you, welcome to the book! If you look in the Preface you’ll find a guide to what’s where in the book, including a section just for you, describing where to find material about C# 4.0 features. That being said, a theme unites the new language features in version 4: they support dynamic programming, with a particular focus on making certain interoperability sce- narios simpler. For example, consider the C# 3.0 code in Example 1-3 that uses part of the Office object model to read the Author property from a Word document. Example 1-3. The horrors of Office interop before C# 4.0 static void Main(string[] args) { var wordApp = new Microsoft.Office.Interop.Word.Application(); object fileName = @"WordFile.docx"; object missing = System.Reflection.Missing.Value; C# 4.0, .NET 4, and Visual Studio 2010 | 7 object readOnly = true; Microsoft.Office.Interop.Word._Document doc = wordApp.Documents.Open(ref fileName, ref missing, ref readOnly, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing); object docProperties = doc.BuiltInDocumentProperties; Type docPropType = docProperties.GetType(); object authorProp = docPropType.InvokeMember("Item", BindingFlags.Default | BindingFlags.GetProperty, null, docProperties, new object[] { "Author" }); Type propType = authorProp.GetType(); string authorName = propType.InvokeMember("Value", BindingFlags.Default |BindingFlags.GetProperty, null, authorProp, new object[] { }).ToString(); object saveChanges = false; doc.Close(ref saveChanges, ref missing, ref missing); Console.WriteLine(authorName); } That’s some pretty horrible code—it’s hard to see what the example does because the goal is lost in the details. The reason it is so unpleasant is that Office’s programming model is designed for dynamic languages that can fill in a lot of the details at runtime. C# 3.0 wasn’t able to do this, so developers were forced to do all the work by hand. Example 1-4 shows how to do exactly the same job in C# 4.0. This is a lot easier to follow, because the code contains only the relevant details. It’s easy to see the sequence of operations—open the document, get its properties, retrieve the Author property’s value, and close the document. C# 4.0 is now able to fill in all the details for us, thanks to its new dynamic language features. Example 1-4. Office interop with C# 4.0 static void Main(string[] args) { var wordApp = new Microsoft.Office.Interop.Word.Application(); Microsoft.Office.Interop.Word._Document doc = wordApp.Documents.Open("WordFile.docx", ReadOnly: true); dynamic docProperties = doc.BuiltInDocumentProperties; string authorName = docProperties["Author"].Value; doc.Close(SaveChanges: false); Console.WriteLine(authorName); } 8 | Chapter 1: Introducing C# This example uses a couple of C# 4.0 features: it uses the new dynamic keyword for runtime binding to members. It also uses the support for optional arguments. The Open and Close methods take 16 and 3 arguments, respectively, and as you can see from Example 1-3, you need to provide all of them in C# 3.0. But Example 1-4 has only provided values for the arguments it wants to set to something other than the default. Besides using these two new features, a project containing this code would usually be built using a third new interop feature called no-PIA. There’s nothing to see in the preceding example, because when you enable no-PIA in a C# project, you do not need to modify your code—no-PIA is essentially a deployment feature. In C# 3.0, you had to install special support libraries called primary interop assemblies (PIAs) on the target machine to be able to use COM APIs such as Office automation, but in C# 4.0 you no longer have to do this. You still need these PIAs on your development machine, but the C# compiler can extract the information your code requires, and copy it into your application. This saves you from deploying PIAs to the target machine, hence the name, “no-PIA”. While these new language features are particularly well suited to COM automation interop scenarios, they can be used anywhere. (The “no-PIA” feature is narrower, but it’s really part of the .NET runtime rather than a C# language feature.) Summary In this chapter we provided a quick overview of the nature of the C# language, and we showed some of its strengths and how the latest version has evolved. There’s one last benefit you should be aware of before we get into the details in the next chapter, and that’s the sheer quantity of C# resources available on the Internet. When the .NET Framework first appeared, C# adoption took off much faster than the other .NET languages. Consequently, if you’re searching for examples of how to get things done, or solutions to problems, C# is an excellent choice because it’s so well represented in blogs, examples, tools, open source projects, and webcasts—Microsoft’s own docu- mentation is pretty evenhanded between C# and VB.NET, but on the Web as a whole, you’re far better served if you’re a C# developer. So with that in mind, we’ll now look at the fundamental elements of C# programs. Summary | 9 CHAPTER 2 Basic Programming Techniques To use a programming language, you must master the fundamentals. You need to un- derstand the elements required to construct a working program, and learn how to use the development tools to build and run code. You also need to become familiar with the everyday features for representing information, performing calculations, and mak- ing decisions. This chapter will introduce these core features of the C# language. Getting Started We’ll be working in Visual Studio, the Microsoft development environment. There are other ways to build C# programs, but Visual Studio is the most widely used and it’s freely available, so we’ll stick with that. If you don’t have Visual Studio, you can download the free Express edition from http://www.microsoft.com/express/. In the first part of this chapter, we’ll create a very simple program so that you can see the bare minimum of steps required to get up and running. We’ll also examine all of the pieces Visual Studio creates for you so that you know exactly what the development environment is doing for you. And then we’ll build some slightly more interesting ex- amples to explore the C# language. To create a new C# program, select the File→New Project menu option, or just use the Ctrl-Shift-N shortcut. This will open Visual Studio’s New Project dialog, shown in Figure 2-1, where you can pick the kind of program you want to build. In the Installed Templates list on the lefthand side, ensure that the Visual C# item is expanded, and inside that, select the Windows item—applications that run locally on Windows are the easiest to create. We’ll get into other kinds of programs such as web applications later in the book. 11 In the dialog’s center, select the Console Application template. This creates an old- fashioned command-line application that runs in a console window. It might not be the most exciting kind of program, but it’s the easiest to create and understand, so that’s where we’ll start. You need to pick a name for your program—by default, Visual Studio will suggest something unimaginative such as ConsoleApplication1. In the Name field near the bot- tom of the dialog, type HelloWorld. (OK, so that’s equally unimaginative, but at least it’s descriptive.) Visual Studio also wants to know where you’d like to put the project on your hard disk—put it wherever you like. It can also create a separate “solution” directory. That’s something you’d do in a larger program made up of multiple com- ponents, but for this simple example, you want the “Create directory for solution” checkbox to be unchecked. When you click the OK button, Visual Studio will create a new project, a collection of files that are used to build a program. C# projects always contain source code files, but they often include other types of files, such as bitmaps. This newly created project will contain a C# source file called Program.cs, which should be visible in Visual Studio’s text editor. In case you’re not following along in Visual Studio as you read this, the code is reproduced in Example 2-1. By the way, there’s no particular significance to the filename Program.cs. Visual Studio doesn’t care what you call your source files; by convention, they have a .cs extension, short for C#, although even that’s optional. Figure 2-1. Visual Studio’s New Project dialog 12 | Chapter 2: Basic Programming Techniques Example 2-1. The code in a freshly created console application using System; using System.Collections.Generic; using System.Linq; using System.Text; namespace HelloWorld { class Program { static void Main(string[] args) { } } } This program doesn’t do anything yet. To turn it into the traditional first example, you’ll need to add one line of code. This will go in between the two lines that contain the most-indented pair of braces ({ and }). The modified version is shown in Exam- ple 2-2, with the new line in bold. Example 2-2. The traditional first example, “Hello, world” using System; using System.Collections.Generic; using System.Linq; using System.Text; namespace HelloWorld { class Program { static void Main(string[] args) { Console.WriteLine("Hello, world"); } } } This example is now ready to run. From the Debug menu select the Start Without Debugging item, or just press Ctrl-F5. The program will run, and because you’ve writ- ten a console application, a console window will open. The first line of this window will contain the text “Hello, world” and this will be followed by a prompt saying “Press any key to continue...” Once you’ve finished admiring the fruits of your creation, press a key to dismiss the window. Don’t use Debug→Start Debugging or F5—this will run the application in Visual Studio’s debugging mode, which doesn’t keep the window open once the application has finished. That’s not helpful for this ex- ample, which will most likely run to completion and then close the window before you’ve had a chance to see the output. Getting Started | 13 Now that we have a complete program, let’s look at the code to see what each part is for—all of the pieces are things you’ll deal with every time you write in C#. Starting from the top, Program.cs has several lines beginning with using: using System; using System.Collections.Generic; using System.Linq; using System.Text; These using directives help the C# compiler work out what external code this particular source file will be using. No code is an island—to get any useful work done, your programs will rely on other code. All C# programs depend on the .NET Framework class library, for example: the one line of code we added to our program uses the class library to display a message. Using directives can declare an intent to use classes from any library—yours, Microsoft’s, or anyone’s. All the directives in our example start with System, which indicates that we want to use something from the .NET Framework. This text that follows the using keyword denotes a namespace. Namespaces and Types The .NET Framework class library is big. To make it easier to find your way around the many services it offers, the library is split into namespaces. For example, the System.IO namespace offers I/O (Input/Output) services such as working with files on disk, while System.Data.SqlClient is for connecting to a SQL Server database. A namespace contains types. A type typically represents either a kind of information or a kind of object. For example, there are types that provide the core forms of information used in all programs, such as System.String which represents text, or the various nu- meric types such as System.Double or System.Int32. Some types are more complex— for example, the System.Net.HttpWebRequest class represents an HTTP request to be sent to a web server. A few types do not represent any particular thing, but simply offer a set of services, such as the System.Math class, which provides mathematical functions such as Sin and Log, and constants such as π or the base of natural logarithms, e. (We will explore the nature of types, objects, and values in much more detail in the next chapter.) All types in the .NET Framework class library belong to a namespace. The purpose of a using directive is to save you from typing the namespace every single time you need to use a class. For example, in a file that has a using System; directive you can just write Math.PI to get the value of π, instead of using the full name, System.Math.PI. You’re not required to write using directives, by the way—if you happen to enjoy typing, you’re free to use the fully qualified name. But since some namespaces get quite long—for example, System.Windows.Media.Imaging—you can see how the shorthand enabled by a using directive can reduce clutter considerably. You might be wondering why namespaces are needed at all if the first thing we usually do is add a bunch of using directives to avoid having to mention the namespace 14 | Chapter 2: Basic Programming Techniques anywhere else. One reason is disambiguation—some type names crop up in multiple places. For example, the ASP.NET web framework has a type called Control, and so do both WPF and Windows Forms. They represent similar concepts, but they are used in completely different contexts (web applications versus Windows applications). Al- though all of these types are called Control, they are distinct thanks to being in different namespaces. This disambiguation also leaves you free to use whatever names you want in your own code even if some names happen to be used already in parts of the .NET class library you never knew existed. Since there are more than 10,000 types in the framework, it’s entirely possible that you might pick a name that’s already being used, but namespaces make this less of a problem. For example, there’s a Bold class in .NET, but if you happen not to be using part of the library it belongs to (WPF’s text services) you might well want to use the name Bold to mean something else in your own code. And since .NET’s own Bold type is hidden away in the System.Windows.Documents namespace, as long as you don’t add a using directive for that namespace you’re free to use the name Bold yourself to mean whatever you like. Even when there’s no ambiguity, namespaces help you find your way around the class library—related types tend to be grouped into one namespace, or a group of related namespaces. (For example, there are various namespaces starting with System.Web con- taining types used in ASP.NET web applications.) So rather than searching through thousands of types for what you need, you can browse through the namespaces—there are only a few hundred of those. You can see a complete list of .NET Framework class library namespa- ces, along with a short description of what each one is for, at http://msdn .microsoft.com/library/ms229335. Visual Studio adds four namespace directives to the Program.cs file in a new console project. The System namespace contains general-purpose services, including basic data types such as String, and various numeric types. It also contains the Console type our program uses to display its greeting and which provides other console-related services, such as reading keyboard input and choosing the color of your output text. The remaining three using directives aren’t used in our example. Visual Studio adds them to newly created projects because they are likely to be useful in many applications. The System.Collections.Generic namespace contains types for working with collec- tions of things, such as a list of numbers. The System.Linq namespace contains types used for LINQ, which provides convenient ways of processing collections of informa- tion in C#. And the System.Text namespace contains types useful for working with text. The using directives Visual Studio adds to a new C# file are there just to save you some typing. You are free to remove them if you happen not to be using those namespaces. And you can add more, of course. Namespaces and Types | 15 Removing Unwanted Using Directives There’s a quick way to remove unwanted using directives. If you right-click anywhere on your C# code, the context menu offers an Organize Usings item. This opens a submenu that includes a Remove Unused Usings item—this works out which using directives are surplus to requirements, and removes them. The submenu offers another option designed to appeal to those who like to keep their source code tidy—its Remove and Sort entry can remove unused using statements and then sort the rest into alpha- betical order. This menu is shown in Figure 2-2. Figure 2-2. Tidying up using directives The using directives are not the end of our simple program’s encounter with name- spaces. In fact, the very next line of code after these directives is also concerned with namespaces: namespace HelloWorld { ... } While using directives declare which namespaces our code consumes, this namespace keyword tells the compiler what namespace we plan to provide—the types we write in our programs belong to namespaces just like the types in the class library.* Here, Visual Studio has presumed that we’d like to put our code into a namespace named after the project we created. This is a common practice, although you’re free to use whatever * Strictly speaking, you can leave out the namespace, in which case your types will end up in the so-called global namespace. But this is considered a poor practice—you’ll normally want your own code to reap the same benefits that class libraries get from namespaces. 16 | Chapter 2: Basic Programming Techniques names you like for your namespaces—there’s no requirement that the namespace name match the program name. The C# compiler will even let you put your own code into namespaces whose names begin with System, but you should not do this (at least, not unless you work for Microsoft and are adding types to some future version of .NET’s class library). You’re likely to cause confusion if you break the convention that System namespaces contain .NET Framework types. Notice that the namespace is followed by an open brace ({). C# uses braces to denote containment—here, everything inside these braces will be in our HelloWorld name- space. Since namespaces contain types, it should come as no great surprise that the next line in the file defines a type. Specifically, it defines a class. The .NET Framework class library isn’t the only thing that gets to define classes—in fact, if you want to write any code at all in C# you must provide a type to contain that code. Some languages (such as C++) do not impose this constraint, but C# is an object- oriented (OO) programming language. We’ll explore OO concepts in the next chapter, but the main impact on our “Hello, world” example is that every bit of C# code must have a type that it calls home. There are a few different ways to define types in C#, which we’ll get to in the next few chapters, but for the present simple example, the distinctions are not yet relevant. So we use the most common, a class: class Program { ... } Again, note the braces—as with the namespace contents, the class’s contents are de- lineated by a pair of braces. We’re still not quite at the code yet—code lives inside a class, but more specifically, it must live inside a particular method inside a class. A method is a named block of code, which may optionally return some data. The class in our example defines a method called Main, and once again we use a pair of braces to show where it starts and ends: static void Main(string[] args) { ... } The first keyword here, static, tells C# that it’s not necessary to create a Program object (Program being the class that contains this method, remember) in order to use this method. As you’ll see in the next chapter, a lot of methods require an object, but our simple example doesn’t need one. Namespaces and Types | 17 The next keyword is void. This tells the compiler that our method doesn’t return any data—it just does some work. Many methods return information. For example, the System.Math class’s Cos method calculates the cosine of its input, and since it doesn’t know what you want to do with that result, it provides it as a return value—the output of the method. But the code in this example is rather more proactive than that—it decides to show a message on the screen, so there’s nothing for it to return.† On meth- ods that return data, you’d write the type of data being returned here, but since there’s nothing to return in this case, the nothingness is denoted by the void keyword. The next part, Main, is the name of the method. This happens to be a special name— the C# compiler will expect your program to provide one static method called Main, and it’ll run that method when the program is launched. The method name is followed by a parameter list, which declares the input the method requires. This particular example’s parameter list is (string[] args), which says that it expects just a single input and that the code will refer to it using the name args. It expects this input to be a sequence of text strings (the square brackets indicating that multiple strings may be passed instead of just one). As it happens, this particular pro- gram doesn’t use this input, but it’s a standard feature of the specially named Main method—command-line arguments are passed in here. We’ll return to this later in the chapter when we write a program that makes use of command-line arguments, but for now, our example doesn’t use it. So we’ll move on to the final part of the example— the code inside the Main method that was the one part we added to Visual Studio’s contributions and which represents the only work this program does: Console.WriteLine("Hello, world"); This shows the C# syntax for invoking a method. Here we’re using a method provided by the Console class, which is part of the .NET Framework class library, and it is defined in the System namespace. We could have written the fully qualified name, in which case the code would look like this: System.Console.WriteLine("Hello, world"); But because of the using System; directive earlier, we can use the shorter version—it means the same thing, it’s just more concise. The Console class provides the ability to display text in a console window and to read input typed by the user in an old-fashioned command-line application. In this case, we’re invoking the class’s WriteLine method, passing it the text "Hello, world". The WriteLine method will write whatever text we provide out to the console window. † This is the essential difference between the so-called functional and procedural approaches to coding, by the way. Code that just performs a computation or calculation and returns the result is called “functional” because it’s similar in nature to mathematical functions such as cosine, and square root. Procedural code tends to perform a sequence of actions. In some languages, such as F#, the functional style dominates, but C# programs typically use a mixture of both styles. 18 | Chapter 2: Basic Programming Techniques You’ll have noticed that the dot (.) is being used to mean different things here. We can use it to delineate the namespace name and the type name; for example, System.Console means the Console type in the System namespace. It can also be used to break up a namespace name, as in System.IO. Our example also uses it to indicate that we want to use a particular method provided by a class, as in Console.WriteLine. And as you’ll see, the dot turns up in a few other places in C#. Broadly speaking, the dot signifies that we want to use something that’s inside something else. The C# compiler works out from context exactly what that means. Although we picked over every line of code in this simple example, we haven’t quite finished exploring what Visual Studio did for us when we asked it to create a new application. To fully appreciate its work, we need to step out of the Program.cs source file and look at the whole project. Projects and Solutions It’s rare for a useful program to be so simple that you would want all of its source code in one file. You may occasionally stumble across horrors such as a single file containing tens of thousands of lines of code, but in the interest of quality (and sanity) it’s best to try to keep your source code in smaller, more manageable chunks—the larger and more complex anything gets the more likely it is to contain flaws. So Visual Studio is built to work with multiple source files, and it provides a couple of concepts for structuring your programs across those files: projects and solutions. A project is a collection of source files that the C# compiler combines to produce a single output—typically either an executable program or a library. (See the sidebar on the next page for more details on the compilation process.) The usual convention in Windows is that executable files have an .exe extension while libraries have a .dll ex- tension. (These extensions are short for executable and dynamic link library, respec- tively.) There isn’t a big difference between the two kinds of file; the main distinction is that an executable program is required to have an entry point—the Main function. A library is not something you’d run independently; it’s designed to be used by other programs, so a DLL doesn’t have its own entry point. Other than that, they’re pretty much the same thing—they’re just files that contain code and data. (The two types of file are so similar that you can use an executable as though it were a library.) So Visual Studio projects work in much the same way for programs and libraries. Projects and Solutions | 19 Source Code, Binary, and Compilation The .exe and .dll files produced by Visual Studio do not contain your source code. If you were to look at the HelloWorld.exe file produced by our example, it would not contain a copy of the text in the Program.cs file. C# is a compiled language, meaning that during the development process, the source is converted into a binary format that is easier for the computer to execute. Visual Studio compiled your code automatically when you ran the program earlier. Not all languages work this way. For example, JavaScript, a language used to add dy- namic behavior to web pages, does not need to be compiled—your web browser down- loads the source for any JavaScript required and runs it directly. But there are a few disadvantages with this. First, source code tends to be rather verbose—it’s important that source code be mean- ingful to humans as well as computers, because when we come to modify a program, we need to understand the code before changing it. But a computer can work with very dense binary representations of information, which makes it possible for compiled code to be much smaller than the source, thus taking up less space on disk and taking less time to download. Second, human-readable representations are relatively hard work for computers to process—computers are more at home with binary than with text. Compilation pro- vides the opportunity to convert all the human-readable text into a form more con- venient for the computer in advance. So compiled code tends to run faster than a system that works directly with the source. (In fact, although JavaScript was not designed to be compiled, modern JavaScript engines have taken to compiling script after down- loading it to speed things up. This still leaves it at a disadvantage to a language such as C# where compilation happens during development—when a script runs for the first time with such a system, the user of the web page has to wait while the script is down- loaded and compiled.) Some languages compile code into native machine language—the binary code that can be executed directly by a computer’s CPU. This offers a performance benefit: code compiled in this way doesn’t require any further processing to run. However, .NET languages don’t do this, because it limits where a compiled program can execute. As we mentioned in the first chapter, .NET languages compile into a so-called Intermediate Language (IL for short). This is a binary representation, so it’s compact and efficient for computers to process, but it’s not specific to any particular CPU type, enabling .NET programs to run on either 32-bit or 64-bit machines, or on different CPU architectures. The .NET Framework converts this IL into native machine language just before running it, a technique referred to as JIT (Just In Time) compilation. JIT compilation offers the best of both worlds: it’s much faster than compiling from the source, but it still retains the flexibility to target different machine types. 20 | Chapter 2: Basic Programming Techniques Some project types produce neither libraries nor executables. For ex- ample, there’s a project type for building .msi (Windows Installer) files from the outputs of other projects. So strictly speaking, a project is a fairly abstract idea: it takes some files and builds them into some kind of output. But projects containing C# code will produce either an EXE or a DLL. A solution is just a collection of related projects. If you are writing a library, you’ll probably want to write an application that uses it—even if the library is ultimately destined to be used by other people, you’ll still want to be able to try it out for testing and debugging purposes, so it’s useful to be able to have one or more applications that exercise the library’s functionality. By putting all of these projects into one solution, you can work with the DLL and its test applications all at once. By the way, Visual Studio always requires a solution—even if you’re building just one project, it is always contained in a solution. That’s why the project’s contents are shown in a panel called the Solution Explorer, shown in Figure 2-3. Figure 2-3. HelloWorld project in the Solution Explorer The Solution Explorer is usually visible on the righthand side of Visual Studio, but if you don’t see it you can open it with the View→Solution Explorer menu item. It shows all the projects in the solution—just the HelloWorld project in this example. And it shows all the files in the solution—you can see the Program.cs file we’ve been examining near the bottom of Figure 2-3. Farther up is an extra file we haven’t looked at, called AssemblyInfo.cs. If you open this you’ll see that Visual Studio puts version number and copyright information in that file—users will see this information if they view the com- piled output’s properties in Windows Explorer. Projects and Solutions | 21 You might find that on your system, the Solution Explorer doesn’t show the Solution node that’s visible at the top of Figure 2-3, and just shows the HelloWorld project. Visual Studio can be configured to hide the sol- ution when it contains just a single project. If you don’t see the solution and would like to, select the Tools→Options menu item, and in the Options dialog that opens select the Projects and Solutions item. One of the options will be the “Always show solution” checkbox—check this if you want to see the solution in the Solution Explorer even when you’ve got only one project. Besides the C# source files, the Solution Explorer as shown in Figure 2-3 also has a References section. This contains a list of all the libraries your project uses. By default, Visual Studio populates this with a list of DLLs from the .NET Framework class library that it thinks you might find useful. You might be experiencing déjà vu right now—didn’t we already tell the compiler which bits of the library we want with using directives? This is a common cause of confusion among developers learning C#. Namespaces are not libraries, and neither one is contained by the other. These facts are obscured by an apparent connection. For example, the System.Data library does in fact define a load of types in the System.Data namespace. But this is just a convention, and one that is only loosely followed. Libraries are often, but not always, named after the namespace with which they are most strongly associated, but it’s common for a library to define types in several different namespaces and it’s common for a namespace’s types to be distributed across several different li- braries. (If you’re wondering how this chaos emerged, see the sidebar below.) Namespaces and Libraries The distribution of types across DLLs in the class library is driven by a combination of efficiency requirements and history. The System.Core library is a good example of the latter. There is no System.Core namespace—this library defines types in numerous namespaces including System, System.IO, and System.Threading. But you’ll also find types in these same three namespaces in the System library and also a library called mscorlib. (All .NET programs have a reference to mscorlib, and since it’s mandatory, Visual Studio doesn’t show it in the Solution Explorer. It’s where critical types such as System.String and System.Int32 are defined.) One of the reasons System.Core exists as a separate DLL is that it first appeared in version 3.5 of .NET. With versions 3.0 and 3.5 of .NET, Microsoft chose to put completely new functionality into new DLLs rather than altering the DLLs that were provided in version 2.0. This packaging decision— choosing which types go in which DLLs—was independent from the conceptual deci- sion of which types belong in which namespaces. History doesn’t explain the whole story, though. Even the very first version of .NET split its namespaces across multiple libraries. One common reason for this was to avoid loading code that is never used. You wouldn’t want a desktop application to waste time and memory by loading the libraries for building web applications. In some cases, 22 | Chapter 2: Basic Programming Techniques namespaces are actually a pretty good guide to partitioning—chances are good that if you use one type from one of the System.Web namespaces, you’re going to be using lots of them. But there are a few cases in which namespaces are not the best way to determine packaging. For example, the System.Printing namespace is split across two libraries: the System.Printing library contains general print-related classes, but the ReachFrame work library adds extra types to the namespace that you may need if you’re working with a particular kind of printable document called an XPS file. If you’re not using that feature, you don’t need a reference to that specialized DLL. This raises a question: how do you know where to find things? It’s frustrating when adding a reference to the System.Printing library fails to give you access to the types in the System.Printing namespace that you were looking for. Fortunately, the help pages for each type tell you both the namespace and the library file (assembly) containing the type. The upshot is that the C# compiler cannot work out which libraries you want from your using directives, because in general it’s not possible to deduce which libraries are required from the namespaces alone. So a project needs to list which libraries it uses, and then individual source files in that project can declare which namespaces they are using. Visual Studio provides you with a set of references that it hopes will be useful, and for this very simple example, we’re not actually using most of them. Visual Studio notices when your code doesn’t use all of the libraries your project references, and automatically omits references to any unused libraries. This makes your binary slightly smaller than it would be if unnecessary references were left in. You can add or remove references to suit whatever program you’re building. To remove a reference, you can just select the library in the Solution Explorer and press the Delete key. (As it happens, our program is so simple that it depends only on the mandatory mscorlib library, so you could remove every DLL shown, and as long as you also remove any unused using directives from the source code, the program will still work.) To add a reference to a library, you can right-click on the References item and choose the Add Reference menu item. We’ll explore all of this in more detail in Chapter 15. It’s almost time to move on from “Hello, world” and start to explore more of the core language features, but first let’s recap what we’ve seen. The one line of executable code in our program invokes the WriteLine method of the System.Console class to print a message. This code lives inside a method whose special name, Main, marks it out as the method to run when the program starts. That method is contained by a class called Program, because C# requires all methods to belong to a type. This class is a member of the HelloWorld namespace, because we chose to follow the convention of having our namespace match the name of the compiled binary. Our program uses the using di- rectives supplied by Visual Studio to be able to refer to the Console class without needing to specify its namespace explicitly. So if you take one more look at the program, you Projects and Solutions | 23 now know what every single line is for. (It is reproduced in Example 2-3, with the unused using directives removed.) Example 2-3. “Hello, world” again (with fewer using directives) using System; namespace HelloWorld { class Program { static void Main(string[] args) { Console.WriteLine("Hello, world"); } } } With the whole example in one place, you can see clearly that the code is indented to reflect the structure. This is a common practice, but it’s not strictly necessary. As far as the C# compiler is concerned, when it comes to the space between elements of the language, there’s no difference between a single space, multiple spaces or tabs, or even blank lines—the syntax treats any contiguous quantity of whitespace as it would a single space.‡ So you are free to use space in your source code to improve legibility. This is why C# requires the use of braces to indicate containment, and it’s also why there’s a semicolon at the end of the line that prints out the message. Since C# doesn’t care whether we have one statement of code per line, split the code across multiple lines, or cram multiple statements onto one line, we need to be explicit about the end of each instruction, marking it with a ; so that the compiler knows where each new step of the program begins. Comments, Regions, and Readability While we’re looking at the structure and layout of source code, we need to examine a language feature that is extremely important, despite having precisely no effect on the behavior of your code. C# lets you add text to your source file that it will completely ignore. This might not sound important, or even useful, but it turns out to be vital if you want to have any hope of understanding code you wrote six months ago. There’s an unfortunate phenomenon known as “write-only code.” This is code that made some kind of sense to whoever wrote it at the time, but is incomprehensible to anyone trying to read it at a later date, even if the person reading it is its author. The best defense against this problem is to think carefully about the names you give the ‡ With the odd exception: in a string constant such as the “Hello, world” text in this example, whitespace is treated literally—C# presumes that if you put, say, three spaces in some text enclosed in double quotes, you really want three spaces. 24 | Chapter 2: Basic Programming Techniques features of your code and the way you structure your programs. You should strive to write your code so that it does what it looks like it does. Unfortunately, it’s sometimes necessary to do things in a nonobvious way, so even if your code is sufficiently clear that it’s easy to see what it does, it may not be at all clear why it does certain things. This tends to happen where your code meets other code— you might be interacting with a component or a service that’s idiosyncratic, or just plain buggy, and which works only if you do things in a particular way. For example, you might find that a component ignores the first attempt to do something and you need to add a redundant-looking line of code to get it to work: Frobnicator.SetTarget(""); Frobnicator.SetTarget("Norfolk"); The problem with this sort of thing is that it’s very hard for someone who comes across this code later on to know what to make of it. Is that apparently redundant line delib- erate? Is it safe to remove? Intrigue and ambiguity might make for engaging fiction, but these characteristics are rarely desirable in code. We need something to explain the mystery, and that’s the purpose of a comment. So you might write this: // Frobnicator v2.41 has a bug where it crashes occasionally if // we try to set the target to "Norfolk". Setting it to an empty // string first seems to work around the problem. Frobnicator.SetTarget(""); Frobnicator.SetTarget("Norfolk"); This is now less mysterious. Someone coming across this code knows why the appa- rently redundant line was added. It’s clear what problem it solves and the conditions under which that problem occurs, which makes it possible to find out whether the problem has been fixed in the most recent version of the offending component, making it possible to remove the fix. This makes it much easier to maintain code in the long run. As far as C# is concerned, this example is identical to the one without comments. The // character sequence tells it to ignore any further text up to the end of the line. So you can either put comments on their own line as shown earlier, or tack them onto the end of an existing line: Frobnicator.SetTarget(""); // Workaround for bug in v2.41 Like most of the C-family languages, C# supports two forms of comment syntax. As well as the single-line // form, you can write a comment that spans multiple lines, denoting the start with /* and the end with */, for example: /* This is part of a comment. This continues to be part of the same comment. Here endeth the comment. */ Comments, Regions, and Readability | 25 Bad Comments While comments can be very useful, many, sadly, are not. There are a couple of particularly common mistakes people make when writing comments, and it’s worth drawing attention to them so that you know what to avoid. Here’s the most common example: // Setting target to empty string Frobnicator.SetTarget(""); // Setting target to Norfolk Frobnicator.SetTarget("Norfolk"); These comments just repeat what the code already said. This is clearly a waste of space, but it’s surprisingly common, particularly from inexperienced developers. This may be because they’ve been told that comments are good, but they have no idea what makes a good comment. A comment should say something that’s not obvious from the code and which is likely to be useful to anyone trying to understand the code. The other common form of bad comment looks like this: // Setting target to Norfolk Frobnicator.SetTarget("Wiltshire"); Here, the comment contradicts the code. It seems like it shouldn’t be necessary to say that you shouldn’t do that, but it’s surprising how often you see this sort of thing in real code. It usually happens because someone modified the code without bothering to update the comment. A quick review of the comments after a code change is always worth doing. (Not least because if you’ve not paid enough attention to detail to notice that the comments are no longer accurate, chances are there are other problems you’ve not noticed.) XML Documentation Comments If you structure your comments in a certain way, Visual Studio is able to present the information in those comments in tool tips whenever developers use your code. As Example 2-4 shows, documentation comments are denoted with three slashes, and they contain XML elements describing the target of the comment—in this case, there’s a description of a method, its parameters, and the information it returns. Example 2-4. XML documentation comments /// /// Returns the square of the specified number. /// /// The number to square. /// The squared value. static double Square(double x) { return x * x; } 26 | Chapter 2: Basic Programming Techniques If a developer starts writing code to invoke this method, Visual Studio will show a pop up listing all available members matching what she’s typed so far, and also adds a tool tip showing the information from the element of the selected method in the list, as Figure 2-4 shows. You’ll see similar information when using classes from the .NET Framework—documentation from its class libraries is provided as part of the .NET Framework SDK included with Visual Studio. (The C# compiler can extract this information from your source files and put it in a separate XML file, enabling you to provide the documentation for a library without necessarily having to ship the source code.) Figure 2-4. Summary information from XML documentation The information shows up as you start to type arguments, as Figure 2-5 shows. The information doesn’t appear here, but there are tools that can build doc- umentation from this information into HTML files or help files. For example, Microsoft provides a tool called Sandcastle, available from http://www.codeplex.com/Sandcastle, which can generate documentation with a similar structure to the documentation for Microsoft’s own class libraries. Figure 2-5. Parameter information from XML documentation We’re moving on from “Hello, world” now, so this is a good time to create a new project if you’re following along in Visual Studio as you read. (Select File→New Project or press Ctrl-Shift-N. Note that, by default, this will create a new solution for your new project. There’s an option in the New Project dialog to add the new project to the existing solution, but in this case, let it create a new one.) Create another console application and call it RaceInfo—the code is going to perform various jobs to analyze the perform- ance of a race car. Let Visual Studio create the project for you, and you’ll end up with much the same code as we had in Example 2-1, but with the Program class in a name- space called RaceInfo instead of HelloWorld. The first task will be to calculate the average speed and fuel consumption of the car, so we need to introduce the C# mechanism for holding and working with data. Comments, Regions, and Readability | 27 Variables C# methods can have named places to hold information. These are called variables, because the information they contain may be different each time the program runs, or your code may change a variable while the program runs. Example 2-5 defines three variables in our program’s Main method, to represent the distance traveled by the car, how long it has been moving, and how much fuel it has consumed so far. These variables don’t vary at all in this example—a variable’s value can change, but it’s OK to create variables whose value is fixed. Example 2-5. Variables static void Main(string[] args) { double kmTravelled = 5.14; double elapsedSeconds = 78.74; double fuelKilosConsumed = 2.7; } Notice that the variable names (kmTravelled, elapsedSeconds, and fuelKilosConsumed) are reasonably descriptive. In algebra it’s common to use single letters as variable names, but in code it is a good practice to use names that make it clear what the variable holds. If you can’t think of a good descriptive name for a variable, that’s often a symptom of trouble. It’s hard to write code that works if it’s not clear what information the code is working with. These names indicate not just what the variables represent, but also their units. This is of no significance to the compiler—we could call the three variables tom, dick, and harry for all it cares—but it’s useful for humans looking at the code. Misunderstandings about whether a particular value is in metric or imperial units have been known to cause some extremely expensive problems, such as the accidental destruction of spacecraft. This particular race team seems to use the metric system. (If you’re wondering why the fuel is in kilograms rather than, say, liters, it’s because in high-performance motor racing, fuel is typically measured by weight rather than volume, just like it is in aviation. Fuel tends to expand or contract as the temperature changes—you get better value for your money if you refill your car in the morning on a cold day than in the middle of a hot day—so mass is more useful because it’s a more stable measure.) Variable Types All three of the variable declarations in Example 2-5 start with the keyword double. This tells the compiler what kind of information the variable holds. For this example, we’re clearly working with numbers, but .NET offers several different numeric types. Table 2-1 shows the complete set, and it may look like a bewildering assortment of 28 | Chapter 2: Basic Programming Techniques options, but in practice the choice usually goes one of three ways: int, double, or decimal, which represent integers, floating-point, or decimal floating-point numbers, respectively. Table 2-1. Numeric types C# name .NET name Purpose float System.Single Whole numbers and a limited range of fractions, with a wide range of values thanks to “floating point.” Occupies 32 bits of space. double System.Double Double-precision version of float—same idea, but using 64 bits. byte System.Byte Non-negative integer. Occupies 8 bits. Represents values from 0 to 255. sbyte System.SByte Signed integer. Occupies 8 bits. Represents values from −128 to 127. short System.Int16 Signed integer. Occupies 16 bits. Represents values from −32,768 to 32,767. ushort System.UInt16 Non-negative integer. Occupies 16 bits. Represents values from 0 to 65,535. int System.Int32 Signed integer. Occupies 32 bits. Represents values from −2,147,483,648 to 2,147,483,647. uint System.UInt32 Nonnegative integer. Occupies 32 bits. Represents values from 0 to 4,294,967,295. long System.Int64 Signed integer. Occupies 64 bits. Represents values from −9,223,372,036,854,775,808 to 9,223,372,036,854,775,807. ulong System.UInt64 Nonnegative integer. Occupies 64 bits. Represents values from 0 to 18,446,744,073,709,551,615. (none) System.Numer ics.BigInteger Signed integer. Grows in size as required. Value range limited only by available memory. decimal System.Decimal Supports whole numbers and fractions. Slightly less efficient than double, but provides more predictable behavior when using decimal fractions. Integers The int type (short for integer) represents whole numbers. That’s clearly no use for our example, because we’re dealing with numbers such as 5.14, and the closest that an int can get to that value is 5. But programs often deal with discrete quantities, such as the number of rows returned by a database query or the number of employees reporting to a particular manager. The principal advantage of an integer type is that it’s exact: there’s no scope for wondering if the number is really 5, or maybe just a number quite close to 5, such as 5.000001. Table 2-1 lists nine types capable of representing integers. The ninth, BigInteger, is a special case that we’ll get to later. The other eight support four different sizes, with a choice between the ability and inability to represent negative numbers. Unsigned numbers may seem less flexible, but they are potentially useful if you need to represent values that should never be negative. However, the unsigned integer types are not widely used—some programming languages don’t support them at all, and so you’ll find that the .NET Framework class library tends to use the signed types even Variables | 29 when the unsigned ones might make more sense. For example, the Count property available on most collection types is of type int—a signed 32-bit integer—even though it does not make sense for a collection to contain a negative number of items. Unsigned integers can also represent larger numbers than their signed equivalents. They don’t need to use up a bit to represent the sign, so they can use that to extend the range instead. However, this is something you should be wary of depending on. If you’re so close to the limits of a type’s range that one more bit makes a difference, you’re probably in danger of overflowing the type’s range in any case, and so you should consider a larger type. Besides the signed/unsigned distinction, the various types offer different sizes, and a correspondingly different range of values. 32 bits is a popular choice because it offers a usefully wide range of values and is efficient for a 32-bit processor to work with. 64- bit types are used for the (fairly rare) occasions when you’re dealing with large enough quantities that a 32-bit representation’s range of a couple of billion is insufficient. 16- bit values are rarely used, although they occasionally crop up when having to deal with old programming interfaces, file formats, or network protocols. The 8-bit byte type is important because binary I/O (e.g., working with files or network connections) is mostly byte-oriented. And for reasons of historical convention, bytes buck the trend in that the unsigned type is used more widely than the signed sbyte type. But outside of I/O, a byte is usually too small to be useful. So in practice, int is the most widely used integer type. The fact that C# even offers you all these other choices can seem a little archaic—it harks back to the time when computers had so little memory that 32-bit numbers looked like an expensive choice. It gets this from its C-family connections, but it does turn out to be useful to have this control when you need to work directly with Windows APIs, as you’ll see in Chapter 19. Notice that most of the types in Table 2-1 have two names. C# uses names such as int and long, but the .NET Framework calls these types by longer names such as System.Int32 and System.Int64. The shorter C# names are aliases, and C# is happy to let you use either. You can write this: int answer = 42; or this: System.Int32 answer = 42; or, if your C# source file has a using System; directive at the top, you can write this: Int32 answer = 42; 30 | Chapter 2: Basic Programming Techniques All of these are equivalent—they produce exactly the same compiled output. The last two are equivalent simply because of how namespaces work, but why does C# support a completely different set of aliases? The answer is historical: C# was designed to be easy to learn for people who are familiar with the so-called C family of languages, which includes C, C++, Java, and JavaScript. Most of the languages in this family use the same names for certain kinds of data types—most use the name int to denote a conveniently sized integer, for example. So C# is merely following suit—it allows you to write code that looks like it would in other C-family languages. By contrast, the .NET Framework supports many different languages, so it takes the prosaic approach of giving these numeric data types descriptive names—it calls a 32- bit integer System.Int32. Since C# lets you use either naming style, opinion is divided on the matter of which you should use.§ The C-family style (int, double, etc.) seems to be the more popular. Version 4 of the .NET Framework introduces an extra integer type that works slightly differently from the rest: BigInteger. It does not have a C-style name, so it’s known only by its class library name. Unlike all the other integer types, which occupy a fixed amount of memory that determines their range, a BigInteger can grow. As the number it represents gets larger, it simply consumes more space. The only theoretical limit on range is the amount of memory available, but in practice, the computational cost of working with vast numbers is likely to be the limiting factor. Even simple arithmetic operations such as multiplication can become rather expensive with sufficiently vast numbers. For example, if you have two numbers each with 1 million decimal digits— each number occupies more than 400 kilobytes of memory—multiplying these together takes more than a minute on a reasonably well-specified computer. BigInteger is useful for mathematical scenarios when you need to be able to work with very large numbers, but in more ordinary situations, int is the most popular integer type. Integers are all very well for countable quantities, but what if you need the ability to represent something other than a whole number? This is where floating-point types come in. Floating point The double and float types both offer the ability to support numbers with a fractional component. For example, you can represent the value 1.5 with either of these types, which you can’t do with any of the integer types. The only difference between double and float is the level of precision available: since floating-point numbers have a fixed size, they can offer only a limited amount of precision. This means that they cannot represent any fraction—the limited precision means floating-point numbers can only represent most numbers approximately. § Whenever more than one way of doing something exists in a programming system, a schism inevitably forms, offering the opportunity for long and pointless arguments over which is “better.” Variables | 31 Floating Point If you’re wondering why these are called floating-point types, the name is a technical description of how they work internally. These numbers contain a fixed number of binary digits to hold the value, and then another number that says where the . should go. So the point is a binary point, the binary equivalent of a decimal point. It’s float- ing because it can move around. A float offers about seven decimal places of precision, whereas a double offers about 17. (Strictly speaking, they offer 23 and 52 binary places of precision, respectively. These are binary formats, so their precision does not correspond to an exact number of decimal places of precision.) So the following code: double x = 1234.5678; double y = x + 0.0001; Console.WriteLine(x); Console.WriteLine(y); prints out what you’d expect: 1234.5678 1234.5679 If instead we use the float type: float x = 1234.5678f; float y = x + 0.0001f; Console.WriteLine(x); Console.WriteLine(y); we get this: 1234.568 1234.568 This often surprises new developers, but it’s normal, and is by no means unique to C#. If only a limited amount of space is available, you simply cannot represent all possible numbers with complete accuracy. Floating point, approximate as it is, is the standard way to represent noninteger numbers in most programming languages, and you’ll see this sort of inaccuracy anywhere. 32 | Chapter 2: Basic Programming Techniques Notice that when modifying the code to use float instead of double, we added the letter f to the end of the constants—0.0001f instead of just 0.0001, for example. This is because C# treats a number with a decimal point as a value of type double, and if we try to store this in a variable of type float, we risk losing data due to the lower precision. Such code is treated as an error, hence the need to explicitly tell C# that we know we’re working with single-precision floating-point values, with the f suffix. If you have a double you really would like to turn into a float, and you are prepared to tolerate the loss of precision, you can tell C# this with a cast operator. For example: double x = 1234.5678; double y = x + 0.0001; float impreciseSum = (float) (x + y); The (float) syntax here is a cast, an explicit instruction to the compiler that we want to convert the type. Since we are being explicit, the com- piler does not treat this as an error. For a lot of applications, limited precision is not too big a problem as long as you’re aware of it, but there’s a slightly subtler problem that afflicts double and float. They are both binary representations, because that’s the most efficient way of packing pre- cision into the space available. However, it means that you can get some surprising- looking results when working in decimal. For example, the number 0.1 cannot be rep- resented accurately as a finite-length binary fraction. (For much the same reason that 1/9 cannot accurately be represented as a finite-length decimal fraction. In either case, you end up with a recurring [i.e., infinitely long] number: 1/9 in decimal is 0.1111 recurring; 1/10 in decimal is 0.1, but in binary it’s 0.00011001100110011 recurring.) Take the following example: float f1 = 0.1f; float f2 = f1 + 0.1f; float f3 = f2 + 0.1f; float f4 = f3 + 0.1f; float f5 = f4 + 0.1f; float f6 = f5 + 0.1f; float f7 = f6 + 0.1f; float f8 = f7 + 0.1f; float f9 = f8 + 0.1f; Console.WriteLine(f1); Console.WriteLine(f2); Console.WriteLine(f3); Console.WriteLine(f4); Console.WriteLine(f5); Console.WriteLine(f6); Console.WriteLine(f7); Console.WriteLine(f8); Console.WriteLine(f9); Variables | 33 (We’ll see how to avoid such highly repetitive code when we get to loops later in the chapter, by the way.) This shows the following rather suspect output: 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8000001 0.9000001 The inability to represent 0.1 accurately is not initially obvious, because .NET rounds the numbers when displaying them, masking the problem. However, as we keep adding numbers together, the inaccuracies add up and eventually start to become visible. As you can imagine, accountants don’t like this sort of thing—if those numbers happened to represent fund transfers measured in billions of dollars, having $0.0000001 billion ($100) suddenly appear out of nowhere every eight transactions would be considered a bad practice. This is why there’s a special numeric type just for working in decimal. Decimal floating point The decimal type (or System.Decimal, as .NET calls it) is superficially very similar to double and float, except its internal representation is adapted to decimal representa- tions. It can represent up to 28 decimal digits of precision, and unlike the two binary floating-point types, any number that can be written as a 28-digit (or fewer) decimal can be represented completely accurately as a decimal variable. The value 0.1 fits com- fortably into 28 digits with room to spare, so this would fix the problem in the previous example. The decimal type still has limited precision; it just has less surprising behavior if you’re looking at all your numbers in decimal. So if you are performing calculations involving money, decimal is likely to be a better choice than double or float. The trade-off is that it’s slightly less efficient—computers are more at home in binary than decimal. For our race information application, we don’t have any particular need for decimal fidelity, which is why we’re using the double type in Example 2-5. Getting back to that example, recall that we defined three variables that hold the dis- tance our car has traveled, how long it took, and how much fuel it burned in the process. Here it is again so that you don’t have to flip back to it: static void Main(string[] args) { double kmTravelled = 5.141; double elapsedSeconds = 78.738; double fuelKilosConsumed = 2.7; } 34 | Chapter 2: Basic Programming Techniques Now that we’ve looked at the numeric types, the structure of these lines is pretty clear. We start with the type of data we’d like to work with, followed by the name we’d like to use, and then we use the = symbol to assign a value to the variable. But assigning constant values isn’t very exciting. You can get the computer to do more useful work, because you can assign an expression into a variable. Expressions and Statements An expression is a piece of code that produces a value of some kind. We’ve actually seen several examples already, the most basic being the numbers we’re assigning into the variables. So in our example, a number such as: 5.141 is an expression. Expressions where we just tell C# what value we want are called literal expressions. More interestingly, expressions can perform calculations. For ex- ample, we could calculate the distance traveled per kilogram of fuel consumed with the expression in Example 2-6. Example 2-6. Dividing one variable by another kmTravelled / fuelKilosConsumed The / symbol denotes division. Multiplication, addition, and subtraction are done with *, +, and -, respectively. You can combine expressions together too. The / operator requires two inputs—the dividend and the divisor—and each input is itself an expression. We were able to use variable names such as kmTravelled because a variable name is valid as an expression— the resultant value is just whatever that variable’s value is. But we could use literals, as Example 2-7 shows. (A trap awaits the unwary here; see the sidebar on the next page.) Example 2-7. Dividing one literal by another 60 / 10 Or we could use a mixture of literals and variable names to calculate the elapsed time in minutes: elapsedSeconds / 60 or a multiplication expression as one of the inputs to a division expression to calculate the elapsed time in hours: elapsedSeconds / (60 * 60) Expressions and Statements | 35 Integer Versus Floating-Point Division There’s a subtle difference between how division works in Examples 2-6 and 2-7. Since the two literals in Example 2-7 do not contain decimal points, the compiler treats them as integers, and so it will perform an integer division. But since the kmTravelled and fuelKilosConsumed variables are both floating-point, it will use a floating-point division operation. In this particular case it doesn’t matter, because dividing 60 by 10 produces another integer, 6. But what if the result had not been a whole number? If we had written this, for example: 3/4 the result would be 0, as this is an integer division—4 does not go into 3. However, given the following: double x = 3; double y = 4; the value of x/y would be 0.75, because C# would use floating-point division, which can deal with nonwhole results. If you wanted to use floating-point calculations with literals, you could write: 3.0/4.0 The decimal point indicates that we want floating-point numbers, and therefore float- ing-point division, so the result is 0.75. (The parentheses ensure that we divide by 60 * 60. Without the parentheses, this ex- pression would divide by 60, and then multiply by 60, which would be less useful. See the sidebar on the next page.) And then we could use this to work out the speed in kilometers per hour: kmTravelled / (elapsedSeconds / (60 * 60)) Expressions don’t actually do anything on their own. We have described a calculation, but the C# compiler needs to know what we want to do with the result. We can do various things with an expression. We could use it to initialize another variable: double kmPerHour = kmTravelled / (elapsedSeconds / (60 * 60)); or we could display the value of the expression in the console window: Console.WriteLine(kmTravelled / (elapsedSeconds / (60 * 60))); Both of these are examples of statements. Whereas an expression describes a calculation, a statement describes an action. In the last two examples, we used the same expression—a calculation of the race car’s speed— but the two statements did different things: one evaluated the expression and assigned it into a new variable, while the other evaluated the expression and then passed it to the Console class’s WriteLine method. 36 | Chapter 2: Basic Programming Techniques Order of Evaluation C# has a set of rules for working out the order in which to evaluate the components of an expression. It does not necessarily work from left to right, because some operators have a higher precedence than others. For example, imagine evaluating this: 1.0 + 3.0 / 4.0 from left to right. Start with 1.0, add 3.0 which gets you to 4.0, and then divide by 4.0— the result would be 1.0. But the conventional rules of arithmetic mean the result should be one and three quarters. And that’s just what C# produces—the result is 1.75. The division is performed before the addition, because division has higher precedence than division. Some groups of operators have equal precedence. For example, multiplication and division have equal precedence. When expressions contain multiple operations with the same precedence, mathematical operations are evaluated from left to right. So 10.0 / 2.0 * 5.0 evaluates to 25.0. But parentheses trump precedence, so 10.0 / (2.0 * 5.0) evaluates to 1.0. Some programming books go into great depths about all the details of precedence, but it makes for exceptionally tedious reading—C# has 15 different levels of precedence. The details are important for compiler writers, but of limited value for developers— code that relies heavily on precedence can be hard to read. Using parentheses to make evaluation order explicit can often improve clarity. But if you would like the gory details, you can find them at http://msdn.microsoft.com/en-us/library/aa691323. An expression’s type matters. The examples we just looked at involve numbers or numeric variables, and are of type double or int. Expressions can be of any type, though. For example, ("Hello, " + "world") is an expression of type string. If you wrote an assignment statement that tried to assign that expression into a variable of type double, the compiler would complain—it insists that expressions are either of the same type as the variable, or of a type that is implicitly convertible to the variable’s type. Implicit conversions exist for numeric types when the conversion won’t lose information—for example, a double can represent any value that an int can, so you’re allowed to assign an integer expression into a double variable. But attempting the opposite would cause a compiler error, because doubles can be larger than the highest int, and they can also contain fractional parts that would be lost. If you don’t mind the loss of information, you can put a cast in front of the expression: int approxKmPerHour = (int) kmPerHour; This casts the kmPerHour (which we declared earlier as a double) to an int, meaning it’ll force the value to fit in an integer, possibly losing information in the process. Expressions and Statements | 37 A variable doesn’t have to be stuck with its initial value for its whole life. We can assign new values at any time. Assignment Statements The previous section showed how to assign an expression’s value into a newly declared variable: double kmPerHour = kmTravelled / (elapsedSeconds / (60 * 60)); If at some later stage in the program’s execution new information becomes available, we could assign a new value into the kmPerHour variable—assignment statements aren’t required to declare new variables, and can assign into existing ones: kmPerHour = updateKmTravelled / (updatedElapsedSeconds / (60 * 60)); This overwrites the existing value in the kmPerHour variable. C# offers some specialized assignment statements that can make for slightly more suc- cinct code. For example, suppose you wanted to add the car’s latest lap time to the variable holding the total elapsed time. You could write this: elapsedSeconds = elapsedSeconds + latestLapTime; This evaluates the expression on the righthand side, and assigns the result to the vari- able specified on the lefthand side. However, this process of adding a value to a variable is so common that there’s a special syntax for it: elapsedSeconds += latestLapTime; This has exactly the same effect as the previous expression. There are equivalents for the other mathematical operators, so -= means to subtract the expression on the right from the variable on the left, *= does the same for multiplication, and so on. Increment and Decrement Operators While we’re looking at how to update values, we should also look at the increment and decrement operators. If we want to maintain a lap count, we could add one each time the car completes a lap: lapCount += 1; The C programming language’s designers considered adding one to be a sufficiently important case to devise an even more special syntax for it, called the increment oper- ator, which C# duly offers: lapCount++; There’s also a decrement operator, --, which subtracts one. This example is a state- ment, but you can also use the increment and decrement operators in the middle of an expression: int currentLap = lapCount++; 38 | Chapter 2: Basic Programming Techniques But be careful. The expression on the right of this assignment statement means “eval- uate the current value of lapCount and then increment lapCount after getting its current value.” So if lapCount was 3 before executing this statement, currentLap would be 3 and lapCount would be 4 after executing it. If you want to use the updated value, you put the increment (or decrement) operator before its target: int currentLap = ++lapCount; You could write a program that consisted entirely of variable declaration, assignment, increment, and method invocation statements. However, such a program wouldn’t be very interesting—it would always execute the same sequence of statements just once in the same order. Fortunately, C# provides some more interesting statements that allow a program to make decisions that dynamically change the flow of execution through the code. This is sometimes referred to as flow control. Flow Control with Selection Statements A selection statement selects which code path to execute next, based on the value of an expression. We could use a selection statement to work out whether the race car is likely to run out of fuel in the next few laps, and display a warning if it is. C# offers two selection statements: if statements and switch statements. To illustrate selection in action, we need to make a slight change to the program. Right now, our example hardcodes all of its data—the distance traveled, fuel consumed, and time elapsed are compiled into the code as literals. This makes selection statements uninteresting—the program would make the same decision every time because the data would always be the same. For the decision to be meaningful, we need to modify the program to accept input. Since we’re writing a console application, we can supply the necessary information as command-line arguments. We could run the program passing in the total distance, elapsed time, and fuel consumed, for example: RaceInfo 20.6 312.8 10.8 We can write a modified version of the program that picks up these command-line values instead of hardcoding them, as shown in Example 2-8. Example 2-8. Reading command-line inputs static void Main(string[] args) { double kmTravelled = double.Parse(args[0]); double elapsedSeconds = double.Parse(args[1]); double fuelKilosConsumed = double.Parse(args[2]); } There are a few interesting features to point out here before we add a selection state- ment. First, recall from earlier that the Main method, our program’s entry point, is passed a sequence of strings representing the command-line arguments in a variable called args. This sequence is an array, a .NET construct for holding multiple items of Flow Control with Selection Statements | 39 a particular type. (You can make arrays of anything—numbers, text, or any type. The string[] syntax indicates that this method expects an array of strings.) In an expression, we can retrieve a particular item from an array by specifying a number in square brackets after the array variable’s name. So the first three lines in our method here use args[0], args[1], and args[2] to get the first, second, and third items in the array—the three command-line arguments in this case. C-family languages tend to number things from zero, and C# follows suit. This may seem a little idiosyncratic, but it makes sense to the com- puter. You can think of it as saying how far into the array you want to look. If you want to look at the thing right at the start of the array, you don’t need to go any distance at all, so an offset of zero gets you the first item. If you’re British, you’ll recognize this logic from floor numbering—the first floor in a building in Great Britain is not the one at street level; you have to go up one flight of stairs to get to the first floor. Also notice the use of double.Parse. Command-line arguments are passed as text, be- cause the user can type anything: RaceInfo Jenson Button Rocks But our program expects numbers. We need to do something to convert the strings into numbers, and that’s what double.Parse does: it expects the text to contain a dec- imal number, and converts it into a double-precision floating-point representation of that number. (If you’re wondering what it would do if the text wasn’t in fact a number, it’ll throw an exception. Chapter 6 explains what that means and how to deal with it gracefully, but for now it means our program would crash with an error.) This example illustrates that method invocations can also be expressions—the double type’s Parse method returns a value of type double, meaning we can use it to initialize a variable of type double. But that’s all by the by—the point here is that our program now gets data that could be different each time the program runs. For example, a race engineer in the pit lane could run the program with new distance, timing, and fuel information each time the car completes a lap. So our program can now usefully make decisions based on its input using selection statements. One such statement is the if statement. if Statements An if statement is a selection statement that decides whether to execute a particular piece of code based on the value of an expression. We can use this to show a low-fuel warning by adding the code in Example 2-9 at the end of our example’s Main method. Most of the code performs calculations in preparation for making the decision. The if statement toward the end of the example makes the decision—it decides whether to execute the block of code enclosed in braces. 40 | Chapter 2: Basic Programming Techniques Example 2-9. if statement double fuelTankCapacityKilos = 80; double lapLength = 5.141; double fuelKilosPerKm = fuelKilosConsumed / kmTravelled; double fuelKilosRemaining = fuelTankCapacityKilos - fuelKilosConsumed; double predictedDistanceUntilOutOfFuel = fuelKilosRemaining / fuelKilosPerKm; double predictedLapsUntilOutOfFuel = predictedDistanceUntilOutOfFuel / lapLength; if (predictedLapsUntilOutOfFuel < 4) { Console.WriteLine("Low on fuel. Laps remaining: " + predictedLapsUntilOutOfFuel); } To test this, we need to run the program with command-line arguments. You could open a command prompt, move to the directory containing the built output of your project, and run it with the arguments you want. (It’ll be in the bin\Debug folder that Visual Studio creates inside your project’s folder.) Or you can get Visual Studio to pass arguments for you. To do that, go to the Solution Explorer panel and double-click on the Properties icon. This will open the project’s properties view, which has a series of tabs on the lefthand side. Select the Debug tab, and in the middle you’ll see a “Com- mand line arguments” text box as shown in Figure 2-6. Figure 2-6. Passing command-line arguments in Visual Studio If you run the program with arguments corresponding to just a few laps (e.g., 15 238 8) it won’t print anything. But try running it with the following arguments: 141.95 2156.2 75.6. It’ll predict that the car has about 1.6 laps of fuel remaining. The if state- ment in Example 2-9 tests the following expression: predictedLapsUntilOutOfFuel < 4 The < symbol means “less than.” So the code in braces following the if statement runs only if the number of predicted laps of fuel is less than 4. Clearly, 1.6 is less than 4, so in this case it’ll run that code, printing out the following: Low on fuel. Laps remaining: 1.60701035044548 Flow Control with Selection Statements | 41 You need to use the right kind of expression in an if statement. In this case, we’ve performed a comparison—we’re testing to see if a variable is less than 4. There are only two possible outcomes: either it’s less than 4 or it isn’t. So this expression is clearly different in nature to the expressions performing mathematical calculations. If you were to modify the program so that it prints the value of that expression: Console.WriteLine(predictedLapsUntilOutOfFuel < 4); it would display either True or False. The .NET Framework has a special type to rep- resent such an either/or choice, called System.Boolean, and as with the numeric types, C# defines its own alias for this type: bool.‖ An if statement requires a Boolean ex- pression. So if you try to use an expression with a numeric result, such as this: if (fuelTankCapacityKilos - fuelKilosConsumed) the compiler will complain with the error “Cannot implicitly convert type ‘double’ to ‘bool’.” This is its way of saying that it expects a bool—either true or false—and you’ve given it a number. In effect, that code says something like “If fourteen and a half then do this.” What would that even mean? The C language decided to answer that question by saying that 0 is equivalent to false, and anything else is equivalent to true. But that was only because it didn’t have a built-in Boolean type, so its if statement had to be able to work with numeric expressions. This turned out to be a frequent cause of bugs in C programs. Since C# does have a built-in bool type, it insists that an if statement’s expression is always of type bool. C# defines several operators which, like the < operator we used in Example 2-9, can compare two numbers to produce a Boolean true/false answer. Table 2-2 shows these. Some of these operators can be applied to non-numeric types too. For example, you can use the == and != operators to compare strings. (You might expect the other com- parison operators to work too, telling you whether one string would come before or after another when sorted alphabetically. However, there’s more than one way to sort strings—it turns out that the method used varies based on language and culture. And rather than have an expression such as text1 < text2 mean different things in different contexts, C# simply doesn’t allow it. If you want to compare strings, you have to call one of the methods provided by the String class that lets you say how you’d like the comparison to work.) ‖ The Boolean type is named after George Boole, who invented a branch of mathematical logic that uses just two values: true and false. His system is fundamental to the operation of all digital electronics, so it’s a shame that C# doesn’t see fit to spell his name properly. 42 | Chapter 2: Basic Programming Techniques Table 2-2. Comparison operators C# operator Meaning < Less than > Greater than <= Less than or equal to >= Greater than or equal to == Equal to != Not equal to Just as you can combine numeric expressions into more complex and powerful ex- pressions, C# provides operators that let you combine Boolean expressions to test multiple conditions. The && operator combines two Boolean expressions into a single expression that’s true only if both conditions are true. In our race example, we might use this to hide the low-fuel warning if we’re near the end of the race and the car has enough fuel to make it to the finish line. Imagine that we added an extra argument to pass in the number of remaining laps in the race, and an additional variable to hold that value; we could write: if ((predictedLapsUntilOutOfFuel < 4) && (predictedLapsUntilOutOfFuel < remainingLapsInRace)) { Console.WriteLine("Low on fuel. Laps remaining: " + predictedLapsUntilOutOfFuel); } This has the same effect as the following slightly more verbose code: if (predictedLapsUntilOutOfFuel < 4) { if (predictedLapsUntilOutOfFuel < remainingLapsInRace) { Console.WriteLine("Low on fuel. Laps remaining: " + predictedLapsUntilOutOfFuel); } } Only if both conditions are true will the message be displayed. There’s also a || oper- ator. Like &&, the || operator combines two Boolean expressions, but will be true if either of them is true. Flow Control with Selection Statements | 43 if...else The if statement examples we’ve looked at so far just decide whether to execute some optional code, but what if we want to choose between two actions? An if statement can optionally include an else section that runs if the condition was false, as in this hypothetical post-race example: if (weWonTheRace) { Sponsors.DemandMoreMoney(); } else { Driver.ReducePay(); } One type of if/else test comes up often enough that C-family languages have a special syntax for it: sometimes you want to pick between one of two values, based on some test. You could write this: string messageForDriver; if (weWonTheRace) { messageForDriver = "Congratulations"; } else { messageForDriver = "You're fired"; } Sometimes it’s more convenient to be able to put this inside an expression. This can be done with the ternary operator, so called because it contains three expressions: a Boo- lean test expression, the expression to use if the test is true, and the expression to use if the test is false. The syntax uses ? and : characters to separate the expressions, so the basic pattern is test ? resultIfTrue : resultIfFalse. We can collapse the previous if...else example to a single assignment statement by using the ternary operator in the expression on the righthand side of the assignment: string messageForDriver = weWonTheRace ? "Congratulations" : "You're fired"; You don’t have to space it out like this, by the way—we put the two options on separate lines to make them easy to see. But some people like to use the ternary operator to condense as much logic as possible into as little space as possible; this is either admir- able conciseness or impenetrable terseness, depending on your personal tastes. You can string multiple if...else tests together. To see how that might be useful in our example, consider how in motor racing, incidents or weather conditions may cause the race stewards to initiate certain safety procedures, such as temporarily disallowing overtaking maneuvers while wreckage is cleared from the track, releasing the safety car for the drivers to follow slowly if the wreckage is particularly spectacular, or in extreme 44 | Chapter 2: Basic Programming Techniques cases “red-flagging” the race—a temporary complete halt followed by a restart. Each of these has its own appropriate response, which can be dealt with by a chain of if...else if...else statements, as shown in Example 2-10. Example 2-10. Testing multiple conditions with if and else string raceStatus = args[3]; if (raceStatus == "YellowFlag") { Driver.TellNotToOvertake(); } else if (raceStatus == "SafetyCar") { Driver.WarnAboutSafetyCar(); } else if (raceStatus == "RedFlag") { if (ourDriverCausedIncident) { Factory.OrderNewCar(); Driver.ReducePay(); if (feelingGenerous) { Driver.Resuscitate(); } } else { Driver.CallBackToPit(); } } else { Driver.TellToDriveFaster(); } While this works, there’s an alternative. This pattern of choosing one option out of many is sufficiently common that C# has a special selection statement to handle it. switch and case Statements A switch statement lets you specify a list of expected values, and what to do for each value. The values can be either strings or integral types. (Integral types include int, short, etc.—you cannot switch on floating-point numbers. Enumeration types, which are discussed in Chapter 3, are considered to be integral types for the purposes of a switch statement.) We can use this to rewrite Example 2-10 as shown in Example 2-11. Flow Control with Selection Statements | 45 Example 2-11. Testing multiple conditions with switch and case string raceStatus = args[3]; switch (raceStatus) { case "YellowFlag": Driver.TellNotToOvertake(); break; case "SafetyCar": Driver.WarnAboutSafetyCar(); break; case "RedFlag": if (ourDriverCausedIncident) { Factory.OrderNewCar(); Driver.ReducePay(); if (feelingGenerous) { Driver.Resuscitate(); } } else { Driver.CallBackToPit(); } break; default: Driver.TellToDriveFaster(); break; } The break keyword you can see at the end of each case is present mainly for consistency with other C-like languages. In C and C++, if you leave off the break, the code will “fall” out of one case through to the next. So if we left off the break in the YellowFlag case, we’d end up telling drivers not to overtake and then warning them about the safety car. This would be a bug—and in general, you almost always don’t want fall- through. It’s unfortunate that in C and C++ fall-through was the default. C# changes this: if you want fall-through you must ask for it explicitly by writing goto case "SafetyCar". But despite fall-through no longer being the implicit default, you still need to write the same break state- ment as you would in other C-family languages when you don’t want fall-through—if you leave it out you’ll get an error. You might be wondering what is the point—this does exactly the same as Exam- ple 2-10, so why do we need a different syntax? As it happens, we don’t—there’s noth- ing you can do with switch and case that you can’t do with if and else. But switch and 46 | Chapter 2: Basic Programming Techniques case offer one useful advantage: they make it clear what we’re doing—we’re looking at a single expression (raceStatus) and we’re choosing one of a number of options based on the value of that expression. A developer familiar with C# can look at this code and understand the structure of the decision-making process at a glance. With the previous example, you would need to look at each else if statement in turn to make sure it wasn’t doing something more complex—chained else if statements are more flexible than switch statements, because each new link in the chain is allowed to test a com- pletely different expression, but that flexibility comes at the cost of making it harder to understand the code. Sometimes a self-imposed constraint can make code easier to read and maintain, and a switch statement is a good example of that. Selection statements make programs considerably more useful than they would oth- erwise be—they enable programs to make decisions. But our examples are still rather straightforward—they run just once, from start to finish, with the odd variation in the execution flow. The amount of work that is done is pretty trivial. So there’s another kind of statement that plays to a computer’s greatest strength: the ability to perform simple repetitive tasks many times over. Iteration Statements An iteration statement allows a sequence of other statements to be executed several times. (Repeated execution is also often known as a loop because, like the race car, the code goes round and round again.) This seems like it could be useful in our race data analysis—race cars usually complete many laps, so we will probably have multiple sets of data to process. It would be annoying to have to write the same code 60 times just to process all the data for a 60-lap race. Fortunately, we don’t have to—we can use one of C#’s iteration statements. Imagine that instead of passing in timing or fuel information as command-line argu- ments, the data was in files. We might have a text file containing one line per lap, with the elapsed time at the end of each lap. Another text file could contain the remaining fuel at the end of each lap. To illustrate how to work with such data, we’ll start with a simple example: finding the lap on which our driver went quickest. Since this code is a little different from the previous example, start a new project if you want to follow along. Make another console application called LapAnalysis. To be able to test our code we’ll need a file containing the timing information. You can add this to your Visual Studio project. Right-click on the LapAnalysis project in the Solution Explorer and select Add→New Item from the context menu. (Or just press Ctrl-Shift-A.) In the Installed Templates section on the left, select the General category under Visual C# Items, and then in the central area select Text File. Call the file LapTimes.txt and click Add. You’ll need this file to be somewhere the program can get to. Go to the Properties panel for the file—this is usually below the Solution Explorer panel, but if you don’t see it, right-click on LapTimes.txt in the Solution Explorer and Iteration Statements | 47 select Properties. In the Properties panel, you should see a Copy to Output Directory property. By default, this is set to “Do not copy”. Change it to “Copy if newer”—Visual Studio will ensure that an up-to-date copy of the file is available in the bin\Debug folder in which it builds your program. You’ll need some data in this file. We’ll be using the following—these numbers represent the elapsed time in seconds since the start of the race at the end of each lap: 78.73 157.2 237.1 313.8 390.7 470.2 The program is going to read in the contents of the file. To do this, it’ll need to use types from the System.IO namespace, so you’ll need to add the following near the top of your Program.cs file: using System.IO; Then inside the Main method, use the following code to read the contents of the file: string[] lines = File.ReadAllLines("LapTimes.txt"); The File type is in the System.IO namespace, and its ReadAllLines method reads in all the lines of a text file and returns an array of strings (string[]) with one entry per line. The easiest way to work through all these entries is with a foreach statement. foreach Statements A foreach statement executes a block of statements once for every item in a collection such as an array. For example, this: foreach (string line in lines) { Console.WriteLine(line); } will display every line of text from the lines array we just built. The block to execute each time around is, as ever, delimited by a { } pair. We have to provide the C# compiler with two things at the start of a foreach loop: the variable we’d like to use to access each item from the collection, and the collection itself. The string line part declares the first bit—the so-called iteration variable. And then the in lines part says that we want to iterate over the items in the lines array. So each time around the loop, line will contain the next string in lines. We can use this to discover the fastest lap time, as shown in Example 2-12. 48 | Chapter 2: Basic Programming Techniques Example 2-12. Finding the fastest lap with foreach string[] lines = File.ReadAllLines("LapTimes.txt"); double currentLapStartTime = 0; double fastestLapTime = 0; foreach (string line in lines) { double lapEndTime = double.Parse(line); double lapTime = lapEndTime - currentLapStartTime; if (fastestLapTime == 0 || lapTime < fastestLapTime) { fastestLapTime = lapTime; } currentLapStartTime = lapEndTime; } Console.WriteLine("Fastest lap time: " + fastestLapTime); The currentLapStartTime begins at zero, but is updated to the end time of the previous lap each time around the loop—we need this to work out how long each lap took, because each line of the file contains the total elapsed race time at each lap. And the fastestLapTime variable contains the time of the fastest lap yet found—it’ll be updated each time a faster lap is found. (We also update it when it’s zero, which it will be the first time we go around.) This finds the fastest lap time—76.7 seconds in the example data we’re using. But it doesn’t tell us which lap that was. Looking at the numbers, we can see that it happens to be the fourth, but it would be nice if the program could tell us. One way to do this is to declare a new variable called lapNumber, initializing it to 1 outside the loop, and adding one each time around, to keep track of the current lap. Then we can record the lap number on which we found the fastest time. Example 2-13 shows a modified ver- sion, with the additional code in bold. Example 2-13. Fastest lap including lap number string[] lines = File.ReadAllLines("LapTimes.txt"); double currentLapStartTime = 0; double fastestLapTime = 0; int lapNumber = 1; int fastestLapNumber = 0; foreach (string line in lines) { double lapEndTime = double.Parse(line); double lapTime = lapEndTime - currentLapStartTime; if (fastestLapTime == 0 || lapTime < fastestLapTime) { fastestLapTime = lapTime; fastestLapNumber = lapNumber; } currentLapStartTime = lapEndTime; lapNumber += 1; } Console.WriteLine("Fastest lap: " + fastestLapNumber); Console.WriteLine("Fastest lap time: " + fastestLapTime); Iteration Statements | 49 If you’re trying this out, this might be a good opportunity to acquaint yourself with Visual Studio’s debugging features—see the sidebar below. The Debugger When your code includes flow control statements that can vary the sequence of oper- ations, or how many times code runs, it can be useful to inspect the execution. If your code doesn’t work quite how you expect, you can watch what it does one line at a time by using Visual Studio’s built-in debugger. If instead of running the program normally you run it with the Debug→Step Into menu item (or the F11 keyboard shortcut if you’re using the C# profile for Visual Studio), it will run the code one line at a time—each time you choose Step Into, it will run one more line of code. And if you hover your mouse pointer over a variable, it will show you the current value, allowing you to see the current state of your program, as well as its current position. You can also arrange for the program to stop in the debugger when it reaches a particular point by setting a breakpoint, either by clicking in the left margin of the code editor or by putting the cursor on the line in question and selecting Debug→Toggle Breakpoint. A red dot appears in the margin to indicate that the code will stop when it reaches this point. Breakpoints are active only if you run the program from within the debugger, so you need to make sure you start with Debug→Start Debugging (or press F5) if you want breakpoints to work. Visual Studio’s debugger is a powerful and flexible system—these simple techniques barely scratch its surface, but they are very useful when trying to diagnose troublesome behavior in a program. Example 2-13 works well enough, but there’s an alternative iteration statement you can use for this sort of scenario: a for statement. for Statements A for statement is a loop in which some variable is initialized to a start value, and is modified each time around the loop. The loop will run for as long as some condition remains true—this means a for loop does not necessarily have to involve a collection, unlike a foreach loop. Example 2-14 is a simple loop that counts to 10. Example 2-14. Counting with a for loop for (int i = 1; i <= 10; i++) { Console.WriteLine(i); } Console.WriteLine("Coming, ready or not!"); The for keyword is followed by parentheses containing three pieces. First, a variable is declared and initialized. Then the condition is specified—this particular loop will 50 | Chapter 2: Basic Programming Techniques iterate for as long as the variable i is less than or equal to 10. You can use any Boolean expression here, just like in an if statement. And finally, there is a statement to be executed each time around the loop—adding one to i in this case. (As you saw earlier, i++ adds one to i. We could also have written i += 1, but the usual if arbitrary con- vention in C-style languages is to use the ++ operator here.) Earlier we recommended using variable names that are long enough to be descriptive, so you might be raising an eyebrow over the use of i as a variable name. There’s a convention with for loops where the iteration variable just counts up from zero—short variable names such as i, j, k, x, and y are often used. It’s not a universal convention, but you’ll see it widely used, particularly with short loops. We’re using this convention in Example 2-14 only because you will come across it sooner or later, and so we felt it was important to show it. But it’s arguably not an especially good way to write clear code, so feel free to choose more meaningful names in your own code. We could use this construct as an alternative way to find the fastest lap time, as shown in Example 2-15. Example 2-15. Finding the fastest lap with for string[] lines = File.ReadAllLines("LapTimes.txt"); double currentLapStartTime = 0; double fastestLapTime = 0; int fastestLapNumber = 0; for (int lapNumber = 1; lapNumber <= lines.Length; lapNumber++) { double lapEndTime = double.Parse(lines[lapNumber - 1]); double lapTime = lapEndTime - currentLapStartTime; if (fastestLapTime == 0 || lapTime < fastestLapTime) { fastestLapTime = lapTime; fastestLapNumber = lapNumber; } currentLapStartTime = lapEndTime; } Console.WriteLine("Fastest lap: " + fastestLapNumber); Console.WriteLine("Fastest lap time: " + fastestLapTime); This is pretty similar to the foreach example. It’s marginally shorter, but it’s also a little more awkward—our program is counting the laps starting from 1, but arrays in .NET start from zero, so the line that parses the value from the file has the slightly ungainly expression lines[lapNumber - 1] in it. (Incidentally, this example avoids using a short iteration variable name such as i because we’re numbering the laps from 1, not 0— short iteration variable names tend to be associated with zero-based counting.) Argu- ably, the foreach version was clearer, even if it was ever so slightly longer. The main Iteration Statements | 51 advantage of for is that it doesn’t require a collection, so it’s better suited to Exam- ple 2-14 than Example 2-15. while and do Statements C# offers a third kind of iteration statement: the while loop. This is like a simplified for loop—it has only the Boolean expression that decides whether to carry on looping, and does not have the variable initialization part, or the statement to execute each time around. (Or if you prefer, a for loop is a fancy version of a while loop—neither for nor foreach does anything you couldn’t achieve with a while loop and a little extra code.) Example 2-16 shows an alternative approach to working through the lines of a text file based on a while loop. Example 2-16. Iterating through a file with a while loop static void Main(string[] args) { using (StreamReader times = File.OpenText("LapTimes.txt")) { while (!times.EndOfStream) { string line = times.ReadLine(); double lapEndTime = double.Parse(line); Console.WriteLine(lapEndTime); } } } The while statement is well suited to the one-line-at-a-time approach. It doesn’t require a collection; it just loops until the condition becomes false. In this example, that means we loop until the StreamReader tells us we’ve reached the end of the file.# (Chap- ter 11 describes the use of types such as StreamReader in detail.) The exclamation mark (!) in front of the expression means not—you can put this in front of any Boolean expression to invert the result. So the loop runs for as long as we are not at the end of the stream. We could have used a for loop to implement this one-line-at-a-time loop—it also iterates until its condition becomes false. The while loop happens to be a better choice here simply because in this example, we have no use for the variable initialization or loop statement offered by for. #You’ll have noticed the using keyword on the line where we get hold of the StreamReader. We use this construct when it’s necessary to indicate exactly when we’ve finished with an object—in this case we need to say when we’re done with the file to avoid keeping operating system file handles open. 52 | Chapter 2: Basic Programming Techniques The approach in Example 2-16 would be better than the previous examples for a par- ticularly large file. The code can start working straight away without having to wait for the entire file to load, and it will use less memory because it doesn’t build the array containing every single line—it can hold just one line at a time in memory. For our example lap time file with just six lines of data, this won’t make any difference, but if you were processing a file with hundreds of thousands of entries, this while-based example could provide noticeably better performance than the array-based examples. This does not mean that while is faster than for or foreach. The per- formance difference here is a result of the code working with the file in a different way, and has nothing to do with the loop construct. In gen- eral, it’s a bad idea to focus on which language features are “fastest.” Performance usually depends on the way in which your code solves a problem, rather than which particular language feature you use. Note that for and while loops might never execute their contents at all. If the condition is false the first time around, they’ll skip the loop entirely. This is often desirable—if there’s no data, you probably want to do no work. But just occasionally it can be useful to write a loop that is guaranteed to execute at least once. We can do this with a variation on the while loop, called the do while loop: do { Console.WriteLine("Waiting..."); } while (DateTime.Now.Hour < 8); The while keyword and condition come at the end, and we mark the start of the loop with the do keyword. This loop always executes at least once, testing the condition at the end of each iteration instead of the start. So this code will repeatedly show the message “Waiting...” until the current time is 8:00 a.m. or later. If it’s already past 8:00 a.m., it’ll still write out “Waiting...” once. Breaking Out of a Loop It can sometimes be useful to abandon a loop earlier than its natural end. In the case of a foreach loop, this might mean stopping before you’ve processed every item in the collection. With for or while loops, you get to write the loop condition so that you can stop under whatever conditions you like, but it can sometimes be more convenient to put the code that makes a decision to abandon a loop somewhere inside the loop body rather than in the condition. For these eventualities, C# provides the break keyword. Iteration Statements | 53 We saw break already in a switch statement in Example 2-11—we used it to say that we’re done with the switch and want to break out of that statement. The break keyword does the same thing in a loop: using (StreamReader times = File.OpenText("LapTimes.txt")) { while (!times.EndOfStream) { string line = times.ReadLine(); if (line == "STOP!") { break; } double lapEndTime = double.Parse(line); Console.WriteLine(lapEndTime); } } This is the loop from Example 2-16, modified to stop if it comes across a line in the input file that contains the text “STOP!” This breaks out immediately, abandoning the rest of the loop and leaping straight to the first line of code after the enclosing loop’s closing brace. (In that case, this happens to be the enclosing using statement’s closing brace, which will close the file handle.) Some people regard this use of break as bad practice. It makes it harder to understand the loop. When a loop contains no break statements, you can understand its lifetime by looking at the while (or for, or foreach) part. But if there are break statements, you need to look at more of the code to get a complete understanding of when the loop will finish. More generally, flow control that jumps suddenly out of the middle of a construct is frowned upon, because it makes it much harder for some- one to understand how execution flows through a program, and pro- grams that are hard to understand tend to be buggy. The computer scientist Edsger Dijkstra submitted a short letter on this topic in 1968 to an academic journal, which was printed under a now infamous head- ing, “Go-to statement considered harmful”. If you’re interested in iconic pieces of computing history, or if you’d like a detailed explanation of exactly why this sort of jumpy flow control is problematic, you can find the original letter at http://www.cs.utexas.edu/users/EWD/ewd02xx/ EWD215.PDF. To recap what we’ve explored so far, we’ve seen how to work with variables to hold information, how to write expressions that perform calculations, how to use selection statements that decide what to do, and how to build iteration statements that can do things repeatedly. There’s one more basic C# programming feature we need to look at to cover the most important everyday coding features: methods. 54 | Chapter 2: Basic Programming Techniques Methods As we saw earlier, a method is a named block of code. We wrote a method already— the Main method that runs when our program starts. And we used methods provided by the .NET Framework class library, such as Console.WriteLine and File.ReadAll Lines. But we haven’t looked at how and why you would introduce new methods other than Main into your own code. Methods are an essential mechanism for reducing your code’s complexity and enhanc- ing its readability. By putting a section of code into its own method with a carefully chosen name that describes what the method does, you can make it much easier for someone looking at the code to work out what your program is meant to do. Also, methods can help avoid repetition—if you need to do similar work in multiple places, a method can help you reuse code. In our race car example, there’s a job we may need to do multiple times: reading in numeric values from a file. We did this for timing information, but we’re going to need to do the same with fuel consumption and distance. Rather than writing three almost identical bits of code, we can put the majority of the code into a single method. The first thing we need to do is declare the method—we need to pick a name, define the information that comes into the method, and optionally define the information that comes back out. Let’s call the method ReadNumbersFromFile, since that’s what it’s going to do. Its input will be a text string containing the filename, and it will return an array of double-precision floating-point numbers. The method declaration, which will go inside our Program class, will look like this: static double[] ReadNumbersFromFile(string fileName) As you may recall from the discussion of Main earlier, the static keyword indicates that we do not need an instance of the containing Program type to be created for this method to run. (We’ll be looking at nonstatic methods in the next chapter when we start dealing with objects.) C# follows the C-family convention that the kind of data coming out of the method is specified before the name and the inputs, so next we have double[], indicating that this method returns an array of numbers. Then we have the name, and then in parentheses, the inputs required by this method. In this example there’s just one, the filename, but this would be a comma-separated list if more inputs were required. After the method declaration comes the method body—the statements that make up the method, enclosed in braces. The code isn’t going to be quite the same as what we’ve seen so far—up until now, we’ve converted the text to numbers one at a time imme- diately before processing them. But this code is going to return an array of numbers, just like File.ReadAllLines returns an array of strings. So our code needs to build up that array. Example 2-17 shows one way of doing this. Methods | 55 Example 2-17. A method for reading numbers from a file static double[] ReadNumbersFromFile(string fileName) { List numbers = new List(); using (StreamReader file = File.OpenText(fileName)) { while (!file.EndOfStream) { string line = file.ReadLine(); // Skip blank lines if (!string.IsNullOrEmpty(line)) { numbers.Add(double.Parse(line)); } } } return numbers.ToArray(); } This looks pretty similar to the example while loop we saw earlier, with one addition: we’re creating an object that lets us build up a collection of numbers one at a time— a List. It’s similar to an array (a double[]), but an array needs you to know how many items you want up front—you can’t add more items onto an existing array. The advantage of a List is that you can just keep adding new numbers at will. That matters here because if you look closely you’ll see we’ve modified the code to skip over blank lines, which means that we actually don’t know how many numbers we’re going to get until we’ve read the whole file. Once you’re done adding numbers to a list, you can call its ToArray() method to get an array of the correct size. This list class is an example of a collection class. .NET offers several of these, and they are so extremely useful that Chapters 7, 8, and 9 are related to working with collections. Notice the return keyword near the end of Example 2-17. This is how we return the information calculated by our method to whatever code calls the method. As well as specifying the value to return, the return keyword causes the current method to exit immediately, and for execution to continue back in the calling method. (In methods with a void return type, which do not return any value, you can use the return keyword without an argument to exit the method. Or you can just let execution run to the end of the method, and it will return implicitly.) If you’re wondering how the method re- members where it’s supposed to go back to, see the sidebar on the next page. With the ReadNumbersFromFile method in place, we can now write this sort of code: double[] lapTimes = ReadNumbersFromFile("LapTimes.txt"); double[] fuelLevels = ReadNumbersFromFile("FuelRemainingByLap.txt"); 56 | Chapter 2: Basic Programming Techniques The Call Stack When you invoke a method, the CLR allocates some memory to keep track of that method’s state. This state includes incoming arguments and local variables. When a method calls out to another method, the method state also remembers where we were in the calling method’s code to be able to carry on later. If you have nested method calls—if a first method calls a second method which calls a third method, for example—you end up with a sequence of method states, and this sequence is often referred to as the call stack. In general, a stack is a sequence of items where you can add or remove items only at the end of the sequence; by convention, we use the terms push and pop to describe adding and removing stack items. So when C# code invokes a new method, it pushes a new method state record onto the call stack. When the method returns, either because execution reaches the end or because we’ve hit a return statement, the current method state is popped from the call stack, and then execution resumes from where the previous method state record says the calling method had reached. You can look at the call stack in the Visual Studio debugger. The Debug→Win- dows→Call Stack menu item displays a window showing a list of all the current methods in the call stack. You can double-click on any of these items to see the current location, and if you’ve opened any of the debug windows that show local variable state from the Debug→Windows menu, these will show local variables for the method you select. It doesn’t take a lot of effort to understand that this code is reading in numbers for lap times and fuel levels from a couple of text files—the code makes this aspect of its behavior much clearer than, say, Example 2-12. When code does what it says it does, you make life much easier for anyone who has to look at the code after you’ve written it. And since that probably includes you, you’ll make your life easier in the long run by moving functionality into carefully named methods. This idea of moving code out of the middle of one method and into a separate method is very common, and is an example of refactoring. Generally speaking, refactoring means restructuring code without changing its behavior, to either simplify it, make it easier to understand and maintain, or avoid duplication. There are so many ways to refactor code that whole books have been written on the topic, but this particular refactoring operation is so useful that Visual Studio can automate it. If you select some code and then right-click on the C# editor window, it offers a Refactor→Extract Method menu item that does this for you. In practice, it’s not always that straightforward—you might need to re- structure the code a little first, before you’re in a position to factor out the pieces you’d like to move into a method. Example 2-17 had to work slightly differently from any of the previous examples to package the code into a reusable method. But while it may require some work, it’s a useful technique to apply. Methods | 57 Summary In this chapter, we looked at some of the most important concepts involved in the everyday writing of C#. We saw how to create and run projects in Visual Studio. We saw how namespaces help us work with the .NET Framework class library and other external code, without getting lost in the thousands of classes on offer. We used vari- ables and expressions to store and perform calculations with data. We used selection statements to make decisions based on input, and iteration statements to perform re- petitive work on collections of data. And we saw how splitting your code into well- named methods can enhance the reusability and readability of your code. In the next chapter, we’ll step outside the world of methods and look at C#’s support for object- oriented programming. 58 | Chapter 2: Basic Programming Techniques CHAPTER 3 Abstracting Ideas with Classes and Structs In the previous couple of chapters, we looked at some basic programming techniques such as loops and conditions, and used some of the data types built into the language and platform, such as int and string. Unfortunately, real programs—even fairly simple ones—are much, much more com- plicated than the examples we’ve built so far. They need to model the behavior of real- world objects like cars and planes, or ideas like mathematical expressions, or behaviors, like the transaction between you and your favorite coffee shop when you buy a double espresso and a brownie with your bank card. Divide and Conquer The best way to manage this complexity is to break a system down into manageable pieces, where each piece is small enough for us to understand completely. We should aim to craft each piece so that it fits neatly into the system as a whole with a small enough number of connections to the other pieces that we can comprehend all of those too. Abstracting Ideas with Methods We’ve already seen one tool for dividing our code into manageable pieces: methods. A method is a piece of a program that encapsulates a particular behavior completely. It’s worth understanding the benefits of methods, because the same principles apply to the classes and structs that are this chapter’s main subject. 59 You will often see the term function used instead of method; they’re related, but not identical. A function is a method that returns something. Some methods just do some work, and do not return any value. So in C#, all functions are methods, but not all methods are functions. Methods offer a contract: if we meet particular conditions, a method will do certain things for us. Conditions come in various forms: we might need to pass arguments of suitable types, perhaps with limits on the range (e.g., negative numbers may not be allowed). We may need to ensure certain things about the program’s environment— maybe we need to check that certain directories exist on disk, or that there’s sufficient free memory and disk space. There may be constraints on when we are allowed to call the method—perhaps we’re not allowed to call it if some related work we started earlier hasn’t completed yet. Likewise, there are several ways in which a method can hold up its side of the bargain. Perhaps it will just return a string or a number that is the result of a calculation involving the method’s inputs. It might change the state of some entity in our system in some way, such as modifying an employee’s salary. It may change something about the sys- tem environment—the method might install a new device driver, or change the current user’s color scheme, for example. Some methods interact with the outside world by sending messages over the network. Some aspects of the contract are formalized—a method’s parameter list defines the number and type of arguments we need to pass, for example, and its return type tells us what, if anything, to expect as a return value. But most of the contract is informally specified—we rely on documentation (or sometimes, conversations with the developer who wrote the method) to understand the full contract. But understand it we must, because the contract is at the heart of how methods make our lives easier. Methods simplify things for us in two ways. If we are the user of a method, then, as long as its internal implementation conforms to the contract, we can treat it as a “black box.” We call it, we expect it to work as described, and we don’t need to worry about how it worked. All its internal complexity is hidden from us, freeing us to think about ideas like “increase this employee’s salary,” without getting bogged down by details such as “open a connection to the database and execute some SQL.” If, on the other hand, we are the developer of a method, we don’t need to worry about who might call us, and why. As long as our implementation works as promised, we can choose any means of implementation we like—perhaps optimizing for speed, or size, or (more often than not) simplicity and maintainability. We can concentrate on details like whether we’re using the right connection string, and whether the SQL query modi- fies the database as intended, without needing to ask ourselves questions like “should we even be adjusting this particular employee’s salary at all?” So, one objective of good design is to hide distracting details and expose a simple model to your client. This practice is called encapsulation, and it’s harder than it looks. 60 | Chapter 3: Abstracting Ideas with Classes and Structs As is so often the case in life, making something look easy takes years of practice and hard work. It can also be a thankless task: if you devise a contract that is a model of clarity, people will probably think it was easy to design. Conversely, unnecessary com- plexity is often mistaken for cleverness. While methods are essential for achieving encapsulation, they do not guarantee it. It’s all too easy to write methods whose contract is unclear. This often happens when developers do something as an afterthought— it can be oh so tempting to add a bit of extra code to an existing method as a quick solution to a problem, but this risks making that’s method’s responsibilities less clear. A method’s name is often a good indicator of the clarity of the contract— if the name is vague, or worse, if it’s an inaccurate description of what the method does, you’re probably looking at a method that does a bad job of encapsulation. One of the great things about methods is that we can use them to keep breaking things into smaller and smaller pieces. Suppose we have some method called PlaceOrder, which has a well-defined responsibility, but which is getting a bit complicated. We can just split its implementation into smaller methods—say, CheckCustomerCredit, AllocateStock, and IssueRequestToWarehouse. These smaller methods do different bits of the work for us. This general technique, sometimes called functional decomposition, has a long history in mathematics. It was explored academically in computing applications as early as the 1930s. Bearing in mind that the first working programmable computers didn’t appear until the 1940s, that’s quite a pedigree. In fact, it has been around for so long that it now seems “obvious” to most people who have had anything to do with computer programming. That’s not the end of the story, though. Methods are great for describing the dynam- ics of a system—how things change in response to particular input data (the method arguments), and the results of those changes (a function’s return value, or a method’s side effects). What they’re not so good at is describing the current state of the system. If we examine a set of functions and a load of variables, how can we work out which pieces of information are supposed to be operated on by which functions? If methods were the only tool available for abstraction, we’d have a hard time telling the difference between the double that describes my blood pressure, and can be operated on by this method: void LowerMyBloodPressure(double pressureDelta) and the double that describes my weight and can be affected by this method: void EatSomeDonuts(int quantityOfDonuts) As programs get ever larger, the number of system state variables floating around in- creases, and the number of methods can explode exponentially. But the problems aren’t Divide and Conquer | 61 just about the sheer number of functions and variables you end up with. As you try to model a more complex system, it becomes harder to work out which functions and variables you actually need—what is a good “decomposition” of the system? Which methods relate to one another, and to which variables? Abstracting Ideas with Objects and Classes In the 1960s, two guys called Dahl and Nygaard (they’re Norwegian) were working on big simulation systems and were struggling with this problem. Because they worked on simulating real things, they realized that their code would be easier to understand if they had some clear way to group together all of the data and functions related to a particular type of real thing (or a particular object, we might say). They designed a programming language that could do this, called Simula 67 (after the year of its birth), and it is generally recognized as the grandmother of all the languages we’d call object-oriented, which (of course) includes C#. They had hit upon two important concepts: • The class: a description of a collection of data and the functions that operate on them • The object: an instance of a collection of data and the functions that operate on them (i.e., an instance of a class) With these simple ideas, we can remove all doubt over which functions operate on which data—the class describes for us exactly what goes with what, and we can handle multiple entities of the same kind by creating several objects of a particular class. Object-oriented analysis As an example, let’s think about a very simple computer system that maintains the information for an air traffic control (ATC) operation. (Safety notice: if you happen to be building an ATC system, I strongly recommend that you don’t base it on this one.) How does (this particular, slightly peculiar) ATC system work? It turns out that we’ve got a bunch of people in a big room in Washington, tracking a large number of planes that buzz around the airport in Seattle. Each plane has an identifier (BA0049, which flies in from London Heathrow, for instance). We need to know the plane’s position, which we’ll represent using three numbers: an altitude (in feet); the distance from the airport control tower (in miles); and a compass heading (measured in degrees from North), which will also be relative to the tower. Just to be clear, that’s not the direction the aircraft itself is facing—it’s the direction we’d have to face in order to be looking at the plane if we’re standing in the tower. We also need to know whether the aircraft is coming in to us, or away from us, and how fast. This, apparently, is quite important. (A more comprehensive model might include a second compass heading, representing 62 | Chapter 3: Abstracting Ideas with Classes and Structs the exact direction the plane is facing. But to keep this example simple, we’ll just track whether planes are approaching or departing.) As the planes come in, the controllers give them permission to take off or land, and instruct them to change their heading, height, or speed. The aim is to avoid them hitting each other at any point. This, apparently, is also quite important. At present they have a system where each controller is responsible for a particular piece of airspace. They have a rack which contains little slips of plastic with the aircraft’s ID on it, ordered by the height at which they are flying. If they are coming in to the airport, they use a piece of blue plastic. If they are going away, they use white plastic. To keep track of the heading, distance, and speed, they just write on the slip with a china graph pencil.* If the plane moves out of their airspace, they hand the plane over to another controller, who slips it into his own rack. So that’s our specification. In reality, a safety-critical system such as ATC would have a more robust spec. However, when lives are not at stake, software specifications are often pretty nebulous, so this example is, sadly, a fair representation of what to expect on your average software project. Armed with this brilliant description we need to come up with a design for a program which can model the system. We’re going to do that using object-oriented techniques. When we do an object-oriented analysis we’re looking for the different classes of object that we are going to describe. Very often, they will correspond to real things in the system. For a class to represent these real objects properly, we need to work out what information it is going to hold, and what functions it will define to manipulate that information. In general, any one piece of information will belong to exactly one object, of exactly one class. Not all of your classes will represent real-world objects. Some will relate to more abstract concepts like collections, or commands. However, de- signs that wander too far into the realms of the wholly abstract are often “clever” but not necessarily “good”. In our ATC example, it’s clear that we have a whole lot of different planes buzzing round the airport. It would therefore seem logical that we would model each one as an object, for which we would define a class called Plane. Because C# is a language with object-oriented features, we have a simple and expressive way of doing that. * A special kind of crayon, designed for writing on glossy surfaces such as plastic. Divide and Conquer | 63 Defining Classes We can start out with the simplest possible class. It will have no methods, and no data, so as a model of a plane in our system, it leaves something to be desired, but it gets us started. If you want to build your own version as you read, create a new Console Application project just as we did in Chapter 2. To add a new class, use the Project→Add Class menu item (or right-click on the project in the Solution Explorer and select Add→Class). It’ll add a new file for the class, and if we call it Plane.cs, Visual Studio will create a new source file with the usual using directives and namespace declaration. And most im- portantly, the file will contain a new, empty class definition, as shown in Example 3-1. Example 3-1. The empty Plane class class Plane { } Right; if we look back at the specification, there’s clearly a whole bunch of information we’ve got about the plane that we need to store somewhere. C# gives us a handy mechanism for this called a property. Representing State with Properties Each plane has an identifier which is just a string of letters and numbers. We’ve already seen a built-in type ideal for representing this kind of data: string. So, we can add a property called Identifier, of type string, as Example 3-2 shows. Example 3-2. Adding a property class Plane { string Identifier { get; set; } } A property definition always states the type of data the property holds (string in this case), followed by its name. By convention, we use PascalCasing for this name—see the sidebar on the next page. As with most nontrivial elements of a C# program, this is followed by a pair of braces, and inside these we say that we want to provide a get- ter and a set-ter for the property. You might be wondering why we need to declare these—wouldn’t any property need to be gettable and settable? But as we’ll see, these explicit declarations turn out to be useful. 64 | Chapter 3: Abstracting Ideas with Classes and Structs PascalCasing and camelCasing Most programming languages, including C#, use whitespace to separate elements of the code—it must be clear where one statement (or keyword, variable, or whatever) ends and the next begins, and we often rely on spaces to mark the boundaries. But this gives us a problem when it comes to naming. Lots of features of a program have names—classes, methods, properties, and variables, for example—and we might want to use multiple words in a name. But we can’t put a space in the middle of a name like this: class Jumbo Jet { } The C# compiler would complain—the space after Jumbo marks the end of the name, and the compiler doesn’t understand why we’ve put a second name, Jet, after that. If we want to use multiple words in a name, we have to do it without using spaces. C# programmers conventionally use two styles of capitalization to put multiple words in a name: • PascalCasing, where each word starts with a capital letter. This is used for types, properties, and methods. • camelCasing, where the first word starts with a lowercase letter and all subsequent words get a capital. This is used for parameters and fields. Pascal casing takes its name from the fact that it was a popular style among Pascal programmers. It’s not a widely used language today, but lots of developers cut their teeth on it a decade or three ago when drainpipe trousers, trilby hats, and black-and- white print T-shirts were the latest in fashion (or at least, they were in parts of Europe). And, by no coincidence whatsoever, Anders Hejlsberg (a key figure in the C# design team) also designed Borland’s Turbo Pascal. As for camel casing, that name comes from the fact that uppercase letters only ever appear in the middle of the name, meaning you get one or more humps in the middle, like a camel. There’s a wrinkle in these conventions. Acronyms generally get treated as though they are words, so if you had a class for an RGB color you might call it ColorRgb, and a color with an alpha channel might be ColorArgb. (The .NET Framework class libraries include types that refer to Argb, and people often mistakenly think that the “Arg” is short for “argument” rather than Alpha, Red, Green, and Blue.) There’s an exception to this exception: two-letter acronyms are usually capitalized. So a person’s intelligence quotient might be recorded as PersonIQ. These naming conventions are optional, but strongly recommended to help people understand your code. MSDN offers an extensive set of guidelines for these sorts of conventions at http://msdn.microsoft.com/library/ms229042. Defining Classes | 65 If we create an instance of this class, we could use this Identifier property to get and set its identifier. Example 3-3 shows this in a modified version of the Main function in our Program.cs file. Example 3-3. Using the Plane class’s property static void Main(string[] args) { Plane someBoeing777 = new Plane(); someBoeing777.Identifier = "BA0049"; Console.WriteLine( "Your plane has identifier {0}", someBoeing777.Identifier); // Wait for the user to press a key, so // that we can see what happened Console.ReadKey(); } But wait! If you try to compile this, you end up with an error message: 'Plane.Identifier' is inaccessible due to its protection level What’s that all about? Protection Levels Earlier, we mentioned that one of the objectives of good design is encapsulation: hiding the implementation details so that other developers can use our objects without relying on (or knowing about) how they work. As the error we just saw in Example 3-3 shows, a class’s members are hidden by default. If we want them to be visible to users of our class, we must change their protection level. Every entity that we declare has its own protection level, whether we specify it or not. A class, for example, has a default protection level called internal. This means that it can only be seen by other classes in its own assembly. We’ll talk a lot more about assemblies in Chapter 15. For now, though, we’re only using one assembly (our ex- ample application itself), so we can leave the class at its default protection level. While classes default to being internal, the default protection level for a class member (such as a property) is private. This means that it is only accessible to other members of the class. To make it accessible from outside the class, we need to change its protec- tion level to public, as Example 3-4 shows. Example 3-4. Making a property public class Plane { public string Identifier { 66 | Chapter 3: Abstracting Ideas with Classes and Structs get; set; } } Now when we compile and run the application, we see the correct output: Your plane has identifier BA0049 Notice how this is an opt-in scheme. If you don’t do anything to the contrary, you get the lowest sensible visibility. Your classes are visible to any code inside your assembly, but aren’t accessible to anyone else; a class’s properties and methods are only visible inside the class, unless you explicitly choose to make them more widely accessible. When different layers specify different protection, the effective accessibility is the low- est specified. For example, although our property has public accessibility, the class of which it is a member has internal accessibility. The lower of the two wins, so the Identifier property is, in practice, only accessible to code in the same assembly. It is a good practice to design your classes with the smallest possible public interface (part of something we sometimes call “minimizing the surface area”). This makes it easier for clients to understand how they’re supposed to be used and often cuts down on the amount of testing you need to do. Having a clean, simple public API can also improve the security characteristics of your class framework, because the larger and more complex the API gets, the harder it generally gets to spot all the possible lines of attack. That being said, there’s a common misconception that accessibility modifiers “secure” your class, by preventing people from accessing private members. Hence this warning: It is important to recognize that these protection levels are a convenient design constraint, to help us structure our applications properly. They are not a security feature. It’s possible to use the reflection features de- scribed in Chapter 17 to circumvent these constraints and to access these supposedly hidden details. To finish this discussion, you should know that there are two other protection levels available to us—protected and protected internal—which we can use to expose (or hide) members to developers who derive new classes from our class without making the members visible to all. But since we won’t be talking about derived classes until Chapter 4, we’ll defer the discussion of these protection levels until then. We can take advantage of protection in our Plane class. A plane’s identifier shouldn’t change mid-flight, and it’s a good practice for code to prevent things from happening that we know shouldn’t happen. We should therefore add that constraint to our class. Fortunately, we have the ability to change the accessibility of the getter and the setter individually, as Example 3-5 shows. (This is one reason the property syntax makes use declare the get and set explicitly—it gives us a place to put the protection level.) Defining Classes | 67 Example 3-5. Making a property setter private class Plane { public string Identifier { get; private set; } } Compiling again, we get a new error message: The property or indexer 'Plane.Identifier' cannot be used in this context because the set accessor is inaccessible The problem is with this bit of code from Example 3-3: someBoeing777.Identifier = "BA0049"; We’re no longer able to set the property, because we’ve made the setter private (which means that we can only set it from other members of our class). We wanted to prevent the property from changing, but we’ve gone too far: we don’t even have a way of giving it a value in the first place. Fortunately, there’s a language feature that’s perfect for this situation: a constructor. Initializing with a Constructor A constructor is a special method which allows you to perform some “setup” when you create an instance of a class. Just like any other method, you can provide it with pa- rameters, but it doesn’t have an explicit return value. Constructors always have the same name as their containing class. Example 3-6 adds a constructor that takes the plane’s identifier. Because the construc- tor is a member of the class, it’s allowed to use the Identifier property’s private setter. Example 3-6. Defining a constructor class Plane { public Plane(string newIdentifier) { Identifier = newIdentifier; } public string Identifier { get; private set; } } 68 | Chapter 3: Abstracting Ideas with Classes and Structs Notice how the constructor looks like a standard method declaration, except that since there’s no need for a return type specifier, we leave that out. We don’t even write void, like we would for a normal method that returns nothing. And it would be weird if we did; in a sense this does return something—the newly created Plane—it just does so implicitly. What sort of work should you do in a constructor? Opinion is divided on the subject— should you do everything required to make the object ready to use, or the minimum necessary to make it safe? The truth is that it is a judgment call—there are no hard and fast rules. Developers tend to think of a constructor as being a relatively low-cost op- eration, so enormous amounts of heavy lifting (opening files, reading data) might be a bad idea. Getting the object into a fit state for use is a good objective, though, because requiring other functions to be called before the object is fully operational tends to lead to bugs. We need to update our Main function to use this new constructor and to get rid of the line of code that was setting the property, as Example 3-7 shows. Example 3-7. Using a constructor static void Main(string[] args) { Plane someBoeing777 = new Plane("BA0049"); Console.WriteLine( "Your plane has identifier {0}", someBoeing777.Identifier); Console.ReadKey(); } Notice how we pass the argument to the constructor inside the parentheses, in much the same way that we pass arguments in a normal method call. If you compile and run that, you’ll see the same output as before—but now we have an identifier that can’t be changed by users of the object. Be very careful when you talk about properties that “can’t be changed” because they have a private setter. Even if you can’t set a property, you may still be able to modify the state of the object referred to by that property. The built-in string type happens to be immune to that be- cause it is immutable (i.e., it can’t be changed once it has been created), so making the setter on a string property private does actually prevent clients from changing the property, but most types aren’t like that. Speaking of properties that might need to change, our specification requires us to know the speed at which each plane is traveling. Sadly, our specification didn’t mention the units in which we were expected to express that speed. Let’s assume it is miles per hour, Defining Classes | 69 and add a suitable property. We’ll use the floating-point double data type for this. Example 3-8 shows the code to add to Plane. Example 3-8. A modifiable speed property public double SpeedInMilesPerHour { get; set; } If we were to review this design with the customer, they might point out that while they have some systems that do indeed want the speed in miles per hour the people they liaise with in European air traffic control want the speed in kilometers per hour. To avoid confusion, we will add another property so that they can get or set the speed in the units with which they are familiar. Example 3-9 shows a suitable property. Example 3-9. Property with code in its get and set public double SpeedInKilometersPerHour { get { return SpeedInMilesPerHour * 1.609344; } set { SpeedInMilesPerHour = value / 1.609344; } } We’ve done something different here—rather than just writing get; and set; we’ve provided code for these accessors. This is another reason we have to declare the acces- sors explicitly—the C# compiler needs to know whether we want to write a custom property implementation. We don’t want to use an ordinary property in Example 3-9, because our SpeedInKilo metersPerHour is not really a property in its own right—it’s an alternative representation for the information stored in the SpeedInMilesPerHour property. If we used the normal property syntax for both, it would be possible to set the speed as being both 100 mph and 400 km/h, which would clearly be inconsistent. So instead we’ve chosen to im- plement SpeedInKilometersPerHour as a wrapper around the SpeedInMilesPerHour property. If you look at the getter, you’ll see that it returns a value of type double. It is equivalent to a function with this signature: public double get_SpeedInKilometersPerHour() 70 | Chapter 3: Abstracting Ideas with Classes and Structs The setter seems to provide an invisible parameter called value, which is also of type double. So it is equivalent to a method with this signature: public void set_SpeedInKilometersPerHour(double value) This value parameter is a contextual keyword—C# only considers it to be a keyword in property or event accessors. (Events are described in Chapter 5.) This means you’re allowed to use value as an identifier in other contexts—for example, you can write a method that takes a pa- rameter called value. You can’t do that with other keywords—you can’t have a parameter called class, for example. This is a very flexible system indeed. You can provide properties that provide real stor- age in the class to store their data, or calculated properties that use any mechanism you like to get and/or set the values concerned. This choice is an implementation detail hidden from users of our class—we can switch between one and the other without changing our class’s public face. For example, we could switch the implementation of these speed properties around so that we stored the value in kilometers per hour, and calculated the miles per hour—Example 3-10 shows how these two properties would look if the “real” value was in km/h. Example 3-10. Swapping over the real and calculated properties public double SpeedInMilesPerHour { get { return SpeedInKilometersPerHour / 1.609344; } set { SpeedInKilometersPerHour = value * 1.609344; } } public double SpeedInKilometersPerHour { get; set; } As far as users of the Plane class are concerned, there’s no discernible difference between the two approaches—the way in which properties work is an encapsulated implemen- tation detail. Example 3-11 shows an updated Main function that uses the new prop- erties. It neither knows nor cares which one is the “real” one. Defining Classes | 71 Example 3-11. Using the speed properties static void Main(string[] args) { Plane someBoeing777 = new Plane("BA0049"); someBoeing777.SpeedInMilesPerHour = 150.0; Console.WriteLine( "Your plane has identifier {0}, " + "and is traveling at {1:0.00}mph [{2:0.00}kph]", someBoeing777.Identifier, someBoeing777.SpeedInMilesPerHour, someBoeing777.SpeedInKilometersPerHour); someBoeing777.SpeedInKilometersPerHour = 140.0; Console.WriteLine( "Your plane has identifier {0}, " + "and is traveling at {1:0.00}mph [{2:0.00}kph]", someBoeing777.Identifier, someBoeing777.SpeedInMilesPerHour, someBoeing777.SpeedInKilometersPerHour); Console.ReadKey(); } Although our public API supports two different units for speed while successfully keeping the implementation for that private, there’s something unsatisfactory about that implementation. Our conversion relies on a magic number (1.609344) that appears repeatedly. Repetition impedes readability, and is prone to typos (I know that for a fact. I’ve typed it incorrectly once already this morning while preparing the example!) There’s an important principle in programming: don’t repeat yourself (or dry, as it’s sometimes abbreviated). Your code should aim to express any single fact or concept no more than once, because that way, you only need to get it right once. It would be much better to put this conversion factor in one place, give it a name, and refer to it by that instead. We can do that by declaring a field. Fields: A Place to Put Data A field is a place to put some data of a particular type. There’s no option to add code like you can in a property—a field is nothing more than data. Back before C# 3.0 the compiler didn’t let us write just get; and set;—we always had to write properties with code as in Example 3-9, and if we wanted a simple property that stored a value, we had to provide a field, with code such as Example 3-12. 72 | Chapter 3: Abstracting Ideas with Classes and Structs Example 3-12. Writing your own simple property // Field to hold the SpeedInMilesPerHour property's value double speedInMilesPerHourValue; public double SpeedInMilesPerHour { get { return speedInMilesPerHourValue; } set { speedInMilesPerHourValue = value; } } When you write just get; and set; as we did in Example 3-8, the C# compiler generates code that’s more or less identical to Example 3-12, except it gives the field a peculiar name to prevent us from accessing it directly. (These compiler-generated properties are called auto properties.) So, if we want to store a value in an object, there’s always a field involved, even if it’s a hidden one provided automatically by the compiler. Fields are the only class members that can hold information—properties are really just methods in disguise. As you can see, a field declaration looks similar to the start of a property declaration. There’s the type (double), and a name. By convention, this name is camelCased, to make fields visibly different from properties. (Some developers like to distinguish fields further by giving them a name that starts with an underscore.) We can modify a field’s protection level if we want, but, conventionally, we leave all fields with the default private accessibility. That’s because a field is just a place for some data, and if we make it public, we lose control over the internal state of our object. Properties always involve some code, even if it’s generated automatically by the com- piler. We can use private backing fields as we wish, or calculate property values any way we like, and we’re free to modify the implementation without ever changing the public face of the class. But with a field, we have nowhere to put code, so if we decide to change our implementation by switching from a field to a calculated value, we would need to remove the field entirely. If the field was part of the public contract of the class, that could break our clients. In short, fields have no innate capacity for encapsulation, so it’s a bad idea to make them public. Example 3-13 shows a modified version of the Plane class. Instead of repeating the magic number for our speed conversion factor, we declare a single field initialized to the required value. Not only does this mean that we get to state the conversion value just once, but we’ve also been able to give it a descriptive name—in the conversions, it’s now obvious that we’re multiplying and dividing by the number of kilometers in a mile, even if you happen not to have committed the conversion factor to memory. Defining Classes | 73 Example 3-13. Storing the conversion factor in a field class Plane { // Constructor with a parameter public Plane(string newIdentifier) { Identifier = newIdentifier; } public string Identifier { get; private set; } double kilometersPerMile = 1.609344; public double SpeedInMilesPerHour { get { return SpeedInKilometersPerHour / kilometersPerMile; } set { SpeedInKilometersPerHour = value * kilometersPerMile; } } public double SpeedInKilometersPerHour { get; set; } } Notice how we’re able to initialize the field to a default value right where we declare it, by using the = operator. (This sort of code is called, predictably enough, a field initial- izer.) Alternatively, we could have initialized it inside a constructor, but if the default is a constant value, it is conventional to set it at the point of declaration. What about the first example of a field that we saw—the one we used as the backing data for a property in Example 3-12? We didn’t explicitly initialize it. In some other languages that would be a ghastly mistake. (Failure to initialize fields correctly is a major source of bugs in C++, for example.) Fortunately, the designers of .NET decided that the trade-off between performance and robustness wasn’t worth the pain, and kindly initialize all fields to a default value for us—numeric fields are set to zero and fields of other types get whatever the nearest equivalent of zero is. (Boolean fields are initialized to false, for example.) 74 | Chapter 3: Abstracting Ideas with Classes and Structs There’s also a security reason for this initialization. Because a new ob- ject’s memory is always zeroed out before we get to see it, we can’t just allocate a whole load of objects and then peer at the “uninitialized” values to see if anything interesting was left behind by the last object that used the same memory. Defining a field for our scale factor is an improvement, but we could do better. Our 1.609344 isn’t ever going to change. There are always that many kilometers per mile, not just for this instance of a Plane, but for any Plane there ever will be. Why allocate the storage for the field in every single instance? Wouldn’t it be better if we could define this value just once, and not store it in every Plane instance? Fields Can Be Fickle, but const Is Forever C# provides a mechanism for declaring that a field holds a constant value, and will never, ever change. You use the const modifier, as Example 3-14 shows. Example 3-14. Defining a constant value const double kilometersPerMile = 1.609344; The platform now takes advantage of the fact that this can never change, and allocates storage for it only once, no matter how many instances of Plane you new up. Handy. This isn’t just a storage optimization, though. By making the field const, there’s no danger that someone might accidentally change it for some reason inside another func- tion he’s building in the class—the C# compiler prevents you from assigning a value to a const field anywhere other than at the point of declaration. In general, when we are developing software, we’re trying to make it as easy as possible for other developers (including our “future selves”) to do the right thing, almost by accident. You’ll often hear this approach called “designing for the pit of success.” The idea is that people will fall into doing the right things because of the choices you’ve made. Some aspects of an object don’t fit well as either a normal modifiable field or a constant value. Take the plane’s identifier, for example—that’s fixed, in the sense that it never changes after construction, but it’s not a constant value like kilometersPerMile. Dif- ferent planes have different identifiers. .NET supports this sort of information through read-only properties and fields, which aren’t quite the same as const. Defining Classes | 75 Read-only Fields and Properties In Example 3-5, we made our Plane class’s Identifier property private. This prevented users of our class from setting the property, but our class is still free to shoot itself in the foot. Suppose a careless developer added some code like that in Example 3-15, which prints out messages in the SpeedInMilesPerHour property perhaps in order to debug some problem he was investigating. Example 3-15. Badly written debugging code public double SpeedInMilesPerHour { get { return SpeedInKilometersPerHour / kilometersPerMile; } set { Identifier += ": speed modified to " + value; Console.WriteLine(Identifier); SpeedInKilometersPerHour = value * kilometersPerMile; } } The first time someone tries to modify a plane’s SpeedInMilesPerHour this will print out a message that includes the identifier, for example: BA0048: speed modified to 400 Unfortunately, the developer who wrote this clearly wasn’t the sharpest tool in the box—he used the += operator to build that debug string, which will end up modifying the Identifier property. So, the plane now thinks its identifier is that whole text, in- cluding the part about the speed. And if we modified the speed again, we’d see: BA0048: speed modified to 400: speed modified to 380 While it might be interesting to see the entire modification history, the fact that we’ve messed up the Identifier is bad. Example 3-15 was able to do this because the SpeedInMilesPerHour property is part of the Plane class, so it can still use the private setter. We can fix this (up to a point) by making the property read-only—rather than merely making the setter private, we can leave it out entirely. However, we can’t just write the code in Example 3-16. Example 3-16. The wrong way to define a read-only property class Plane { // Wrong! public string Identifier { get; } 76 | Chapter 3: Abstracting Ideas with Classes and Structs ... } That won’t work because there’s no way we could ever set Identifier—not even in the constructor. Auto properties cannot be read-only, so we must write a getter with code. Example 3-17 will compile, although as we’re about to see, the job’s not done yet. Example 3-17. A better, but incomplete, read-only property class Plane { public Plane(string newIdentifier) { _identifier = newIdentifier; } public string Identifier { get { return _identifier; } } private string _identifier; ... } This turns out to give us two problems. First, the original constructor from Exam- ple 3-6 would no longer compile—it set Identifier, but that’s now read-only. That was easy to fix, though—Example 3-17 just sets the explicit backing field we’ve added. More worryingly, this hasn’t solved the original problem—the developer who wrote the code in Example 3-15 has “cleverly” realized that he can “fix” his code by doing exactly the same thing as the constructor. As Example 3-18 shows he has just used the _identifier field directly. Example 3-18. “Clever” badly written debugging code public double SpeedInMilesPerHour { get { return SpeedInKilometersPerHour / kilometersPerMile; } set { _identifier += ": speed modified to " + value; Console.WriteLine(Identifier); SpeedInKilometersPerHour = value * kilometersPerMile; } } That seemed like a long journey for no purpose. However, we can fix this problem— we can modify the backing field itself to be read-only, as shown in Example 3-19. Defining Classes | 77 Example 3-19. A read-only field private readonly string _identifier; That will foil the developer who wrote Example 3-15 and Example 3-18. But doesn’t it also break our constructor again? In fact, it doesn’t: read-only fields behave differently from read-only properties. A read-only property can never be modified. A read-only field can be modified, but only by a constructor. Since read-only fields only become truly read-only after construction completes, it makes them perfect for properties that need to be able to be different from one instance to another, but which need to be fixed for the lifetime of an instance. Before we move on from const and readonly fields, there’s another property our Plane needs for which const seems like it could be relevant, albeit in a slightly different way. In addition to monitoring the speed of an aircraft, we also need to know whether it is approaching or heading away from the airport. We could represent that with a bool property called something like IsApproaching (where true would mean that it was approaching, and false would, by implication, indicate that it was heading away). That’s a bit clumsy, though. You can often end up having to negate Boolean properties—you might need to write this sort of thing: if (!plane.IsApproaching) { ... } That reads as “if not plane is approaching” which sounds a bit awkward. We could go with: if (somePlane.IsApproaching == false) { ... } That’s “if is approaching is false” which isn’t much better. We could offer a second, calculated property called IsNotApproaching, but our code is likely to be simpler and easier to read (and therefore likely to contain fewer bugs) if, instead of using bool, we have a Direction property whose value could somehow be either Approaching or Leaving. We’ve just seen a technique we could use for that. We could create two constant fields of any type we like (int, for example), and a property of type int called Direction (see Example 3-20). Example 3-20. Named options with const int class Plane { public const int Approaching = 0; public const int Leaving = 1; // ... public int Direction { get; set; } } 78 | Chapter 3: Abstracting Ideas with Classes and Structs This lets us write code that reads a bit more naturally than it would if we had used just true and false: someBoeing777.Direction = Plane.Approaching; if (someAirbusA380.Direction == Plane.Leaving) { /* Do something */ } But there’s one problem: if our Direction property’s type is int, there’s nothing to stop us from saying something like: someBoeing777.Direction = 72; This makes no sense, but the C# compiler doesn’t know that—after all, we told it the property’s type was int, so how’s it supposed to know that’s wrong? Fortunately, the designers of C# have thought of this, and have given us a kind of type for precisely this situation, called an enum, and it turns out to be a much better solution for this than const int. Related Constants with enum The enum† keyword lets us define a type whose values can be one of a fixed set of possibilities. Example 3-21 declares an enum for our Direction property. You can add this to an existing source file, above or below the Plane class, for example. Alternatively, you could add a whole new source file to the project, although Visual Studio doesn’t offer a file template for enum types, so either you’d have to add a new class and then change the class keyword to enum, or you could use the Code File template to add a new, empty source file. Example 3-21. Direction enum enum DirectionOfApproach { Approaching, Leaving } This is similar in some respects to a class declaration. We can optionally begin with a protection level but if, like Example 3-21, we omit that, we get internal protection by default. Then there’s the enum specifier itself, followed by the name of the type, which by convention we PascalCase. Inside the braces, we declare the members, again using PascalCasing. Notice that we use commas to separate the list of constants—this is where the syntax starts to part company with class. Unusually, the members are pub- licly accessible by default. That’s because an enum has no behavior, and so there are no implementation details—it’s just a list of named values, and those need to be public for the type to serve any useful purpose. † It’s short for “enumeration,” by the way. So it’s often pronounced “e-noom” or, depending on where you’re from, “e-nyoom.” However, some developers (and one of the authors) ignore the etymology and pronounce it “ee numb” because that’s how it looks like it should sound. Related Constants with enum | 79 Notice that we’ve chosen to call this DirectionOfApproach, and not the plural DirectionsOfApproach. By convention, we give enum types a sin- gular name even though they usually contain a list. This makes sense because when you use named entries from an enumeration, you use them one at a time, and so it would look odd if the type name were plural. Obviously, there won’t be any technical consequences for break- ing this convention, but following it helps make your code consistent with the .NET Framework class libraries. We can now declare our Direction property, using the enumeration instead of an in- teger. Example 3-22 shows the property to add to the Plane class. Example 3-22. Property with enum type public DirectionOfApproach Direction { get; set; } There are some optional features we can use in an enum declaration. Example 3-23 uses these, and they provide some insight into how enum types work. Example 3-23. Explicit type and values for enum enum DirectionOfApproach : int { Approaching = 0, Leaving = 1 } In this declaration, we have explicitly specified the governing type for the enumeration. This is the type that stores the individual values for an enumeration, and we specify it with a colon and the type name. By default, it uses an int (exactly as we did in our original const-based implementation of this property), so we’ve not actually changed anything here; we’re just being more explicit. The governing type must be one of the built-in integer types: byte, sbyte, short, ushort, uint, long, or ulong. Example 3-23 also specifies the numbers to use for each named value. As it happens, if you don’t provide these numbers, the first member is assigned the value 0, and we count off sequentially after that, so again, this example hasn’t changed anything, it’s just showing the values explicitly. We could, if we wanted, specify any value for any particular member. Maybe we start from 10 and go up in powers of 2. And we’re also free to define duplicates, giving the same value several different names. (That might not be useful, but C# won’t stop you.) We normally leave all these explicit specifiers off, and accept the defaults. However, the sidebar on the next page describes a scenario in which you would need to control the numbers. 80 | Chapter 3: Abstracting Ideas with Classes and Structs Bit Fields with [Flags] You can create a special kind of enum called a [Flags] enum, also known as a bit field. A bit field is just an ordinary numeric value used in a particular way. When you view a bit field value in binary, each bit represents a particular setting. For example, we could define a bit field to represent the toppings on a bowl of ice cream. We might use the least significant bit to indicate whether a chocolate sauce topping is required. And we could use a different bit to indicate whether chocolate sprinkles are required. The thing that makes bit field enum types different from normal ones is that you can use any combination of values. Because each value gets a whole bit of the number to itself, you can choose for that bit to be either 0 or 1 independently of the value of any other bits. You indicate that your enum works this way by annotating it with a [Flags] attribute, and specifying the values of the members to correspond to the relevant bit patterns. (Actually, the [Flags] attribute turns out to be optional—the compiler ignores it, and lets you use any enum as though it were a bit field. The .NET Framework only uses the attribute to work out how to convert enumeration values to text. However, it’s a useful signpost to tell other developers how your enum is meant to be used.) Typically, you define a name for each bit, and you can also name some common combinations: [Flags] enum Toppings { None = 0x00, // Special zero value ChocolateSauce = 0x01, ToffeeSauce = 0x02, ChocolateSprinkles = 0x04, Chocoholic = 0x05, // Combined value, sets 2 bits Greedy = 0x07 // Everything! } We’re using hexadecimal representations because it’s easier to relate them to the binary values—each hex digit corresponds exactly to four binary digits. We can combine the values together using the | operator (binary OR), for example: // (011) Toppings saucy = Toppings.ChocolateSauce | Toppings.ToffeeSauce; We can use the binary AND operator (&) to see whether a particular flag has been set: static bool DoYouWantChocolateSauceWithThat(Toppings t) { return (t & Toppings.ChocolateSauce) != 0; } When defining bit fields, you might not want to allow certain combinations. For ex- ample, you might reject the saucy combination, requiring customers to pick, at most, one kind of sauce. Unfortunately, there are no language or platform mechanisms for enforcing that kind of constraint, so you’d need to write code to check for illegal com- binations in any method that accepted arguments of this type. (Or you could consider an alternative design that does not use an enum at all.) Related Constants with enum | 81 If you don’t specify explicit values, the first item in your list is effectively the default value for the enum (because it corresponds to the zero value). If you provide explicit values, be sure to define a value that corresponds to zero—if you don’t, fields using your type will default to a value that’s not a valid member of the enum, which is not desirable. We can now access the enumeration property like this: someBoeing777.Direction = DirectionOfApproach.Approaching; We’ve clearly made some progress with our Plane class, but we’re not done yet. We have a read-only property for its Identifier. We can store the speed, which we can get and set using two different properties representing different units, using a const field for the conversion factor. And we know the direction, which will be either the Approach ing or the Leaving member of an enum. We still need to store the aircraft’s position. According to the specification, we’ve got two polar coordinates (an angle and a distance) for its position on the ground, and another value for its height above sea level. We’re likely to need to do a lot of calculations based on this position information. Every time we want to create a function to do that, we’d need three parameters per point, which seems overly complex. (And error-prone—it’d be all too easy to inadvertently pass two numbers from one position, and a third number from a different position.) It would be nicer if we could wrap the numbers up into a single, lightweight, “3D point” type that we can think of in the same kind of way we do int or double—a basic building block for other classes to use with minimum overhead. This is a good candidate for a value type. Value Types and Reference Types So far, we’ve been building a class. When creating an instance of the class, we stored it in a named variable, as Example 3-24 shows. Example 3-24. Storing a reference in a variable Plane someBoeing777 = new Plane("BA0049"); someBoeing777.Direction = DirectionOfApproach.Approaching; We can define another variable with a different name, and store a reference to the same plane in that new variable, as shown in Example 3-25. Example 3-25. Copying a reference from one variable to another Plane theSameBoeing777ByAnotherName = someBoeing777; 82 | Chapter 3: Abstracting Ideas with Classes and Structs If we change a property through one variable, that change will be visible through the other. Example 3-26 modifies our plane’s Direction property through the second var- iable, but then reads it through the first variable, verifying that they really are referring to the same object. Example 3-26. Using one object through two variables theSameBoeing777ByAnotherName.Direction = DirectionOfApproach.Leaving; if (someBoeing777.Direction == DirectionOfApproach.Leaving) { Console.WriteLine("Oh, they are the same!"); } As Shakespeare might have said, if only he’d found his true vocation as a C# developer: That which we call someBoeing777 By any other name would smell as sweet. Assuming you like the smell of jet fuel. When we define a type using class, we always get this behavior—our variables behave as references to an underlying object. We therefore call a type defined as a class a reference type. It’s possible for a reference type variable to be in a state where it isn’t referring to any object at all. C# has a special keyword, null, to represent this. You can set a variable to null, or you can pass null as an argument to a method. And you can also test to see if a field, variable, or argument is equal to null in an if statement. Any field whose type is a reference type will automatically be initialized to null before the constructor runs, in much the same way as numeric fields are initialized to zero. The enum we declared earlier and the built-in numeric types (int, double) behave dif- ferently, though, as Example 3-27 illustrates. Example 3-27. Copying values, not references int firstInt = 3; int secondInt = firstInt; secondInt = 4; if (firstInt != 4) { Console.WriteLine("Well. They're not the same at all."); } When we assign firstInt to secondInt, we are copying the value. In this case, the var- iables hold the actual value, not a reference to a value. We call types that behave this way value types. Value Types and Reference Types | 83 People often refer to reference types as being allocated “on the heap” and value types “on the stack.” C++ programmers will be familiar with these concepts, and C++ pro- vided one syntax in the language to explicitly create items on the stack (a cheap form of storage local to a particular scope), and a different syntax for working on the heap (a slightly more expensive but sophisticated form of storage that could persist beyond the current scope). C# doesn’t make that distinction in its syntax, because the .NET Framework itself makes no such distinction. These aspects of memory management are completely opaque to the developer, and it is actively wrong to think of value types as being always allocated on a stack. For people familiar with C++ this can take a while to get used to, especially as the myth is perpetuated on the Web, in the MSDN documentation and elsewhere. (For example, at the time of this writing, http://msdn.microsoft.com/library/aa288471 states that structs are created on the stack, and while that happens to be true of the ones in that example when running against the current version of .NET, it would have been helpful if the page had mentioned that it’s not always true. For example, if a class has a field of value type, that field doesn’t live on the stack—it lives inside the object, and in all the versions of .NET released so far, objects live on the heap.) The important difference for the C# developer between these two kinds of types is the one of reference versus copy semantics. As well as understanding the difference in behavior, you also need to be aware of some constraints. To be useful, a value type should be: • Immutable • Lightweight Something is immutable if it doesn’t change over time. So, the integer 3 is immutable. It doesn’t have any internal workings that can change its “three-ness”. You can replace the value of an int variable that currently contains a 3, by copying a 4 into it, but you can’t change a 3 itself. (Unlike, say, a particular Plane object, which has a Direction property that you can change anytime you like without needing to replace the whole Plane.) There’s nothing in C# that stops you from creating a mutable value type. It is just a bad idea (in general). If your type is mutable, it is prob- ably safer to make it a reference type, by declaring it as a class. Mutable value types cause problems because of the copy semantics—if you mod- ify a value, it’s all too easy to end up modifying the wrong one, because there may be many copies. 84 | Chapter 3: Abstracting Ideas with Classes and Structs It should be fairly apparent that a value type also needs to be pretty lightweight, because of all that copying going on. Every time you pass it into a function, or assign it to a variable, a copy is made. And copies are generally the enemy of good performance. If your value type consists of more than two or three of the built-in types, it may be getting too big. These constraints mean it is very rare that you will actually want to declare a value type yourself. A lot of the obviously useful ones you might want are already defined in the .NET Framework class libraries (things like 2D points, times, and dates). Custom value types are so rare that it was hard to come up with a useful example for this book that wasn’t already provided in the class libraries. (If you were wondering why our example application represents aircraft positions in such an idiosyncratic fashion, this is the reason.) But that doesn’t mean you should never, ever declare a value type. Value types can have performance benefits when used in arrays (although as with most performance issues, this is not entirely clear-cut), and the immutability and copy semantics can make them safer when passing them in to functions—you won’t normally introduce side effects by working with a value type because you end up using a copy, rather than modifying shared data that other code might be relying on. Our polar 3D point seems to comply with the requirements. Any given point is just that: a specific point in 3D space—a good candidate for immutability. (We might want to move a plane to a different point, but we can’t change what a particular point means.) It is also no more than three doubles in size, which is small enough for copy semantics. Example 3-28 shows our declaration of this type, which we can add to our project. (As with enum, Visual Studio doesn’t offer a template for value types. Again, we can use the Class template, replacing the class with the code we want.) Example 3-28. A value type struct PolarPoint3D { public PolarPoint3D(double distance, double angle, double altitude) { Distance = distance; Angle = angle; Altitude = altitude; } public double Distance { get; private set; } public double Angle { get; private set; Value Types and Reference Types | 85 } public double Altitude { get; private set; } } If you think that it looks just like a class declaration, but using the struct keyword instead of class, you’d be right—these two kinds of types are very similar. However, if we try to compile it, we get an error on the first line of the constructor: The 'this' object cannot be used before all of its fields are assigned to So, although the basic syntax of a struct looks just like a class there are important differences. Remember that when you allocate an instance of a particular type, it is always initialized to some default value. With classes, all fields are initialized to zero (or the nearest equivalent value). But things work slightly differently with value types— we need to do slightly more work. Anytime we write a struct, C# automatically generates a default, parameterless con- structor that initializes all of our storage to zero, so if we don’t want to write any custom constructors, we won’t have any problems. (Unlike with a class, we aren’t allowed to replace the default constructor. We can define extra constructors, but the default con- structor is always present and we’re not allowed to write our own—see the sidebar on the next page for details.) Example 3-28 has hit trouble because we’re trying to provide an additional constructor, which initializes the properties to particular values. If we write a constructor in a struct, the compiler refuses to let us invoke any methods until we’ve initialized all the fields. (It doesn’t do the normal zero initialization for custom constructors.) This re- striction turns out to include properties, because get and set accessors are methods under the covers. So C# won’t let us use our properties until the underlying fields have been initialized, and we can’t do that because these are auto properties—the C# com- piler has generated hidden fields that we can only access through the properties. This is a bit of a chicken-and-egg bootstrapping problem! Fortunately, C# gives us a way of calling one of our constructors from another. We can use this to call the default constructor to do the initialization; then our constructor can set the properties to whatever values it wishes. We call the constructor using the this keyword, and the standard function calling syntax with any arguments enclosed in parentheses. As Example 3-29 shows, we can invoke the default constructor with an empty argument list. 86 | Chapter 3: Abstracting Ideas with Classes and Structs Value Types and Default Constructors Why aren’t we allowed to define a custom default constructor for a value type, given that we’re allowed to do that for a reference type? The short answer is that the speci- fication for the relevant behavior in the .NET Framework doesn’t let you. (The speci- fication in question is called the Common Language Infrastructure [CLI], incidentally.) The slightly longer answer is: for efficiency reasons. By mandating that the default constructor for any value type always initializes everything to zero, large arrays of value types can be constructed very cheaply, just by allocating the required amount of mem- ory and zeroing out the whole array in one step. And similarly, it simplifies the initial- ization of fields and variables—everything can be initialized to zero. Example 3-29. Calling one constructor from another public PolarPoint3D(double distance, double angle, double altitude) : this() { Distance = distance; Angle = angle; Altitude = altitude; } You add the call just before the opening brace for the body of the constructor, and prefix it with a colon. We can also use this technique to avoid writing common initi- alization code multiple times. Say we wanted to provide another utility constructor that just took the polar coordinates, and initialized the altitude to zero by default. Instead of repeating all the code from the first constructor, we could just add this extra con- structor to our definition for PolarPoint3D, as shown in Example 3-30. Example 3-30. Sharing common initialization code public PolarPoint3D(double distance, double angle) : this(distance, angle, 0) { } public PolarPoint3D( double distance, double angle, double altitude) : this() { Distance = distance; Angle = angle; Altitude = altitude; } Incidentally, this syntax for calling one constructor from another works equally well in classes, and is a great way of avoiding code duplication. Value Types and Reference Types | 87 Too Many Constructors, Mr. Mozart You should be careful of adding too many constructors to a class or struct. It is easy to lose track of which parameters are which, or to make arbitrary choices about which constructors you provide and which you don’t. For example, let’s say we wanted to add yet another constructor to PolarPoint3D that lets callers pass just the angle and altitude, initializing the distance to a default of zero, as Example 3-31 shows. Example 3-31. A constructor too far public PolarPoint3D( double altitude, double angle ) : this( 0, angle, altitude ) { } Even before we compile, we can see that there’s a problem—we happen to have added the altitude parameter so that it is the first in the list, and angle stays second. In our main constructor, the altitude comes after the angle. Because they are both just doubles, there’s nothing to stop you from accidentally passing the parameters “the wrong way round.” This is the exactly the kind of thing that surprises users of your class, and leads to hard-to-find bugs. But while inconsistent parameter ordering is bad design, it’s not a showstopper. However, when we compile, things get even worse. We get another error: Type 'PolarPoint3D' already defines a member called 'PolarPoint3D' with the same parameter types We have too many constructors. But how many is too many? Overloading When we define more than one member in a type with the same name (be it a con- structor or, as we’ll see later, a method) we call this overloading. Initially, we created two constructors (two overloads of the constructor) for Polar Point3D, and they compiled just fine. This is because they took different sets of param- eters. One took three doubles, the other two. In fact, there was also the third, hidden constructor that took no parameters at all. All three constructors took different num- bers of parameters, meaning there’s no ambiguity about which constructor we want when we initialize a new PolarPoint3D. The constructor in Example 3-31 seems different: the two doubles have different names. Unfortunately, this doesn’t matter to the C# compiler—it only looks at the types of the parameters, and the order in which they are declared. It does not use names for 88 | Chapter 3: Abstracting Ideas with Classes and Structs disambiguation. This should hardly be surprising, because we’re not required to pro- vide argument names when we call methods or constructors. If we add the overload in Example 3-31, it’s not clear what new PolarPoint3D(0, 0) would mean, and that’s why we get an error—we’ve got two members with the same name (PolarPoint3D—the constructor), and exactly the same parameter types, in the same order. Looking at overloaded functions will emphasize that it really is only the method name and the parameters that matter—a function’s return type is not considered to be a disambiguating aspect of the member for overload purposes. That means there’s nothing we can do about it: we’re going to have to get rid of this third constructor (just delete it); and while we’re in the code, we’ll finish up the dec- laration of the data portion of our Plane by adding a property for its position, shown in Example 3-32. Example 3-32. Using our custom value type for a property public PolarPoint3D Position { get; set; } Overloaded Methods and Default Named Parameters Just as with constructors, we can provide more than one method with the same name, but a different list of parameter types. It is, in general, a bad idea to provide two over- loads with the same name if they perform a semantically different operation (again— that’s the kind of thing that surprises developers using your class), so the most common reason for overloading is to provide several different ways to do something. We can provide users of our code with flexible methods that take lots of arguments to control different aspects of the code, and we can also provide developers that don’t need this flexibility with simpler options by providing overloads that don’t need as many arguments. Suppose we added a method to our Plane class enabling messages to be sent to aircraft. Perhaps in our first attempt we define a method whose signature looks like this: public void SendMessage(string messageText) But suppose that as the project progresses, we find that it would be useful to be able to delay transmission of certain messages. We could modify the SendMessage method so that it accepts an extra argument. There’s a handy type in the framework called TimeSpan which lets us specify duration. We could modify the method to make use of it: public void SendMessage(string messageText, TimeSpan delay) Alas! If we already had code in our project depending on the original signature, we’d start to see this compiler error: No overload for method 'SendMessage' takes '1' arguments Overloading | 89 We’ve changed the signature of that method, so all our clients are sadly broken. They need to be rewritten to use the new method. That’s not great. A better alternative is to provide both signatures—keep the old single-parameter con- tract around, but add an overload with the extra argument. And to ensure that the overloads behave consistently (and to avoid duplicating code) we can make the simpler method call the new method as its actual implementation. The old method was just the equivalent of calling the new method with a delay of zero, so we could replace it with the method shown in Example 3-33. This lets us provide the newly enhanced SendMessage, while continuing to support the old, simpler version. Example 3-33. Implementing one overload in terms of another public void SendMessage(string messageName) { SendMessage(messageName, TimeSpan.Zero); } (TimeSpan.Zero is a static field that returns a duration of zero.) Until C# 4.0 that’s as far as we could go. However, the C# designers noticed that a lot of member overloads were just like this one: facades over an über-implementation, with a bunch of parameters defaulted out to particular values. So they decided to make it easier for us to support multiple variations on the same method. Rather than writing lots of overloads, we can now just specify default values for a method’s arguments, which saves us typing a lot of boilerplate, and helps make our default choices more transparent. Let’s take out the single-parameter method overload we just added, and instead change the declaration of our multiparameter implementation, as shown in Example 3-34. Example 3-34. Parameter with default value public void SendMessage( string messageName, TimeSpan delay = default(TimeSpan)) Even though we’ve only got one method, which supports two arguments, code that tries to call it with a single argument will still work. That’s because default values can fill in for missing arguments. (If we tried to call SendMessage with no arguments at all, we’d get a compiler error, because there’s no default for the first argument here.) But it doesn’t end there. Say we had a method with four parameters, like this one: public void MyMethod( int firstOne, double secondInLine = 3.1416, string thirdHere = "The third parameter", TimeSpan lastButNotLeast = default(TimeSpan)) { // ... } 90 | Chapter 3: Abstracting Ideas with Classes and Structs If we want to call it and specify the first parameter (which we have to, because it has no default), and the third, but not the second or the fourth, we can do so by using the names of the parameters, like this: MyMethod(127, thirdHere: "New third parameter"); With just one method, we now have many different ways to call it—we can provide all the arguments, or just the first and second, or perhaps the first, second, and third. There are many combinations. Before named arguments and defaults were added in C# 4.0, the only way to get this kind of flexibility was to write an overload for each distinct combination. Under the Hood with Default Parameters Default and named parameters are very useful features, but we need to warn you of a subtle potential problem. Although they are more-or-less equivalent to providing a bunch of different function overloads, as far as the syntax for the caller goes, under the covers, they are implemented very differently. The compiler marks a parameter to indicate that it is optional using the OptionalAttri bute and there’s a DefaultParameterValueAttribute to specify a default value. These two attributes have been around for quite a while—they were originally added for the benefit of Visual Basic, long before C# started using them. (Attributes are discussed in Chapter 17.) When you call a method (or constructor), the C# compiler always emits a complete call—the compiled code passes a full set of arguments to the method, even if your source code left some arguments out. For example, in our Plane example, if you wrote: SendMessage("SomeMessage"); but the Plane class only has the method shown in Example 3-34, the compiler actually generates code equivalent to this: SendMessage("SomeMessage", default(TimeSpan)); In other words, it plugs in the default value at compile time. This means if you’re using some external library that uses default values, and a newer version of the library comes out that changes the default values for some method or constructor, your code won’t pick up those new values unless you recompile your code. There’s also a subtler problem you can run into. Some parts of the .NET Framework require you to provide a particular constructor overload. For example, it you write a custom control for WPF, and you want to use it from Xaml, it must have a default constructor. (WPF and Xaml are described in Chapter 20.) If all your constructors take parameters, then even if you provide default values for all the parameters, that’s not good enough. You can write, say, new MyControl() in C#, but only because the C# compiler is implicitly passing the missing values for you. Not everything in the world of .NET understands the concept of default arguments. (C# itself didn’t until version 4.0.) Sometimes only a genuine no-arguments constructor will do. Overloading | 91 This is not just limited to normal methods—you can use this same syntax to provide default values for parameters in your constructors, if you wish. Being forced to delete the extra constructor we tried to add back in Example 3-31 was a little disappointing—we’re constraining the number of ways users of our type can initialize it. Named arguments and default values have helped, but can we do more? Object Initializers Until C# 3.0, the only real solution to this was to write one or more factory methods. These are described in the sidebar below. But now we have another option. Factory Methods A factory method is a static method that builds a new object. There’s no formal support for this in C#, it’s just a common solution to a problem—a pattern, as popular idioms are often called in programming. We can get around the overload ambiguity problems by providing factory methods with different names. And the names can make it clear how we’re initializing the instance: public static PolarPoint3D FromDistanceAndAngle( double distance, double angle) { return new PolarPoint3D(distance, angle, 0); } public static PolarPoint3D FromAngleAndAltitude( double angle, double altitude) { return new PolarPoint3D(0, angle, altitude); } We rather like this approach, although some people frown on it as insufficiently dis- coverable. (Most developers aren’t expecting to find static methods that act rather like constructors, and if nobody finds these methods, we’re wasting our time in providing them.) However, this pattern is used all over the .NET Framework libraries—DateTime, TimeSpan, and Color are popular types that all use this technique. With C# 3.0 the language was extended to support object initializers—an extension to the new syntax that lets us set up a load of properties, by name, as we create our object instance. Example 3-35 shows how an object initializer looks when we use it in our Main function. 92 | Chapter 3: Abstracting Ideas with Classes and Structs Example 3-35. Using object initializers static void Main(string[] args) { Plane someBoeing777 = new Plane("BA0049") { Direction = DirectionOfApproach.Approaching, SpeedInMilesPerHour = 150 }; Console.WriteLine( "Your plane has identifier {0}," + " and is traveling at {1:0.00}mph [{2:0.00}kph]", // Use the property getter someBoeing777.Identifier, someBoeing777.SpeedInMilesPerHour, someBoeing777.SpeedInKilometersPerHour); someBoeing777.SpeedInKilometersPerHour = 140.0; Console.WriteLine( "Your plane has identifier {0}," + " and is traveling at {1:0.00}mph [{2:0.00}kph]", // Use the property getter someBoeing777.Identifier, someBoeing777.SpeedInMilesPerHour, someBoeing777.SpeedInKilometersPerHour); Console.ReadKey(); } Object initializers are mostly just a convenient syntax for constructing a new object and then setting some properties. Consequently, this only works with writable properties—you can’t use it for immutable types,‡ so this wouldn’t work with our PolarPoint3D. We still use the constructor parameter for the read-only Identifier property; but then we add an extra section in braces, between the closing parenthesis and the semicolon, in which we have a list of property assignments, separated by commas. What’s partic- ularly interesting is that the purpose of the constructor parameter is normally identifiable only by the value we happen to assign to it, but the object initializer is “self-documenting”—we can easily see what is being initialized to which values, at a glance. ‡ This is a slight oversimplification. In Chapter 8, we’ll encounter anonymous types, which are always immutable, and yet we can use object initializers with those. In fact, we are required to. But anonymous types are a special case. Object Initializers | 93 The job isn’t quite done yet, though. While there’s nothing technically wrong with using both the constructor parameter and the object initializer, it does look a little bit clumsy. It might be easier for our clients if we allow them to use a default, parameterless constructor, and then initialize all the members using this new syntax. As we’ll see in Chapter 6, we have other ways of enforcing invariants in the object state, and dealing with incorrect usages. Object initializers are certainly a more expressive syntax, and on the basis that self-documenting and transparent is better, we’re going to change how Plane works so that we can initialize the whole object with an object initializer. As with any design consideration, there is a counter argument. Some classes may be downright difficult to put into a “default” (zero-ish) state that isn’t actively dangerous. We’re also increasing the size of the public API by the changes we’re making—we’re adding a public setter. Here, we’ve decided that the benefits outweigh the disadvantages in this par- ticular case (although it’s really a judgment call; no doubt some devel- opers would disagree). First, as Example 3-36 shows, we’ll delete the special constructor from Plane, and then make Identifier an ordinary read/write property. We can also remove the _identifier backing field we added earlier, because we’ve gone back to using an auto property. Example 3-36. Modifying Plane to work better with object initializers class Plane { // Remove the constructor that we no longer require // public Plane(string newIdentifier) // { // Identifier = newIdentifier; // } public string Identifier { get; // remove the access modifier // to make it public set; } // ... } We can now use the object initializer syntax for all the properties we want to set. As Example 3-37 shows, this makes our code look somewhat neater—we only need one style of code to initialize the object. 94 | Chapter 3: Abstracting Ideas with Classes and Structs Example 3-37. Nothing but object initializer syntax Plane someBoeing777 = new Plane { Identifier = "BA0049", Direction = DirectionOfApproach.Approaching, SpeedInMilesPerHour = 150 }; Object initializer syntax provides one big advantage over offering lots of specialized constructors: people using your class can provide any combination of properties they want. They might decide to set the Position property inline in this object initializer too, as Example 3-38 does—if we’d been relying on constructors, default or named argu- ments wouldn’t have helped if there was no constructor available that accepted a Position. We’ve not had to provide an additional constructor overload to make this possible—developers using our class have a great deal of flexibility. Of course, this approach only makes sense if our type is able to work sensibly with default values for the properties in question. If you absolutely need certain values to be provided on initialization, you’re better off with constructors. Example 3-38. Providing an extra property Plane someBoeing777 = new Plane { Identifier = "BA0049", Direction = DirectionOfApproach.Approaching, SpeedInMilesPerHour = 150, Position = new PolarPoint3D(20, 180, 14500) }; So, we’ve addressed the data part of our Plane; but the whole point of a class is that it can encapsulate both state and operations. What methods are we going to define in our class? Defining Methods When deciding what methods a class might need, we generally scan our specifications or scenarios for verbs that relate to the object of that class. If we look back at the ATC system description at the beginning of this chapter, we can see several plane-related actions, to do with granting permissions to land and permissions to take off. But do we need functions on the Plane class to deal with that? Possibly not. It might be better to deal with that in another part of the model, to do with our ground control, runways, and runway management (that, you’ll be pleased to hear, we won’t be building). But we will periodically need to update the position of all the planes. This involves changing the state of the plane—we will need to modify its Position. And it’s a change of state whose details depend on the existing state—we need to take the direction and Defining Methods | 95 speed into account. This sounds like a good candidate for a method that the Plane class should offer. Example 3-39 shows the code to add inside the class. Example 3-39. A method public void UpdatePosition(double minutesToAdvance) { double hours = minutesToAdvance / 60.0; double milesMoved = SpeedInMilesPerHour * hours; double milesToTower = Position.Distance; if (Direction == DirectionOfApproach.Approaching) { milesToTower -= milesMoved; if (milesToTower < 0) { // We've arrived! milesToTower = 0; } } else { milesToTower += milesMoved; } PolarPoint3D newPosition = new PolarPoint3D( milesToTower, Position.Angle, Position.Altitude); } This method takes a single argument, indicating how much elapsed time the calculation should take into account. It looks at the speed, the direction, and the current position, and uses this information to calculate the new position. This code illustrates that our design is some way from being finished. We never change the altitude, which suggests that our planes are going to have a hard time reaching the ground. (Although since this code makes them stop moving when they get directly above the tower, they’ll probably reach the ground soon enough...) Apparently our initial spec- ification did not fully and accurately describe the problem our software should be solving. This will not come as astonishing news to anyone who has worked in the software industry. Clearly we need to talk to the client to get clarification, but let’s implement what we can for now. Notice that our code is able to use all of the properties—SpeedInMilesPerHour, Direction, and so on—without needing to qualify them with a variable. Whereas in Example 3-35 we had to write someBoeing777.SpeedInMilesPerHour, here we just write SpeedInMilesPerHour. Methods are meant to access and modify an object’s state, and so you can refer directly to any member of the method’s containing class. There’s one snag with that. It can mean that for someone reading the code, it’s not always instantly obvious when the code uses a local variable or argument, and when it uses some member of the class. Our properties use PascalCasing, while we’re using 96 | Chapter 3: Abstracting Ideas with Classes and Structs camelCasing for arguments and variables, which helps, but what it we wanted to access a field? Those conventionally use camelCasing too. That’s why some developers put an underscore in front of their field names—it makes it more obvious when we’re doing something with the object’s state. But there’s an alternative—a more explicit style, shown in Example 3-40. Example 3-40. Explicit member access public void UpdatePosition(double minutesToAdvance) { double hours = minutesToAdvance / 60; double milesMoved = this.SpeedInMilesPerHour * hours; double milesToTower = this.Position.Distance; if (this.Direction == DirectionOfApproach.Approaching) { milesToTower -= milesMoved; if (milesToTower < 0) { // We've arrived! milesToTower = 0; } } else { milesToTower += milesMoved; } PolarPoint3D newPosition = new PolarPoint3D( milesToTower, this.Position.Angle, this.Position.Altitude); } This is almost the same as Example 3-39, except every member access goes through a variable called this. But we’ve not defined any such variable—where did that come from? The UpdatePosition method effectively has an implied extra argument called this, and it’s the object on which the method has been invoked. So, if our Main method were to call someBoeing777.UpdatePosition(10), the this variable would refer to whatever ob- ject the Main method’s someBoeing777 variable referred to. Methods get a this argument by default, but they can opt out, because sometimes it makes sense to write methods that don’t apply to any particular object. The Main method of our Program class is one example—it has no this argument, because the .NET Framework doesn’t presume to create an object; it just calls the method and lets us decide what objects, if any, to create. You can tell a method has no this argument because it will be marked with the static keyword—you may recall from Chapter 2 that this means the method can be run without needing an instance of its defining type. Aside from our Main method, why might we not want a method to be associated with a particular instance? Well, one case comes to mind for our example application. Defining Methods | 97 There’s a rather important feature of airspace management that we’re likely to need to cope with: ensuring that we don’t let two planes hit each other. So, another method likely to be useful is one that allows us to check whether one plane is too close to another one, within some margin of error (say, 5,000 feet). And this method isn’t associated with any single plane: it always involves two planes. Now we could define a method on Plane that accepted another Plane as an argument, but that’s a slightly misleading design—it has a lack of symmetry which suggests that the planes play different roles, because you’re invoking the method on one while pass- ing in the other as an argument. So it would make more sense to define a static method—one not directly associated with any single plane—and to have that take two Plane objects. Declaring Static Methods We’ll add the method shown in Example 3-41 to the Plane class. Because it is marked static, it’s not associated with a single Plane, and will have no implicit this argument. Instead, we pass in both of the Plane objects we want to look at as explicit arguments, to emphasize the fact that neither of the objects is in any way more significant than the other in this calculation. Example 3-41. Detecting when Planes are too close public static bool TooClose(Plane first, Plane second, double minimumMiles) { double x1 = first.Position.Distance * Math.Cos(first.Position.Angle); double x2 = second.Position.Distance * Math.Cos(second.Position.Angle); double y1 = first.Position.Distance * Math.Sin(first.Position.Angle); double y2 = second.Position.Distance * Math.Sin(second.Position.Angle); double z1 = first.Position.Altitude / feetPerMile; double z2 = second.Position.Altitude / feetPerMile; double dx = x1 - x2; double dy = y1 - y2; double dz = z1 - z2; double distanceSquared = dx * dx + dy * dy + dz * dz; double minimumSquared = minimumMiles * minimumMiles; return distanceSquared < minimumSquared; } private const double feetPerMile = 5280; We’ve seen plenty of function declarations like this before, but we’ll quickly recap its anatomy. This one returns a bool to indicate whether we’re safe (true) or not (false). In its parameter list, we have the references to the two Plane objects, and a double for the margin of error (in miles). 98 | Chapter 3: Abstracting Ideas with Classes and Structs Because there’s no implicit this parameter, any attempt to use nonstatic members of the class without going through an argument or variable such as first and second in Example 3-41 will cause an error. This often catches people out when learning C#. They try adding a method to the Program class of a new program, and they forget to mark it as static (or don’t realize that they need to), and then are surprised by the error they get when attempting to call it from Main. Main is a static method, and like any static method, it cannot use nonstatic members of its contain- ing type unless you provide it with an instance. Example 3-41 performs some calculations to work out how close the planes are. The details aren’t particularly important here—we’re more interested in how this uses C# methods. But just for completeness, the method converts the position into Cartesian coordinates, and then calculates the sum of the squares of the differences of the coor- dinates in all three dimensions, which will give us the square of the distance between the two planes. We could calculate the actual distance by taking the square root, but since we only want to know whether or not we’re too close, we can just compare with the minimum distance squared. (Computers are much faster at squaring than they are at calculating square roots, so given that we could do it either way, we may as well avoid the square root.) Static Fields and Properties It isn’t just functions that we can declare as static. Fields and properties can be static, too. In fact, we’ve already seen a special kind of static field—the const value we defined for the conversion between miles and kilometers. There was only one conversion factor value, however many objects we instantiated. The only difference between a const field and a static field is that we can modify the static field. (Remember: the const field was immutable.) So, a static property or field effectively lets us get or set data associated with the class, rather than the object. No matter how many objects we create, we are always getting and setting the same value. Let’s look at a trivial illustration, shown in Example 3-42, to explore how it works, before we think about why we might want to use it. Example 3-42. Static state public class MyClassWithAStaticProperty { public static bool TrueOrFalse { get; set; } public void SayWhetherTrueOrFalse() Static Fields and Properties | 99 { Console.WriteLine("Object is {0}", TrueOrFalse); } } class Program { static void Main(string[] args) { // Create two objects MyClassWithAStaticProperty object1 = new MyClassWithAStaticProperty(); MyClassWithAStaticProperty object2 = new MyClassWithAStaticProperty(); // Check how the property looks to each object, // and accessed through the class name object1.SayWhetherTrueOrFalse(); object2.SayWhetherTrueOrFalse(); Console.WriteLine("Class is {0}", MyClassWithAStaticProperty.TrueOrFalse); // Change the value MyClassWithAStaticProperty.TrueOrFalse = true; // And see that it has changed everywhere object1.SayWhetherTrueOrFalse(); object2.SayWhetherTrueOrFalse(); Console.WriteLine("Class is {0}", MyClassWithAStaticProperty.TrueOrFalse); Console.ReadKey(); } } If you compile and run this code in a console application project, you’ll see the following output: Object is False Object is False Class is False Object is True Object is True Class is True This demonstrates that there’s clearly just the one piece of information here, no matter how many different object instances we may try to look at it through. But why might we want this kind of static, class-level data storage? The principal use for class-level data is to enforce the reality that there is exactly one instance of some piece of data throughout the whole system. If you think about it, that’s exactly what our miles-to-kilometers value is all about—we only need one instance of that number for the whole system, so we declare it as const (which, as we’ve already 100 | Chapter 3: Abstracting Ideas with Classes and Structs seen, is like a special case of static). A similar pattern crops up in lots of places in the .NET Framework class library. For example, on a computer running Windows, there is a specific directory containing certain OS system files (typically C:\Windows \system32). The class library provides a class called Environment which offers, among other things, a SystemDirectory property that returns that location, and since there’s only one such directory, this is a static property. Another common use for static is when we want to cache information that is expensive to calculate, or which is frequently reused by lots of different objects of the same type. To get a benefit when lots of objects use the common data, it needs to be available to all instances. Static Constructors We can even apply the static keyword to a constructor. This lets us write a special constructor that only runs once for the whole class. We could add the constructor in Example 3-43 to our Plane class to illustrate this. Example 3-43. Static constructor static Plane() { Console.WriteLine("Plane static constructor"); } With this code in place, you would see the message printed out by that constructor just once at the beginning of the program—static constructors run exactly once. In case you’re wondering, yes, static fields can be marked as readonly. And just as a normal readonly field can only be modified in a constructor, a static readonly field can only be modified in a static constructor. But when exactly do static constructors run? We know when regular members get initialized and when normal constructors run—that happens when we new up the ob- ject. Everything gets initialized to zero, and then our constructor(s) are called to do any other initialization that we need doing. But what about static initialization? The static constructor will run no later than the first time either of the following hap- pens: you create an instance of the class; you use any static member of the class. There are no guarantees about the exact moment the code will run—it’s possible you’ll see them running earlier than you would have expected for optimization reasons. Field initializers for static fields add some slight complication. (Remember, a field ini- tializer is an expression that provides a default value for a field, and which appears in the field declaration itself, rather than the constructor. Example 3-44 shows some ex- amples.) .NET initializes the statics in the order in which they are declared. So, if you Static Fields and Properties | 101 reference one static field from the initializer for another static field in the same class, you need to be careful, or you can get errors at runtime. Example 3-44 illustrates how this can go wrong. (Also, the .NET Framework is somewhat noncommittal about ex- actly when field initializers will run—in theory it has more freedom than with a static constructor, and could run them either later or earlier than you might expect, although in practice, it’s not something you’d normally need to worry about unless you’re writing multithreaded code that depends on the order in which static initialization occurs.) Example 3-44. Unwise ordering of static field initializers class Bar { public bool myField; } // Bad - null reference exception on construction class Foo { public static bool field2 = field1.myField; public static Bar field1 = new Bar(); } // OK - initialized in the right order class Foo { public static Bar field1 = new Bar(); public static bool field2 = field1.myField; } Summary We saw how to define classes from which we can create instances called objects, and that this can be useful when attempting to model real-world entities. We can also define value types, using the struct keyword, and the main difference is that when we assign variables or pass arguments, value types always copy the whole value, whereas ordinary classes (which are reference types) only copy a reference to the underlying object. We also saw a simpler kind of type: enum. This lets us define named sets of constant values, and is useful when we need a value representing a choice from a fixed set of options. So, now we know how to abstract basic ideas of information storage (through fields and simple properties) and manipulation (through functions and calculated proper- ties), using classes and objects. In the next chapter, we’re going to look at how we can extend these ideas further using a concept called polymorphism to model a hierarchy of related classes that can extend or refine some basic contract. 102 | Chapter 3: Abstracting Ideas with Classes and Structs CHAPTER 4 Extensibility and Polymorphism In the previous chapter, we saw how to define various types of classes and specify their members—fields, properties, and functions. In this chapter, we’re going to start by looking at this again in more detail, and try to understand what underlying concepts we’re implementing when we use these different coding patterns. We’ll then introduce a couple of new concepts—inheritance and polymorphism—and the language features that help us implement them. We’ve finished our ATC application, by the way. Having gotten a reputation for build- ing robust mission-critical software on time and to spec, we’ve now been retained by the fire department to produce a training and simulation system for them. Exam- ple 4-1 shows what we have so far. Example 4-1. Classes representing firefighters and fire trucks class Firefighter { public string Name { get; set; } public void ExtinguishFire() { Console.WriteLine("{0} is putting out the fire!", Name); } public void Drive(Firetruck truckToDrive, Point coordinates) { if (truckToDrive.Driver != this) { // We can't drive the truck if we're not the driver // But just silently failing is BADBAD // What we need is some kind of structured means // of telling the client about the failure // We'll get to that in Chapter 6 return; } truckToDrive.Drive(coordinates); } 103 } class Firetruck { public Firefighter Driver { get; set; } public void Drive(Point coordinates) { if (Driver == null) { // We can't drive if there's no driver return; } Console.WriteLine("Driving to {0}", coordinates); } } We have a model of the Firetruck, which uses a Firefighter as its Driver. The truck can be instructed to drive somewhere (if it has a driver), and you can tell a Firefighter to drive the truck somewhere (if he is the designated driver). You can think of this as modeling a relationship between a Firetruck and its Driver. That driver has to be a Firefighter. In object-oriented design, we call this relationship between classes an association. Association Through Composition and Aggregation An association is a kind of flexible, “arms length” relationship between two entities in the system. There are no particular constraints about the direction of the relationship: the firefighter can be associated with the truck, or the truck with the firefighter. Or both. Any particular firefighter may have associations with other types, and we can always assign another driver to the fire truck; there’s no exclusivity. For instance, we can do something like this: Firetruck truckOne = new Firetruck(); Firefighter joe = new Firefighter { Name = "Joe" }; Firefighter frank = new Firefighter { Name = "Frank" }; truckOne.Driver = joe; // Later... truckOne.Driver = frank; But what about the 30 foot retractable ladder that we happen to have on the fire truck; what kind of relationship exists between the ladder and the fire truck? Here’s our ladder class: class Ladder { public double Length { get; set; } } 104 | Chapter 4: Extensibility and Polymorphism This particular ladder is one of those powered, extensible rotating things that are built right into the truck. So let’s add a property to represent that (see Example 4-2). Example 4-2. Fire truck with integral ladder class Firetruck { public Firefighter Driver { get; set; } readonly Ladder ladder = new Ladder { Length = 30.0 }; public Ladder Ladder { get { return ladder; } } // ... } When we construct the Truck, it creates a 30-foot ladder for itself, with a read-only property to retrieve it. We call this “made of” association between classes composition. The ladder is a built- in part of the fire truck, but the fire truck can never be a part of the ladder, and the truck itself is responsible for the life cycle of its own ladder. What if we need to manage other equipment on the truck—a detachable coil of hose, for example: class Hose { } We could add a property to the Truck to get and set that (modeling a particular coil of hose being connected to the hose system on the truck): public Hose Hose { get; set; } This is another kind of composition relationship—one component of the Truck is a hose, and the truck certainly can’t be a part of the hose; but the containing object (the truck) no longer controls the creation and lifetime of its own piece of apparatus. Instead, we say that it aggregates the hose. Of course, there are no hard-and-fast rules about these terms and the code you write; they are just concepts which we use when we are designing systems. The definitions we’ve used come from the Unified Modeling Language (UML) 2.0, and we’re just mapping them to C# language features. Association Through Composition and Aggregation | 105 Nonetheless, it is useful to have a common conceptual language for describing our systems and the common characteristics of the code we use to implement them. Equally, when you are looking at someone else’s code (remembering that “someone else” includes “past you”) it is helpful to be able to translate what was written into these standard modeling concepts. So we have a software model of the Firetruck, which has a Ladder and a Hose and uses a Firefighter as its Driver. What about the fire chief? The fire chief is just another firefighter. He can drive a truck. He can put out fires. But he can do other stuff too. For instance, he can delegate responsibility for putting out a fire to another firefighter. The question we ask ourselves is this: is the FireChief a Firefighter with extra responsibilities? If the answer is yes, we are describing an is-a association (the FireChief is a Firefighter) which we can represent by an inheritance relationship. Inheritance and Polymorphism We’ll get into the nuances of the question in the preceding paragraph in a minute, but let’s assume for the time being that our answer to the question is yes (which, on face value, seems reasonable). Example 4-3 shows how we use inheritance in C#. Example 4-3. Inheritance in C# class FireChief : Firefighter { public void TellFirefighterToExtinguishFire (Firefighter colleague) { colleague.ExtinguishFire(); } } Notice that we use the colon in the class declaration to indicate that FireChief is a Firefighter. We then say that Firefighter is a base class of FireChief. Looking at the relationship from the other direction, we can also say that FireChief is a derived class of Firefighter. We’ve added the extra function that allows the chief to tell a firefighter to extinguish a fire—which encapsulates that extra responsibility. What we haven’t had to do is to duplicate all the functionality of the firefighter; that comes along anyway. We can now use the fire chief just as we would a firefighter, as shown in Example 4-4. Example 4-4. Using base class functionality inherited by a derived class Firetruck truckOne = new Firetruck(); FireChief bigChiefHarry = new FireChief { Name = "Harry" }; truckOne.Driver = bigChiefHarry; bigChiefHarry.Drive(truckOne, new Point(100,300)); 106 | Chapter 4: Extensibility and Polymorphism Firefighter joe = new Firefighter { Name = "Joe" }; bigChiefHarry.TellFirefighterToExtinguishFire(joe); Because bigChiefHarry is an object of type FireChief, and a FireChief is a Fire fighter, we can assign him to be the driver of a truck and tell him to drive it somewhere. But because he is a FireChief, we can also ask him to tell Joe to put out the fire when he gets there. Wherever we talk about a FireChief, we can treat the object as a Firefighter. This use of one type as though it were one of its bases is an example of polymorphism. Equally, we could phrase that the other way around: we can successfully substitute an instance of a more-derived class where we expect a base class. This is known as the Liskov Substitution Principle (LSP) after computer scientist Barbara Liskov, who ar- ticulated the idea in a paper she delivered in 1987. It is quite possible to derive one class from another in a way that means we can’t treat the derived class as its base type. The derived class could change the meaning or behavior of a function with the same signature as its base, or throw errors in situations where the base promised that everything would be fine—say, the base accepted parameters in the range 1–10, where the derived class accepts parameters in the range 2–5. This violates the LSP, which is a very poor design practice, but it is very easy to slip into, especially if the classes evolve independently over time. What happens if our client doesn’t know that Harry is a fire chief, though? What if we refer to the object via a reference typed to Firefighter instead? FireChief bigChiefHarry = new FireChief { Name = "Harry" }; // Another reference to Harry, but as a firefighter Firefighter stillHarry = bigChiefHarry; Firefighter joe = new Firefighter { Name = "Joe" }; stillHarry.TellFirefighterToExtinguishFire(joe); You know that stillHarry is referencing an object that is a FireChief, with that extra method on it. But the compiler produces a long, semicomprehensible error full of useful suggestions if you try to compile and execute this code: 'Firefighter' does not contain a definition for 'TellFirefighterToExtinguishFire' and no extension method 'TellFirefighterToExtinguishFire' accepting a first argument of type 'Firefighter' could be found (are you missing a using directive or an assembly reference?) Inheritance and Polymorphism | 107 The compiler is being rather tactful. It is assuming that you must’ve forgotten to include some external reference that’s got a suitable extension method definition to fix your problem. (We’ll be looking at that technique in a later chapter, by the way.) Unfortunately, the real reason for our bug is hidden in the error’s opening salvo: we’re trying to talk to a FireChief method through a variable that is strongly typed to be a Firefighter, and you can’t call on any members of the derived class through a reference typed to a base. So, if we can’t use a derived member from a reference to a base type, is there any way we can refine these classes so that Harry never puts out a fire, but always passes re- sponsibility to his Number One when he’s asked to do so, regardless of whether we happen to know that he’s a FireChief? After all, he knows that he’s the boss! To get started, we’ll have to make a few changes to the model to accommodate this idea of the chief’s Number One. In other words, we need to create an association be- tween the FireChief and his NumberOne. Remember that we typically implement this as a read/write property, which we can add to the FireChief: public Firefighter NumberOne { get; set; } And let’s change the main function so that it does what we want (see Example 4-5). Example 4-5. Using base class methods to keep the compiler happy // A reference to Joe, Harry's number one Firefighter joe = new Firefighter { Name = "Joe" }; // Firefighter harry is really a firechief, with joe as his NumberOne Firefighter harry = new FireChief { Name = "Harry", NumberOne = joe }; // Harry is just a firefighter, so he can extinguish fires // but we want him to get joe to do the work harry.ExtinguishFire(); But if we compile that, here’s the output we get: Harry is putting out the fire! That’s not what we want at all. What we want is a different implementation for that ExtinguishFire method if we’re actually a FireChief, rather than an ordinary Firefighter. 108 | Chapter 4: Extensibility and Polymorphism Replacing Methods in Derived Classes So the implementation for the ExtinguishFire method that we want on the FireChief looks like this: public void ExtinguishFire() { // Get our number one to put out the fire instead TellFirefighterToExtinguishFire(NumberOne); } What happens if we just add that function to our FireChief and compile and run? Well, it compiles, but when we run it, it still says: Harry is putting out the fire! It seems to have completely ignored our new function! Let’s go back and have a look at that compiler output again. You’ll see that although it built and ran, there’s a warning (you may have to rebuild to get it to appear again; Choose Rebuild Solution from the Build menu): 'FireChief.ExtinguishFire()' hides inherited member 'Firefighter.ExtinguishFire()'. Use the new keyword if hiding was intended. It is a good idea to leave all your compiler warnings on and work until you are both error and warning free. That way, when something crops up unexpectedly like this, you can spot it easily, rather than burying it in a pile of stuff you’re habitually ignoring. It is telling us that, rather than replacing the implementation on the base class, our method (with matching signature) is hiding it; and that if this is what we really meant to do, we should add the keyword new to the method. Hiding Base Members with new OK, let’s do that: public new void ExtinguishFire() { // Get our number one to put out the fire instead TellFirefighterToExtinguishFire(NumberOne); } We typically add the new modifier between the accessibility modifier and the return value. Replacing Methods in Derived Classes | 109 Compile and run again. You’ll notice that we’ve gotten rid of the warning, but the output hasn’t changed: Harry is putting out the fire! What’s going on? This method-hiding approach is actually letting a single object provide different im- plementations for the ExtinguishFire method. The implementation we get is based on the type of the variable we use, rather than the type of object to which the variable refers. You can see that happening if we use the code in Example 4-6 in our client. Example 4-6. Different reference type, different method // A reference to Joe, Harry's number one Firefighter joe = new Firefighter { Name = "Joe" }; // Firefighter harry is really a firechief, with joe as his NumberOne FireChief harry = new FireChief { Name = "Harry", NumberOne = joe }; Firefighter harryAsAFirefighter = harry; // Harry is just a firefighter, so he can extinguish fires // but as a firechief he gets joe to do the work harry.ExtinguishFire(); // While as a firefighter he does it himself harryAsAFirefighter.ExtinguishFire(); The output we get now looks like this: Joe is putting out the fire! Harry is putting out the fire! When we talk to our Harry object through a FireChief reference, he gets Joe to put out the fire. If we talk to the object through a Firefighter reference, he does it himself. Same object, but two completely different implementations. Why might we want to do that? Let’s say we had multiple fire chiefs on a job, but it is our policy that a chief acting as another chief’s Number One is not allowed to delegate the job again. Our code models exactly this behavior, as shown in Example 4-7. Of course, whether that’s desirable behavior is another matter entirely— we’ve ended up with such radically different approaches to putting out a fire that it might be better to separate them back out into functions with different names. When you go through a refactoring process such as this, it is a good idea to check that you’re still happy with the semantic implications of your code. Ideally, you want to end up with a neat design, but a superficially neat design that makes no sense is not helpful. 110 | Chapter 4: Extensibility and Polymorphism Example 4-7. Making twisted use of method hiding // A reference to Joe, Harry's number one Firefighter joe = new Firefighter { Name = "Joe" }; // FireChief harry has joe as his NumberOne FireChief harry = new FireChief { Name = "Harry", NumberOne = joe }; FireChief tom = new FireChief { Name = "Tom", NumberOne = harry }; // Harry is just a firefighter, so he can extinguish fires // but as a firechief he gets joe to do the work harry.ExtinguishFire(); // But if Tom is asked to extinguish a fire, he asks Harry to do it // Our policy dictates that Harry has to do it himself, not delegate to // Joe this time. tom.ExtinguishFire(); Harry delegates to Joe when he is asked to do it himself, because we are calling through a reference to a FireChief. Tom is also a FireChief, and we are calling through a reference to him as a FireChief, so he delegates to Harry; but when Harry is asked to do it in his role as a Firefighter (remember, the NumberOne property is a reference to a Firefighter), he does it himself, because we are now calling the method through that reference typed to Firefighter. So our output looks like this: Joe is putting out the fire! Harry is putting out the fire! That’s all very well, but we don’t actually want that restriction—the fire chief should be allowed to pass the work off to his subordinate as often as he likes, regardless of who he asked to do it. There’s one big caveat regarding everything we’ve just shown about method hiding: I can’t think of the last time I used this feature in a real application, but I see the warning from time to time and it usually alerts me to a mistake in my code. We’ve wanted to illustrate how method hiding works, but we discour- age you from using it. The main reason to avoid method hiding with new is that it tends to surprise your clients, and that, as we’ve established, is not a good thing. (Would you really expect behavior to change be- cause the type of the variable, not the underlying object, changes?) While method hiding is absolutely necessary for some corner cases, we usually treat this warning as an error, and think very carefully about what we’re doing if it comes up. 9 times out of 10, we’ve got an inad- vertent clash of names. Replacing Methods in Derived Classes | 111 Replacing Methods with virtual and override What we actually want to do is to change the implementation based on the type of the object itself, not the variable we’re using to get at it. To do that we need to replace or override the default implementation in our base class with the one in our derived class. A quick glance at the C# spec shows us that there is a keyword to let us do just that: override. Let’s switch to the override modifier on the FireChief implementation of the ExtinguishFire() method: public override void ExtinguishFire() { // Get our number one to put out the fire instead TellFirefighterToExtinguishFire(NumberOne); } Notice that we removed the new modifier and replaced it with override instead. But if you compile, you’ll see that we’re not quite done (i.e., we get a compiler error): 'FireChief.ExtinguishFire()': cannot override inherited member 'Firefighter.ExtinguishFire()' because it is not marked virtual, abstract, or override We’re not allowed to override the method with our own implementation because our base class has to say we’re allowed to do so. Fortunately, we wrote the base class, so we can do that (as the compiler error suggests) by marking the method in the base with the virtual modifier: class Firefighter { public virtual void ExtinguishFire() { Console.WriteLine("{0} is putting out the fire!", Name); } // ... } Why do we have this base-classes-opt-in system? Why is everything not virtual by de- fault (like, say, Java)? Arguments continue on this very issue, but the designers of C# chose to go with the nonvirtual-by-default option. There are a couple of reasons for this: one has to do with implicit contracts, and another is related to versioning. There is also (potentially) a small performance overhead for virtual function dispatch, but this is negligible in most real-world scenarios. As always, test before optimizing for this! 112 | Chapter 4: Extensibility and Polymorphism We already saw how our public API is effectively a contract with our clients. With virtual functions, though, we are defining not only a contract for the caller, as usual, but also a contract for anyone who might choose to override that method. That requires more documentation, and a greater degree of control over how you implement the method. By declaring a method as virtual, the base class gives derived classes permission to replace whole pieces of its own innards. That’s a very powerful but very dangerous technique, rather like organ transplant surgery on an animal you’ve never seen before. Even a trained surgeon might balk at replacing the kidneys of a dromedary armed with nothing more than developer-quality documentation about the process. For example, some method in your base class calls its MethodA, then its MethodB, to do some work. You then (perhaps unknowingly) rely on that ordering when you provide overrides for MethodA and MethodB. If a future version of the base class changes that ordering, you will break. Let’s go back to our example to look at that in more detail, because it is really important. First, let’s change the implementation of Firefighter.ExtinguishFire so that it makes use of a couple of helper methods: TurnOnHose and TrainHoseOnFire (see Example 4-8). Example 4-8. Virtual methods and method ordering class Firefighter { // This calls TrainHoseOnFire and TurnOnHose as part of the // process for extinguishing the fire public virtual void ExtinguishFire() { Console.WriteLine("{0} is putting out the fire!", Name); TrainHoseOnFire(); TurnOnHose(); } private void TurnOnHose() { Console.WriteLine("The fire is going out."); } private void TrainHoseOnFire() { Console.WriteLine("Training the hose on the fire."); } // ... } Replacing Methods in Derived Classes | 113 Let’s also simplify our Main function so that we can see what is going on, as shown in Example 4-9. Example 4-9. Calling a virtual method static void Main(string[] args) { // A reference to Joe, Harry's number one Firefighter joe = new Firefighter { Name = "Joe" }; joe.ExtinguishFire(); Console.ReadKey(); } If we compile and run, we’ll see the following output: Joe is putting out the fire! Training the hose on the fire. The fire is going out. All is well so far, but what happens if we add a trainee firefighter into the mix? The trainee is extremely fastidious and follows his instructor’s guidelines to the letter. We’re going to make a class for him and override the TurnOnHose and TrainHoseOnFire methods so that the work is done in the trainee’s own particular idiom. Hang on a moment, though! Our helper methods are private members. We can’t get at them, except from other members of our Firefighter class itself. Before we can do anything, we need to make them accessible to derived classes. Inheritance and Protection In the preceding chapter, we mentioned that there were two additional accessibility modifiers that we would deal with later: protected and protected internal. Well, this is where they come into their own. They make members accessible to derived classes. If you want a member to be available either to derived classes or to other classes in your own assembly, you mark that member protected internal. It will be visible to other classes in the library, or to clients that derive classes from your base, but inaccessible to other clients who just reference your assembly. If, on the other hand, you want your class to make certain methods available only to derived classes, you just mark those methods protected. In terms of code out there in the wild, this is the most common usage, but it is not necessarily the best one! 114 | Chapter 4: Extensibility and Polymorphism Both protected internal and internal are much underused access modifiers. They are a very convenient way of hiding away library im- plementation details from your consumers, and reducing the amount of documentation and surface-area testing you need. I suspect that they are unpopular (as with most “hidden by default” or “secure by default” schemes) because they can sometimes get in your way. There are a fair number of implementation details of classes in the .NET Framework that are internal (or private) that people would very much like to access, for example. A common reason for taking something useful and applying the inter nal modifier is that it was not possible to fully document (or understand the full implications of) the “hook” this would provide into the frame- work. And rather than open up potential security or reliability problems, they are marked internal until a later date: perhaps much, much later, tending toward never. Although there is an intention to revisit these things, real-world pressures mean that they often remain unchanged. This is another example of the “lock down by default” strategy which helps improve software quality. That doesn’t make it any less irritating when you can’t get at the inner workings, though! So we’ll mark those methods in the base class virtual and protected, as shown in Example 4-10. Example 4-10. Opening methods up to derived classes protected virtual void TurnOnHose() { Console.WriteLine("The fire is going out."); } protected virtual void TrainHoseOnFire() { Console.WriteLine("Training the hose on the fire."); } We can now create our TraineeFirefighter class (see Example 4-11). Example 4-11. Overriding the newly accessible methods class TraineeFirefighter : Firefighter { private bool hoseTrainedOnFire; protected override void TurnOnHose() { if (hoseTrainedOnFire) { Console.WriteLine("The fire is going out."); } Inheritance and Protection | 115 else { Console.WriteLine("There's water going everywhere!"); } } protected override void TrainHoseOnFire() { hoseTrainedOnFire = true; Console.WriteLine("Training the hose on the fire."); } } As you can see, the trainee is derived from Firefighter. We added an extra Boolean field to keep track of whether the trainee has actually trained the hose on the fire, and then provided our own implementations of TrainHoseOnFire and TurnOnHose that make use of that extra field. This is intended to model the detailed but slightly peculiar and occasionally erratic way in which the trainee follows the instructions for these opera- tions in his copy of How to Be a Firefighter, rather than allowing common sense to prevail. We also need a quick update to our main function to use our trainee. Let’s add the following code at the end: // A reference to Bill, the trainee Firefighter bill = new TraineeFirefighter { Name = "Bill" }; bill.ExtinguishFire(); If we compile and run, we see the following output: Joe is putting out the fire! Training the hose on the fire. The fire is going out. Bill is putting out the fire! Training the hose on the fire. The fire is going out. Well done, Bill; all that training came in handy, exactly as we’d expect. Although it works, you’ll notice that we’ve duplicated some code from our base class into our derived class—the bit that actually does the work in each of those methods. It would be better if we could just call on our base class implementation to do the job for us. As you’d expect, C# has this in hand, with the base keyword. Calling Base Class Methods If we ever want to call on the implementation of a member in our base class (bypassing any of our own overrides), we can do so through the special base name: base.CallOnTheBase(); 116 | Chapter 4: Extensibility and Polymorphism Using that, we can reimplement our TraineeFirefighter and remove that duplicate code, as shown in Example 4-12. Example 4-12. Avoiding duplication by calling the base class class TraineeFirefighter : Firefighter { private bool hoseTrainedOnFire; protected override void TurnOnHose() { if (hoseTrainedOnFire) { // Call on the base implementation base.TurnOnHose(); } else { Console.WriteLine("There's water going everywhere!"); } } protected override void TrainHoseOnFire() { hoseTrainedOnFire = true; base.TrainHoseOnFire(); } } So, what happens if in a later version we change the implementation of the Extinguish Fire method on the base class? Maybe we found an optimization that means it is faster to implement it like this: public virtual void ExtinguishFire() { Console.WriteLine("{0} is putting out the fire!", Name); // We've swapped these around TurnOnHose(); TrainHoseOnFire(); } Let’s imagine that this Firefighter class is being implemented by one of our colleagues. She tested the new implementation against her Firefighter unit test suite, exactly as required, and everything passed just fine—fires were extinguished. Then she handed it over to us to use (with our new TraineeFirefighter class that we’re working on). If we compile and run, we get the following output: Joe is putting out the fire! Training the hose on the fire. The fire is going out. Bill is putting out the fire! There's water going everywhere! Training the hose on the fire. Calling Base Class Methods | 117 So the Firefighter code works fine, just as our colleague promised; but our Trainee Firefighter has made a bit of a mess. This is a shame, as he has not done anything different—we didn’t change a single line of our TraineeFirefighter code that was working just a moment earlier. The problem is that, while our documentation for ExtinguishFire told us that it would call both of those virtual methods it didn’t promise to do so in any particular order. And there was no documentation at all on our protected virtual methods to tell us how we should override them, or whether there were any particular constraints or invariants we should maintain. This is a very common combination of problems when designing an inheritance hierarchy—poor documentation on the base class, and in- sufficiently defensive implementations in a derived class. Creating a class hierarchy is not an easy thing to do. And this is when we’re only making selected methods virtual—imagine the chaos if all methods were virtual by default! In the next chapter, we’re going to look at some alternative ways to vary behavior that are more easily documented and potentially more robust than deriving from a base class. That’s not to say that you shouldn’t make use of such a powerful concept as polymorphism; it is just that you should take care when you do so. Let’s just recap the implications of all that, as it is a bit complicated. Back in Chapter 3, when we designed our first class, we talked about its public contract, and how that encapsulated the implementation details which allowed us to use it as though it was a black box. With the addition of public and protected virtual members, we’re opening that black box and creating a second contract: for people who derive their own classes, which, as we just saw, is a whole lot more complex. The designers of C# decided that should be an opt-in complexity: unless we specify that a member is virtual we don’t have to worry about it. Along the same lines, they’ve also provided a way to ensure that we don’t have to worry about anyone deriving from us at all. Thus Far and No Farther: sealed Having got through all of that, you’re probably rightly concerned that, simple though it is in theory, the practical implications of inheritance are actually rather complex and require a lot of documentation, testing, and imagining how people might use and abuse your virtual methods. And we have to do that for every class down the hierarchy. 118 | Chapter 4: Extensibility and Polymorphism When we designed our FireChief, we happily provided an override for the Extinguish Fire method, without giving a thought for the fact that someone else might override that method in his own derived class. In fact, we didn’t even consider the possibility that anyone might derive from FireChief at all. No documentation, nothing. Now there are several members on our own base class that could be overridden by a class that derives from FireChief. Does that have any implications for our own docu- mentation or testing? Can we even tell? And how could we have guessed that was going to happen when we built our FireChief class, since there was only one virtual member on the base at that time? This looks like it has the potential to become a rich future source of bugs and security holes. Fortunately, we can eliminate this problem at a stroke by saying that we didn’t design our FireChief to be derived from, and stopping anyone from doing so. We do that by marking the FireChief class sealed. Let’s see how that looks: sealed class FireChief : Firefighter { // ... } We apply the sealed modifier before the class keyword and after any accessibility modifiers if they are present. So, what happens if we try to derive a new class from FireChief now? class MasterChief : FireChief { } Compile it, and you’ll see the following error: 'MasterChief': cannot derive from sealed type 'FireChief' That’s put a stop to that. Let’s delete our MasterChief so that everything builds again. Not only can sealing classes be very useful (and defensive), but if you decide later that you want to unseal the class and allow people to derive their own types, it doesn’t (normally) break binary compatibility for the type. Sealing a previously unsealed class, however, does break compatibility. We now have three different types of firefighter. Let’s remind ourselves how they are related (see Figure 4-1). Thus Far and No Farther: sealed | 119 Figure 4-1. Three types of firefighter Nonvirtual by Default, but Not sealed? Why, you may ask, if we are nonvirtual by default aren’t we also sealed by default, with an “unseal” keyword? Notice, for instance, that we’ve been talking about classes so far—value types (struct) are sealed (with no opt-out), so you can’t derive from them. There’s no performance hit to marking a class sealed. There are potential security ad- vantages to marking a class sealed (no one can sneakily exploit polymorphism to insert code where you weren’t expecting it). So why not make them all sealed? It is certainly much less problematic to present an unsealed class than it is to present a virtual method; if there are no virtual methods, all you can do is to bolt extra bits on, which do no harm to anyone. It also conforms to the expectations of a generation of C++ and Java developers in this regard. Plenty of people argue that we should have both unsealed-by-default and virtual-by- default, and they certainly have a point, particularly with regard to convenience; but the designers of C# took a different view. No doubt, the debate will continue. Those three types of firefighter basically differ in the strategy that they use for putting out fires. There’s a base class that provides a default implementation, and a couple of classes that override the virtual methods to do things differently. Let’s say we wanted to support lots of different types of firefighter, all of whom were expected to have a different approach to fighting fire, from the trainee, to the chief, to Gulliver (who has his own idiosyncratic way of putting out a fire in Lilliput). We still want the handy Name property and the Drive method, and we still want anyone to be able to call an ExtinguishFire method. 120 | Chapter 4: Extensibility and Polymorphism Noticing that our FireChief, for example, doesn’t make use of the base implementation at all; we don’t want to provide a standard for that method. We’ll just let all imple- menters decide for themselves how it is going to work. We’re shooting for something that looks like Figure 4-2. Figure 4-2. Abstract base classes Requiring Overrides with abstract An abstract base class is intended to provide the “scaffolding” for a hierarchy of related classes, but it is not intended to be instantiated itself, because it isn’t “finished.” It requires that classes derived from it add in some missing bits. Let’s turn our current firefighter into an abstract base for the others to use, and see how that works. First, we can add the abstract modifier to the class itself, and see what happens: abstract class Firefighter { // ... } As usual, we add the modifier before the class keyword (and after any accessibility modifiers, if present). If we build, we now get a compiler error: Cannot create an instance of the abstract class or interface 'Firefighter' Requiring Overrides with abstract | 121 That’s because we’re trying to create an instance of the Firefighter in our main function: Firefighter joe = new Firefighter { Name = "Joe" }; This is no longer allowed, because the Firefighter class is now abstract. OK, we’ll comment that out temporarily while we carry on refactoring. We want it to continue to build as we go so that we can see if we’ve introduced any other errors: //Firefighter joe = new Firefighter { Name = "Joe" }; //joe.ExtinguishFire; Build and run, and we get the output we expect—Bill is still spraying the water around: Bill is putting out the fire! There's water going everywhere! Training the hose on the fire. One other thing: if we’re creating an abstract base class, we usually name it something such as FooBase to distinguish it from a regular class. This is by no means a hard-and- fast rule, but it is pretty common. So let’s rename Firefighter to FirefighterBase, and make sure we change it where it is referenced elsewhere—on the Firetruck, FireChief, and TraineeFirefighter classes. The easiest way to do that is to use the automatic rename refactoring in the IDE. Just type over the old name in the declaration, click on the Smart Tag that appears, and choose Rename Firefighter to FirefighterBase from the menu. You could do it by hand if you wanted, though. The whole purpose of this was to get rid of the default implementation we have for putting out fires, so let’s turn Firefighterbase.ExtinguishFire into an abstract method. Just like the modifier for the class, we use the abstract keyword, but this time we also remove the method body and add a semicolon at the end of the declaration: abstract class FirefighterBase { public abstract void ExtinguishFire(); } If you try building again now, you can see that we have a new compiler error: 'TraineeFirefighter' does not implement inherited abstract member 'FirefighterBase.ExtinguishFire()' Remember, we are required to override an abstract method; our class isn’t finished until we do so (unlike a virtual method, where we are invited to override it, but it will fall back on the base if we don’t). While our FireChief does override the method, our TraineeFirefighter doesn’t. So we need to add a suitable implementation: class TraineeFirefighter : FirefighterBase { // Override the abstract method public override void ExtinguishFire() 122 | Chapter 4: Extensibility and Polymorphism { // What are we going to put here? } // ... } But what are we going to put into that ExtinguishFire override? Before, we depended on our base class for the implementation, but our base is now abstract, so we don’t have one available anymore! That’s because we’ve forgotten about our regular Firefighter. Let’s add a class for him back into the hierarchy: class Firefighter : FirefighterBase { public override void ExtinguishFire() { Console.WriteLine("{0} is putting out the fire!", Name); TurnOnHose(); TrainHoseOnFire(); } } Notice we’ve given him the “standard firefighter” implementation for ExtinguishFire. If we take one more look at the base class, we can see that we still have those two virtual implementation helpers. While everything builds correctly at the moment, they don’t really belong there; they are really a part of the Firefighter implementation, so let’s move them in there. We end up with the code in Example 4-13. Example 4-13. Refactored base classes abstract class FirefighterBase { public abstract void ExtinguishFire(); public string Name { get; set; } public void Drive(Firetruck truckToDrive, Point coordinates) { if (truckToDrive.Driver != this) { // We can't drive the truck if we're not the driver return; } truckToDrive.Drive(coordinates); } } class Firefighter : FirefighterBase { public override void ExtinguishFire() { Console.WriteLine("{0} is putting out the fire!", Name); TrainHoseOnFire(); Requiring Overrides with abstract | 123 TurnOnHose(); } protected virtual void TurnOnHose() { Console.WriteLine("The fire is going out."); } protected virtual void TrainHoseOnFire() { Console.WriteLine("Training the hose on the fire."); } } But we’re still not quite done! If you build this you’ll see another compiler error: 'TraineeFirefighter.TurnOnHose()': no suitable method found to override 'TraineeFirefighter.TrainHoseOnFire()': no suitable method found to override Our trainee firefighter really is a kind of firefighter, and depends on those two virtual functions we just moved. The error message is telling us that we can’t override a method that isn’t actually present in the base. We need to change its base class from FirefighterBase to Firefighter. This has the advantage that we can also get rid of its duplicate override of the ExtingushFire method (see Example 4-14). Example 4-14. Using the newly refactored base classes class TraineeFirefighter : Firefighter { protected override void TurnOnHose() { if (hoseTrainedOnFire) { Console.WriteLine("The fire is going out."); } else { Console.WriteLine("There's water going everywhere!"); } } private bool hoseTrainedOnFire; protected override void TrainHoseOnFire() { hoseTrainedOnFire = true; Console.WriteLine("Training the hose on the fire."); } } 124 | Chapter 4: Extensibility and Polymorphism We also need to uncomment our two lines about Joe in the Main function—everything should work again: Firefighter joe = new Firefighter { Name = "Joe" }; joe.ExtinguishFire(); We can build and run to check that. We get the expected output: Joe is putting out the fire! Training the hose on the fire. The fire is going out. Bill is putting out the fire! There's water going everywhere! Training the hose on the fire. Let’s remind ourselves of the current class hierarchy (see Figure 4-2). Our FireChief is no longer an “ordinary” Firefighter, with an override for putting out fires, but he does take advantage of our common scaffolding for “firefighters in general” that we modeled as an abstract base class called FirefighterBase. Our Firefighter also takes advantage of that same scaffolding, but our TraineeFirefighter really is a Firefighter—just with its own idiosyncratic way of doing some of the internal methods that Firefighter uses to get the job done. Back to the requirements for our fire department application: let’s say we want to keep track of who is actually in the fire station at any particular time, just in case there is a fire on the premises and we can take a roll call (health and safety is very important, especially in a fire station). There are two types of folks in the fire station: the firefighters and the administrators. Example 4-15 shows our new Administrator class. Example 4-15. A class representing administrative staff class Administrator { public string Title { get; set; } public string Forename { get; set; } public string Surname { get; set; } public string Name { get { StringBuilder name = new StringBuilder(); AppendWithSpace(name, Title); AppendWithSpace(name, Forename); AppendWithSpace(name, Surname); return name.ToString(); } } void AppendWithSpace(StringBuilder builder, string stringToAppend) { // Don't do anything if the string is empty Requiring Overrides with abstract | 125 if (string.IsNullOrEmpty(stringToAppend)) { return; } // Add a space if we've got any text already if (builder.Length > 0) { builder.Append(" "); } builder.Append(stringToAppend); } } If you look at our Firefighter class, it had a single string property for a Name. With the Administrator, you can independently get and set the Title, Forename, and Surname. We then provided a special read-only property that returns a single formatted string for the whole Name. It uses a framework class called StringBuilder to assemble the name from the individual components as efficiently as possible. AppendWithSpace is a utility function that does the actual work of concatenating the substrings. It works out whether it needs to append anything at all using a static method on string that checks whether it is null or empty, called IsNullOrEmpty; finally, it adds an extra space to separate the individual words. To do the roll call we want to write some code such as that in Example 4-16. Example 4-16. Using the Administrator class static void Main(string[] args) { FireStation station = new FireStation(); // A reference to Joe, Harry's number one Firefighter joe = new Firefighter { Name = "Joe" }; // A reference to Bill, the trainee FirefighterBase bill = new TraineeFirefighter { Name = "Bill" }; // Harry is back FireChief bigChiefHarry = new FireChief { Name = "Harry"}; // And here's our administrator - Arthur Administrator arthur = new Administrator { Title = "Mr", Forename = "Arthur", Surname = "Askey" }; station.ClockIn(joe); station.ClockIn(bill); station.ClockIn(bigChiefHarry); station.ClockIn(arthur); 126 | Chapter 4: Extensibility and Polymorphism station.RollCall(); Console.ReadKey(); } When you are designing a class framework it can often be a good idea to write some example client code. You can then ensure that your design is a good abstract model while supporting clean, simple code at point- of-use. Clearly, we’re going to need a FireStation class that is going to let our administrators and firefighters ClockIn (registering their presence in the station), and where we can do a RollCall (displaying their names). But what type is that ClockIn function going to take, given that we haven’t specified any common base class that they share? All Types Are Derived from Object .NET comes to our rescue again. It turns out that every type in the system is derived from Object. Every one—value types (struct) and reference types (class) alike, even the built-in types such as Int32. It is easy to see how that would work for a class declaration in C#. If you don’t specify a particular base class, you get Object by default. But what about a struct, or enum, or the built-in types; what happens if we try to talk to them through their Object “base class”? Boxing and Unboxing Value Types Let’s give it a try. This code snippet will compile and work quite happily: // Int variable int myIntVariable = 1; object myObject = myIntVariable; What happens under the covers is that the runtime allocates a new object and puts a copy of the value inside it. This is called boxing, and, as you might expect given that it involves allocating objects and copying values, it is relatively expensive when compared to a straightforward assignment. You can also convert back the other way: // Int variable int myIntVariable = 1; object myObject = myIntVariable; int anotherIntVariable = (int)myObject; All Types Are Derived from Object | 127 Notice how we use the type name in parentheses to perform the conversion back to an int. In general, this sort of conversion from one type to another is known as a “cast,” and will work for classes too (although we’ll see a more explicit way of doing that later in this chapter). The runtime looks at that box object for us and checks that it contains a value of the correct type. If so, it will copy the value back out of the box and into the new variable. What if it isn’t of the correct type? The runtime will throw an Invalid CastException. You can find out more about exceptions in Chapter 6. That process is known as unboxing, and is also quite expensive (although not as ex- pensive as boxing, as it doesn’t need to allocate the object). Although these performance costs are individually fairly small, if you are processing large numbers of value types in a way that requires them to be repeatedly boxed and unboxed the costs can add up quite rapidly; so you should be aware of boxing and unboxing when you are profiling your application. So the only common base of both Firefighter and Administrator is Object at the mo- ment (remember, everything is ultimately derived from Object). That seems a bit low level, but it is all we have to go on for now, so we’ll make do. Example 4-17 shows our first pass at a FireStation. Example 4-17. FireStation class class FireStation { List clockedInStaff = new List(); public void ClockIn(object staffMember) { if (!clockedInStaff.Contains(staffMember)) { clockedInStaff.Add(staffMember); } } public void RollCall() { foreach(object staffMember in clockedInStaff) { // Hmmm... What to do? } } } 128 | Chapter 4: Extensibility and Polymorphism Our ClockIn method is making use of a list of objects to keep track of who is in the station. To do that it is using the generic collection class List we first saw in Chap- ter 2. Using the List.Contains method, the implementation checks that they weren’t already in the station, and adds them if necessary. Everything is fine so far. Then we reach the RollCall method. We’re using foreach to iterate over the clocked-in staff, but we don’t actually have a method to call to get their names! We want a way of indicating that these disparate object types (firefighters, fire chiefs, and administrators) all support giving out their name. We saw one way of doing that already: we could create a common base class, and move the Name functionality in there. Let’s see what happens if we try to do that. Practically speaking, we have two completely different implementations of the Name property. We saw that we can model that situation with an abstract base class from which Firefighter and Administrator both derive, both implementing the method in their own way. Here’s our NamedPerson base with an abstract property for the Name: abstract class NamedPerson { public abstract string Name { get; } } There’s no problem when we implement this on our Administrator: class Administrator : NamedPerson { public override string Name { get { StringBuilder name = new StringBuilder(); AppendWithSpace(name, Title); AppendWithSpace(name, Forename); AppendWithSpace(name, Surname); return name.ToString(); } } // ... } Notice how we derived from NamedPerson and added the override modifier to our Name property so that it overrides the abstract method in our base. All Types Are Derived from Object | 129 That’s fine so far. What about our FirefighterBase? Let’s try doing exactly the same thing: abstract class FirefighterBase : NamedPerson { public abstract void ExtinguishFire(); public override string Name { get; set; } // ... } If we compile that, we get an error: 'FirefighterBase.Name.set': cannot override because 'NamedPerson.Name' does not have an overridable set accessor We run into difficulty because FirefighterBase has both a getter and a setter for the Name property, but our base allows only a getter. Properties Under the Hood We’re going to dive into the guts of the thing here, so feel free to skip this if you’re not deeply interested in how this works under the covers. If you do look under the hood at the IL code emitted by the compiler, using a tool such as ildasm, you can see that properties consist of two pieces: the property metadata, and (either or both of) two functions called get_PropName and set_PropName, which actually implement the getter/setter. If you’ve chosen to use the simple property syntax, there’s also a field created with a name that is something like k__BackingField. At the IL level, there’s no difference between a property override and a new property declaration. It is the metadata on the getter and setter functions that determines whether they are virtual (and indeed what their accessibility might be). So the fact that we can’t override a property that has a getter in the base class with one that has a getter and a setter in the derived class is a feature of the C# language, not the underlying platform. Well, we could work around that with another member to set the name; but as you can see in Example 4-18, it is all getting a bit ugly. Example 4-18. Mounting evidence that all is not well in our class hierarchy abstract class FirefighterBase : NamedPerson { public abstract void ExtinguishFire(); public override string Name { get { return RealName; 130 | Chapter 4: Extensibility and Polymorphism } } public string RealName { get; set; } // ... } Not only is it ugly, but we have to replace all our object initializers to refer to our new RealName property, so it is making us do unnecessary work, which is never good: Firefighter joe = new Firefighter { RealName = "Joe" }; Are you feeling uncomfortable with this approach yet? Let’s push on with it just a little bit further, and see what happens if we want to support a second behavior. Say we had a SalariedPerson abstract base that provides us with the contract for getting/setting a person’s salary. We’re going to need to apply that to both the FirefighterBase and the Administrator, to tie in with the billing system: abstract class SalariedPerson { public abstract decimal Salary { get; set; } } We’re providing a decimal property for the Salary that must be implemented by any SalariedPerson. So, what happens if we now try to derive from this class for our Administrator, as shown in Example 4-19? Example 4-19. The final straw: Our class hierarchy needs a rethink class Administrator : NamedPerson, SalariedPerson { private decimal salary; public override decimal Salary { get { return salary; } set { salary = value; } } // ... } All Types Are Derived from Object | 131 C++ developers will be familiar with this syntax for specifying multiple base classes. Another compiler error: Class 'Administrator' cannot have multiple base classes: 'NamedPerson' and 'SalariedPerson' C# Does Not Support Multiple Inheritance of Implementation This is a pretty fundamental roadblock! You cannot derive your class from more than one base class. When the designers of .NET were thinking about the platform fundamentals, they looked at this issue of multiple inheritance and how they’d support it across multiple languages, including C#, VB, and C++. They decided that the C++ approach was too messy and prone to error (particularly when you think about how to resolve members that appear in both base classes with the same signature). The implications of multiple inheritance were probably just too difficult to come to grips with, and therefore were unlikely to bring net productivity gains. With that view prevailing, single inheritance of implementation is baked into the platform. In more recent interviews, the .NET team has reflected that perhaps there might have been a way of allowing multiple inheritance of imple- mentation, without introducing all the complexity of C++ multiple in- heritance. That’s the benefit of 20/20 hindsight; we (or our children) will just have to wait until the next platform generation and see how the argument goes then. So are we really stymied? No! While we can’t support multiple inheritance of imple- mentation, we can support multiple inheritance of interface. C# Supports Multiple Inheritance of Interface With our abstract base classes in this example, we’re not really trying to provide a base implementation for our objects at all. We’re trying to mark them as supporting a par- ticular contract that we can then rely on in our client code. C# provides us with a mechanism for doing exactly that: interface. Let’s rewrite our NamedPerson and SalariedPerson classes as interfaces instead, as shown in Exam- ple 4-20. 132 | Chapter 4: Extensibility and Polymorphism Example 4-20. Defining interfaces interface INamedPerson { string Name { get; } } interface ISalariedPerson { decimal Salary { get; set; } } We use much the same syntax as we do for a class definition, but using the keyword interface instead. Notice also that we dropped the abstract modifier on the members; an interface is implicitly without implementation. There are no accessibility modifiers on the mem- bers either; an interface member is only ever allowed to be public. The only other change we’ve made is to prefix our interface name with an I. This is not a rule, but another one of those naming conventions to which most people conform. We can now implement those interfaces on our Administrator, as shown in Exam- ple 4-21. Example 4-21. Implementing interfaces class Administrator : INamedPerson, ISalariedPerson { public decimal Salary { get; set; } public string Name { get { StringBuilder name = new StringBuilder(); AppendWithSpace(name, Title); AppendWithSpace(name, Forename); AppendWithSpace(name, Surname); return name.ToString(); } } // ... } C# Supports Multiple Inheritance of Interface | 133 And we can implement them on our FirefighterBase, as shown in Example 4-22. Example 4-22. The same interfaces in a different part of the class hierarchy abstract class FirefighterBase : INamedPerson, ISalariedPerson { public string Name { get; set; } public decimal Salary { get; set; } // ... } Notice that we can happily implement the setter on our FirefighterBase, even though the interface only requires a getter. The restrictions on how you implement the interface—as long as you conform to the contract it specifies—are much looser than those on overrides of a base class. Also, C# doesn’t allow you to use the simple property syntax to define virtual properties or their overrides, but there is no such restriction when you’re implementing an interface. So we’ve been able to use simple property syntax here rather than having to implement using full-blown properties. We can now make use of this interface in our FireStation class. Instead of a list of objects, we can use a list of INamedPerson, and call on the Name property in our RollCall method, as shown in Example 4-23. Example 4-23. Modifying the FireStation class to use an interface class FireStation { List clockedInStaff = new List(); public void ClockIn(INamedPerson staffMember) { if (!clockedInStaff.Contains(staffMember)) { clockedInStaff.Add(staffMember); Console.WriteLine("Clocked in {0}", staffMember.Name); } } public void RollCall() { foreach (INamedPerson staffMember in clockedInStaff) { Console.WriteLine(staffMember.Name); 134 | Chapter 4: Extensibility and Polymorphism } } } If you’ve been following through the code in Visual Studio (which I thoroughly recommend), you’ll also need to change your object initial- izers back to this form: Firefighter joe = new Firefighter { Name = "Joe" }; If we compile and run, we get the output we hoped for—a roll call of everyone in the station: Clocked in Joe Clocked in Bill Clocked in Harry Clocked in Mr Arthur Askey Joe Bill Harry Mr Arthur Askey Deriving Interfaces from Other Interfaces Interfaces support inheritance too, just like classes. If you want, you could create a named, salaried person interface like this: interface INamedSalariedPerson : INamedPerson, ISalariedPerson { } What happens if you have conflicting names? Imagine the interface ISettable NamedPerson: interface ISettableNamedPerson { string Name { get; set; } } What happens if we implement both INamedPerson and ISettableNamedPerson on our FirefighterBase? abstract class FirefighterBase : INamedPerson, ISettableNamedPerson, ISalariedPerson { // ... } The answer is that everything is just fine! Each interface requires that we implement a string property called Name; one requires at least a getter, the other a getter and a setter. Deriving Interfaces from Other Interfaces | 135 When we access the property through the relevant interface, it can resolve correctly which member we meant; there’s no requirement for a separate implementation for each interface. But what if that was actually wrong? What if our Name property on INamedPerson had entirely different semantics from the one on ISettableNamedPerson? Let’s suppose that one is intended to allow only letters and numbers with no spaces and the other is just our freeform “any old text” implementation with which we are familiar. Whenever our client expects an INamedPerson we need to provide the second imple- mentation, and whenever the client expects an ISettableNamedPerson, the first. We can do that by explicitly implementing the interfaces. Explicit Interface Implementation To explicitly implement a particular member of an interface, you drop the accessibility modifier and add the interface name as a prefix, as shown in Example 4-24. Example 4-24. Explicit interface implementation class AFootInBothCamps : INamedPerson, ISettableNamedPerson { private string settableName; string INamedPerson.Name { get { Console.WriteLine("Accessed through the INamedPerson interface"); return settableName; } } string ISettableNamedPerson.Name { get { return settableName; } set { Console.WriteLine( "Accessed through the " + "ISettableNamedPerson interface"); if( settableName != null && settableName.Contains(" ") ) { // You can't set it if it contains the space // character return; } settableName = value; 136 | Chapter 4: Extensibility and Polymorphism } } } Example 4-25 shows how we’re going to access them from our main function. Example 4-25. Calling different interface implementations of the same member name on the same object class Program { static void Main(string[] args) { AFootInBothCamps both = new AFootInBothCamps(); ISettableNamedPerson settablePerson = both; INamedPerson namedPerson = both; settablePerson.Name = "hello"; Console.WriteLine(settablePerson.Name); Console.WriteLine(namedPerson.Name); Console.ReadKey(); } } Notice how we’re creating our object, and then providing two additional references to it: one through a variable of type ISettableNamedPerson and one through INamedPerson. We then call on the Name property through each of those interfaces, and get the following output: Accessed through the ISettableNamedPerson interface hello Accessed through the INamedPerson interface hello But what if we try to access it through a reference typed to the class itself? Console.WriteLine(both.Name); Add the following line to the main function and compile, and we get a compiler error! 'AFootInBothCamps' does not contain a definition for 'Name' and no extension method 'Name' accepting a first argument of type 'AFootInBothCamps' could be found (are you missing a using directive or an assembly reference?) We’ve seen that error before; it means we’re trying to talk to a member that doesn’t exist. What’s happened is that the members that are explicitly implemented exist only if we are accessing them through the relevant interfaces. However, as long as we explicitly implement one of the two (or two of the three, or however many we’re stuck with), we can choose one interface as our “default” and implement it using the regular syntax, as shown in Example 4-26. Deriving Interfaces from Other Interfaces | 137 Example 4-26. Implementing one of the interfaces implicitly class AFootInBothCamps : INamedPerson, ISettableNamedPerson { private string settableName; // Regular implementation syntax public string Name { get { Console.WriteLine("Accessed through the INamedPerson interface"); return settableName; } } string ISettableNamedPerson.Name { get { return settableName; } set { Console.WriteLine("Accessed through the ISettableNamedPerson " + "interface"); if( settableName != null && settableName.Contains(" ") ) { // You can't set it if it contains the space // character return; } settableName = value; } } } Now we can compile and run, and the default implementation for our class is the one for the INamedPerson interface: Accessed through the ISettableNamedPerson interface hello Accessed through the INamedPerson interface hello Accessed through the INamedPerson interface hello 138 | Chapter 4: Extensibility and Polymorphism In real life, you don’t often come across this need for explicit interface implementation. If you have control over all the code in the application, you should avoid designing in a clash where the names are the same but the semantics are different. Like overloads or overrides with different meanings, it surprises other developers. The .NET Framework contains a few examples where it uses explicit interface implementation to hide the interface members from the public API of a class, even though there is no clash. The authors are uncon- vinced that this improves matters. More often, you will come across this usage where you don’t have con- trol of the code—with two third-party libraries, for instance, both of which declare interfaces with different semantics but a clash of names. Even then, this is not a problem unless you happen to need to implement both interfaces on one class. Even rarer! (Abstract) Base Classes Versus Interfaces We’ve clearly simplified our code by introducing interfaces into our model. Would we ever want to use abstract base classes rather than an interface? Well, we’ve already seen an example where an abstract base class is a good choice—if there’s additional implementation scaffolding that we wish to bring along with us, and the abstract members are plumbed into that structure. It would be unnecessary to in- troduce an interface just for the abstract member. In general, an interface is a good way of defining just the contract, without providing any implementation at all, especially if that contract is shared between different parts of a system. It also has strict versioning rules: if you add or remove members from an interface and ship your assembly, it is a different interface and will no longer be binary compatible with code that implements the previous version of the interface (although clients that just call through the interface will still work fine if you haven’t removed or changed something on which they depended). With an abstract base class, you can generally add members and it remains binary compatible with older code (although a new ab- stract member will also break anyone who inherits from it, of course). Right, let’s go back to our FireStation class for a minute, and imagine an interface we could create to formalize the contract for clocking in: our billing system might define this contract for us so that we can plug into it. Deriving Interfaces from Other Interfaces | 139 As it happens, our FireStation provides an implementation which can ClockIn named people, but our billing system’s IClockIn contract is much more generic—it can clock in anything of type Object, as we had in our original implementation: interface IClockIn { void ClockIn(object item); } We can now implement IClockIn on our FireStation, as shown in Example 4-27. Example 4-27. Implementing the IClockIn interface class FireStation : IClockIn { List clockedInStaff = new List(); public void ClockIn(INamedPerson staffMember) { if (!clockedInStaff.Contains(staffMember)) { clockedInStaff.Add(staffMember); Console.WriteLine("Clocked in {0}", staffMember.Name); } } public void RollCall() { foreach (INamedPerson staffMember in clockedInStaff) { Console.WriteLine(staffMember.Name); } } public void ClockIn(object item) { // What to do here } } Our original ClockIn method is unchanged, and we’ve added a new overload that takes an object, and therefore matches the requirement in our interface. But how do we implement that new method? We want to check that the person being clocked in is an INamedPerson, and if it is, perform our usual operation. Otherwise, we want to tell the user that we can’t clock him in. In other words, we need a manual check for the type of the object. 140 | Chapter 4: Extensibility and Polymorphism The Last Resort: Checking Types at Runtime C# provides us with a couple of keywords for checking the type of an object: as and is. Here’s how we can use them in our ClockIn implementation: public void ClockIn(object item) { if (item is INamedPerson) { ClockIn(item as INamedPerson); } else { Console.WriteLine("We can't check in a '{0}'", item.GetType()); } } Notice how we are using the type name to check if the item is of that type. And then we call our other overload of ClockIn by explicitly converting to a reference of our INamedPerson type, using as. It checks to see if our object would be accessible through a reference of the specified type. It looks at the whole inheritance hierarchy for the object (up and down) to see if it matches, and if it does, it provides us a reference of the relevant type. What if you don’t bother with the is check and just use as? Conveniently, the as op- eration just converts to a null reference if it can’t find a suitable type match: public void ClockIn(object item) { INamedPerson namedPerson = item as INamedPerson; if(namedPerson != null) { ClockIn(namedPerson); } else { Console.WriteLine("We can't check in a '{0}'", item.GetType()); } } This is the form in which you most often see a test like this, because it is marginally more efficient than the previous example. In the first version, the runtime has to perform the expensive runtime type checking twice: once for the if() statement and once to see whether we can actually perform the conversion, or whether null is required. In the second case, we do the expensive check only once, and then do a simple test for null. The Last Resort: Checking Types at Runtime | 141 Summary So far, we’ve seen how to create classes; to model relationships between instances of those classes through association, composition, and aggregation; and to create rela- tionships between classes by derivation. We also saw how virtual functions enable derived classes to replace selected aspects of a base class. We saw how to use protected and protected internal to control the visibility of mem- bers to derived classes. Then, we saw how we can use either abstract classes and methods or interfaces to define public contracts for a class. Finally, we looked at a means of examining the inheritance hierarchy by hand, and verifying whether an object we are referencing through a base class is, in fact, an instance of a more derived class. In the next chapter, we are going to look at some other techniques for code reuse and extensibility that don’t rely on inheritance. 142 | Chapter 4: Extensibility and Polymorphism CHAPTER 5 Composability and Extensibility with Delegates In the preceding two chapters, we saw how to encapsulate behavior and information with classes. Using the concepts of association, composition, aggregation, and deriva- tion, we modeled relationships between those classes and looked at some of the benefits of polymorphism along with the use and abuse of virtual functions and their implied contracts with derived classes. In this chapter, we’ll look at a functional (rather than class-based) approach to com- position and extensibility, and see how we can use this to implement some of the pat- terns that have previously required us to burn our one and only base class and override virtual functions; and all with the added benefit of a looser coupling between classes. Let’s start with another example. This time, we want to build a system that processes incoming (electronic) documents prior to publication. We might want to do an auto- mated spellcheck, repaginate, perform a machine translation for a foreign-language website, or perform one of any other number of operations that our editors will devise during the development process and beyond. After some business analysis, our platform team has given us a class called Document, which is shown in Example 5-1. This is their baby, and we’re not allowed to mess with it. Example 5-1. The Document class public sealed class Document { // Get/set document text public string Text { get; set; } // Date of the document 143 public DateTime DocumentDate { get; set; } public string Author { get; set; } } It has simple properties for its Text, the DocumentDate, and the Author, and no other methods. What Is Coupling? Two classes are said to be coupled if a change to one requires a change to another. We saw examples of that in the previous chapter. When we created our NamedPerson class, it required changes to the FirefighterBase and the Administrator classes. We therefore say that FirefighterBase and Administrator are coupled to NamedPerson. Of course, any class or function that refers to another class or function is coupled to that class—that’s unavoidable (indeed, desirable). But to make testing simpler and systems more reliable, we try to ensure that we minimize the number of other types to which any class or function is coupled, and that we minimize the number of couplings between any two types. That way, any given change to a class will have a minimal number of knock-on effects elsewhere in the system. We also try to ensure that we organize classes into conceptual groupings called layers so that more tightly coupled classes live together in one layer, and that there are a minimal number of well-controlled couplings between layers. As part of that layered approach, it is usual to try to ensure that most couplings go one-way; classes of a “lower” layer should not depend on classes in a layer above. That way, we can further limit (and understand) the way changes propagate through the system. The layers act like firewalls, blocking the further impact of a change. As usual with software design, these disciplines are not hard-and-fast rules, and they are not imposed by the platform or language; but they are common practices that the platform and language are designed to support. Now we want to be able to process the document. At the very least, we want to be able to Spellcheck, Repaginate, or Translate it (into French, say). Because we can’t change the Document class, we’ll implement these methods in a static utility class of common processes, as we learned in Chapter 3. Example 5-2 shows this class, although the implementations are obviously just placeholders—we’re illustrating how to structure the code here, and trying to write a real spellchecker would be a rather large distraction. 144 | Chapter 5: Composability and Extensibility with Delegates Example 5-2. Some document processing methods static class DocumentProcesses { public static void Spellcheck( Document doc ) { Console.WriteLine("Spellchecked document."); } public static void Repaginate( Document doc) { Console.WriteLine("Repaginated document."); } public static void TranslateIntoFrench( Document doc ) { Console.WriteLine("Document traduit."); } // ... } Now we can build a simple example of a document processor that translates, spell- checks, and then repaginates the document (see Example 5-3). Example 5-3. Processing a document static class DocumentProcessor { public static void Process(Document doc) { DocumentProcesses.TranslateIntoFrench(doc); DocumentProcesses.Spellcheck(doc); DocumentProcesses.Repaginate(doc); } } And we can call on it from our main function, to process a couple of documents, as shown in Example 5-4. Example 5-4. A program to test the document processing classes class Program { static void Main(string[] args) { Document doc1 = new Document { Author = "Matthew Adams", DocumentDate = new DateTime(2000, 01, 01), Text = "Am I a year early?" }; Document doc2 = new Document { Author = "Ian Griffiths", Composability and Extensibility with Delegates | 145 DocumentDate = new DateTime(2001, 01, 01), Text = "This is the new millennium, I promise you." }; Console.WriteLine("Processing document 1"); DocumentProcessor.Process(doc1); Console.WriteLine(); Console.WriteLine("Processing document 2"); DocumentProcessor.Process(doc2); Console.ReadKey(); } } Compile and run that, and you’ll see the following output: Processing document 1 Document traduit. Spellchecked document. Repaginated document. Processing document 2 Document traduit. Spellchecked document. Repaginated document. We encapsulated a particular set of processing instructions, executed in a particular order, in this (static) DocumentProcessor class so that we can easily reuse it with dif- ferent client applications that want a standard, reliable means of performing our “trans- late into French” process. So far, this should all be pretty familiar. But what about a different set of processing operations, one that leaves the document in its native language and just spellchecks and repaginates? We could just create a second DocumentProcessor-like class, and encapsulate the rele- vant method calls in a process function: static class DocumentProcessorStandard { public static void Process(Document doc) { DocumentProcesses.Spellcheck(doc); DocumentProcesses.Repaginate(doc); } } And then we could add some calls to that processor in our Main method: Console.WriteLine(); Console.WriteLine("Processing document 1 (standard)"); DocumentProcessorStandard.Process(doc1); Console.WriteLine(); Console.WriteLine("Processing document 2 (standard)"); DocumentProcessorStandard.Process(doc2); 146 | Chapter 5: Composability and Extensibility with Delegates Nothing is intrinsically wrong with any of this; it clearly works, and we have a nice enough design that neatly encapsulates our processing. We note that each DocumentProcessor is coupled to the Document class, and also to each method that it calls on the DocumentProcesses class. Our client is coupled to the Document and each DocumentProcessor class that it uses. If we go back to the specification we showed earlier, we see that we are likely to be creating a lot of different functions to modify the document as part of the production process; they’ll slip in and out of use depending on the type of document, other systems we might have to work with, and the business process of the day. Rather than hardcoding this process in an ever-increasing number of processor classes (and coupling those to an ever-increasing number of DocumentProcesses), it would ob- viously be better if we could devolve this to the developers on our production team. They could provide an ordered set of processes (of some kind) to the one and only DocumentProcessor class that actually runs those processes. We can then focus on making the process-execution engine as efficient and reliable as possible, and the production team will be able to create sequences of processes (built by either us, them, contractors, or whoever), without having to come back to us for updates all the time. Figure 5-1 represents that requirement as a diagram. Figure 5-1. Document processor architecture The document is submitted to the document processor, which runs it through an or- dered sequence of processes. The same document comes out at the other end. Composability and Extensibility with Delegates | 147 OK, let’s build a DocumentProcessor class that implements that (see Example 5-5). Example 5-5. An adaptable document processor class DocumentProcessor { private readonly List processes = new List(); public List Processes { get { return processes; } } public void Process(Document doc) { foreach(DocumentProcess process in Processes) { process.Process(doc); } } } Our document processor has a List of DocumentProcess objects (a hypothetical type we’ve not written yet). A List is an ordered collection—that is to say that the item you Add at index 0 stays at index 0, and is first out of the block when you iterate the list, and so on. That means our Process method can just iterate over the collection of DocumentProcess objects, and call some equally hypothetical Process method on each to do the processing. But what type of thing is a DocumentProcess? Well, we already saw a solution we can use—we could create a DocumentProcess abstract base, with a Process abstract method: abstract class DocumentProcess { public abstract void Process(Document doc); } We then need to create a derived class for every processing operation, as shown in Example 5-6. Example 5-6. Implementations of the abstract DocumentProcess class SpellcheckProcess : DocumentProcess { public override void Process(Document doc) { DocumentProcesses.Spellcheck(doc); } } 148 | Chapter 5: Composability and Extensibility with Delegates class RepaginateProcess : DocumentProcess { public override void Process(Document doc) { DocumentProcesses.Repaginate(doc); } } class TranslateIntoFrenchProcess : DocumentProcess { public override void Process(Document doc) { DocumentProcesses.TranslateIntoFrench(doc); } } Now we can configure a processor in our client by adding some process objects to the list (see Example 5-7). Example 5-7. Configuring a document processor with processes static DocumentProcessor Configure() { DocumentProcessor rc = new DocumentProcessor(); rc.Processes.Add(new TranslateIntoFrenchProcess()); rc.Processes.Add(new SpellcheckProcess()); rc.Processes.Add(new RepaginateProcess()); return rc; } See how we are adding the processes to the processor in the same order we had in our function calls previously? Our process objects are logically similar to function calls, and the order in which they appear is logically similar to a program, except that they are composed at runtime rather than compile time. We can then use this configuration method in our client, and call on the processor to process our documents, as shown in Example 5-8. Example 5-8. Using the dynamically configured processor static void Main(string[] args) { Document doc1 = new Document { Author = "Matthew Adams", DocumentDate = new DateTime(2000, 01, 01), Text = "Am I a year early?" }; Document doc2 = new Document { Author = "Ian Griffiths", DocumentDate = new DateTime(2001, 01, 01), Text = "This is the new millennium, I promise you." }; Composability and Extensibility with Delegates | 149 DocumentProcessor processor = Configure(); Console.WriteLine("Processing document 1"); processor.Process(doc1); Console.WriteLine(); Console.WriteLine("Processing document 2"); processor.Process(doc2); Console.ReadKey(); } If you compile and run, you’ll see the same output as before: Processing document 1 Document traduit. Spellchecked document. Repaginated document. Processing document 2 Document traduit. Spellchecked document. Repaginated document. This is a very common pattern in object-oriented design—encapsulating a method in an object and/or a process in a sequence of objects. What’s nice about it is that our DocumentProcessor is now coupled only to the Document class, plus the abstract base it uses as a contract for the individual processes. It is no longer coupled to each and every one of those processes; they can vary without requiring any changes to the processor itself, because they implement the contract demanded by the abstract base class. Finally, the processing sequence (the “program” for the DocumentProcessor) is now the responsibility of the client app, not the processor library; so our different production teams can develop their own particular sequences (and, indeed, new processes) without having to refer back to the core team and change the document processor in any way. In fact, the only thing that is a bit of a pain about this whole approach is that we have to declare a new class every time we want to wrap up a simple method call. Wouldn’t it be easier just to be able to refer to the method call directly? C# provides us with a tool to do just that: the delegate. Functional Composition with delegate We just wrote some code that wraps up a method call inside an object. The call itself is wrapped up in another method with a well-known signature. You can think of a delegate as solving that same sort of problem: it is an object that lets us wrap up a method call on another object (or class). 150 | Chapter 5: Composability and Extensibility with Delegates But while our DocumentProcess classes have their methods hardcoded into virtual func- tion overrides, a delegate allows us to reference a specific function (from a given class or object instance) at runtime, then use the delegate to execute that function. So, in the same way that a variable can be considered to contain a reference to an object, a delegate can be thought to contain a reference to a function (see Figure 5-2). Figure 5-2. Delegates and variables Before we get into the specific C# syntax, I just want to show you that there isn’t anything mystical about a delegate; in fact, there is a class in the .NET Framework called Delegate which encapsulates the behavior for us. As you might expect, it uses properties to store the reference to the function. There are two, in fact: Method (which indicates which member function to use) and Target (which tells us the object on which the method should be executed, if any). As you can see, the whole thing is not totally dissimilar in concept from our previous DocumentProcess base class, but we don’t need to derive from Delegate to supply the function to call. That ability has moved into a property instead. That’s all there is to a delegate, really. Functional Composition with delegate | 151 However, it is such a powerful and useful tool that the C# language designers have provided us with special language syntax to declare new Delegate types, assign the appropriate function, and then call it in a much more compact and expressive fashion. It also allows the compiler to check that all the parameter and return types match up along the way, rather than producing errors at runtime if you get it wrong. It is so compact, expressive, and powerful that you can probably get through your entire C# programming career without ever worrying about the classes the C# compiler emits which derive from that Delegate class and implement it all. So, why have we just spent a page or so discussing these implementation details, if we’re never going to see them again? While you don’t usually need to use the Delegate class directly, it is easy to get confused by language-specific voodoo and lose track of what a delegate really is: it is just an object, which in turn calls whichever func- tion we like, all specified through a couple of properties. Let’s start by defining a new delegate type to reference our document processing functions. As I mentioned earlier, rather than using that Delegate class, C# lets us define a delegate type using syntax which looks pretty much like a function declaration, prefixed with the keyword delegate: delegate void DocumentProcess(Document doc); That defines a delegate type for a method which returns void, and takes a single Document parameter. The delegate’s type name is DocumentProcess. Delegates Under the Hood Anyone who has sensibly decided not to go any further into the implementation details can skip this sidebar. For those still reading... When you declare a delegate like this, under the covers C# emits a class called DocumentProcess, derived from MulticastDelegate (which is a subclass of Delegate). Among other things, that emitted class has a function called Invoke(int param) which matches the signature we declared on the delegate. So how is Invoke implemented? Surprisingly, it doesn’t have any method body at all! Instead, all of the members of the emitted class are marked as special by the compiler, and the runtime actually provides the implementations so that it can (more or less) optimally dispatch the delegated function. Having added the delegate, we have two types called DocumentProcess, which is not going to work. Let’s get rid of our old DocumentProcess abstract base class, and the three 152 | Chapter 5: Composability and Extensibility with Delegates classes we derived from it. Isn’t it satisfying, getting rid of code? There is less to test and you are statistically likely to have fewer bugs. So how are we going to adapt our DocumentProcessor to use our new definition for the DocumentProcess type? Take a look at Example 5-9. Example 5-9. Modifying DocumentProcess to use delegates class DocumentProcessor { private readonly List processes = new List(); public List Processes { get { return processes; } } public void Process(Document doc) { foreach(DocumentProcess process in Processes) { // Hmmm... this doesn't work anymore process.Process(doc); } } } We’re still storing a set of DocumentProcess objects, but those objects are now delegates to member functions that conform to the signature specified by the DocumentProcess delegate. We can still iterate over the process collection, but we no longer have a Process method on the object. The equivalent function on the delegate type is a method called Invoke which matches the signature of our delegated function: process.Invoke(doc); While this works just fine, it is such a common thing to need to do with a delegate that C# lets us dispense with .Invoke entirely and treat the delegate as though it really was the function to which it delegates: process(doc); Here’s the final version of our Process method: public void Process(Document doc) { foreach(DocumentProcess process in Processes) { process(doc); } } Functional Composition with delegate | 153 This can take a bit of getting used to, because our variable names are usually camelCased and our method names are usually PascalCased. Using function call syntax against a camelCased object can cause severe cognitive dissonance. I’ve still never really gotten used to it myself, and I always feel like I need a sit-down and a cup of coffee when it happens. Now we need to deal with the Configure method that sets up our processes. Rather than creating all those process classes, we need to create the delegate instances instead. You can construct a delegate instance just like any other object, using new, and passing the name of the function to which you wish to delegate as a constructor parameter: static DocumentProcessor Configure() { DocumentProcessor rc = new DocumentProcessor(); rc.Processes.Add(new DocumentProcess(DocumentProcesses.TranslateIntoFrench)); rc.Processes.Add(new DocumentProcess(DocumentProcesses.Spellcheck)); rc.Processes.Add(new DocumentProcess(DocumentProcesses.Repaginate)); return rc; } However, C# has more syntactic shorthand that can do away with a lot of that boil- erplate code. It can work out which delegate type you mean from context, and you only need to provide the method name itself: static DocumentProcessor Configure() { DocumentProcessor rc = new DocumentProcessor(); rc.Processes.Add(DocumentProcesses.TranslateIntoFrench); rc.Processes.Add(DocumentProcesses.Spellcheck); rc.Processes.Add(DocumentProcesses.Repaginate); return rc; } Not only have we achieved the same end in much less code, but we’ve actually reduced coupling between our subsystems still further—our DocumentProcessor doesn’t depend on any classes other than the Document itself; it will work with any class, static or oth- erwise, that can provide a method that conforms to the appropriate signature, as de- fined by our delegate. So far, we’ve only provided delegates to static functions, but this works just as well for an instance method on a class. Let’s imagine we need to provide a trademark filter for our document, to ensure that we pick out any trademarks in an appropriate typeface. Example 5-10 shows our TrademarkFilter class. Example 5-10. Another processing step class TrademarkFilter { readonly List trademarks = new List(); 154 | Chapter 5: Composability and Extensibility with Delegates public List Trademarks { get { return trademarks; } } public void HighlightTrademarks(Document doc) { // Split the document up into individual words string[] words = doc.Text.Split(' ', '.', ','); foreach( string word in words ) { if( Trademarks.Contains(word) ) { Console.WriteLine("Highlighting '{0}'", word); } } } } It maintains a list of Trademarks to pick out, and has a HighlightTrademarks method that does the actual work. Notice that it is coupled only to the Document—it knows nothing about our processor infrastructure. Neither have we burned our base; we didn’t have to inherit from any particular class to fit in with the processor framework, leaving it free for, say, our forthcoming “highlighter framework.” Example 5-11 shows how we add it to our configuration code. Example 5-11. Adding a processing step with a nonstatic method static DocumentProcessor Configure() { DocumentProcessor rc = new DocumentProcessor(); rc.Processes.Add(DocumentProcesses.TranslateIntoFrench); rc.Processes.Add(DocumentProcesses.Spellcheck); rc.Processes.Add(DocumentProcesses.Repaginate); TrademarkFilter trademarkFilter = new TrademarkFilter(); trademarkFilter.Trademarks.Add("O'Reilly"); trademarkFilter.Trademarks.Add("millennium"); rc.Processes.Add(trademarkFilter.HighlightTrademarks); return rc; } We create our TrademarkFilter object and add a few “trademarks” to its list. To specify a delegate to the method on that instance we use our reference to the instance and the name of the function on that instance. Notice that the syntax is very similar to a method call on an object, but without the parentheses. Functional Composition with delegate | 155 If we compile and run, we get the expected output: Processing document 1 Document traduit. Spellchecked document. Repaginated document. Processing document 2 Document traduit. Spellchecked document. Repaginated document. Highlighting 'millennium' This pattern is very common in object-oriented design: an overall process encapsulated in a class is customized by allowing a client to specify some action or actions for it to execute somewhere within that process. Our DocumentProcess delegate is typical for this kind of action—the function takes a single parameter of some type (the object our client wishes us to process), and returns void. Because we so often need delegates with this kind of signature, the framework provides us with a generic type that does away with the need to declare the delegate types ex- plicitly, every time. Generic Actions with Action Action is a generic type for a delegate to a function that returns void, and takes a single parameter of some type T. We used a generic type before: the List (List-of- T) where T represents the type of the objects that can be added to the list. In this case, we have an Action-of-T where T represents the type of the parameter for the function. So, instead of declaring our own delegate: delegate void DocumentProcess( Document doc ); we could just use an Action<> like this: Action A quick warning: although these are functionally equivalent, you cannot use an Action polymorphically as a DocumentProcess—they are, of course, different classes under the covers. We’re choosing between an implementation that uses a type we’re de- claring ourselves, or one supplied by the framework. Although there are sometimes good reasons for going your own way, it is usually best to take advantage of library code if it is an exact match for your requirement. So, we can delete our own delegate definition, and update our DocumentProcessor to use an Action instead, as shown in Example 5-12. 156 | Chapter 5: Composability and Extensibility with Delegates Example 5-12. Modifying the processor to use the built-in Action delegate type class DocumentProcessor { private readonly List> processes = new List>(); public List> Processes { get { return processes; } } public void Process(Document doc) { foreach (Action process in Processes) { process(doc); } } } Compile and run, and you’ll see that we still get our expected output. If you were watching the IntelliSense as you were typing in that code, you will have noticed that there are several Action<> types in the framework: Action, Action, Action, and so on. As you might expect, these allow you to define delegates to methods which return void, but which take two, three, or more parameters. .NET 4 provides Action<> delegate types going all the way up to 16 pa- rameters. (Previous versions stopped at four.) OK, let’s suppose that everything we’ve built so far has been deployed to the integration test environment, and the production folks have come back with a new requirement. Sometimes they configure a processing sequence that fails against a particular docu- ment—and it invariably seems to happen three hours into one of their more complex processes. They have some code which would let them do a quick check for some of their more compute-intensive processes and establish whether they are likely to fail. They want to know if we can implement this for them somehow. One way we might be able to do this is to provide a means of supplying an optional “check” function corresponding to each “action” function. We could then iterate all of the check functions first (they are supposed to be quick), and look at their return values. If any fail, we can give up (see Figure 5-3). We could implement that by rewriting our DocumentProcessor as shown in Exam- ple 5-13. Generic Actions with Action | 157 Example 5-13. Adding quick checking to the document processor class DocumentProcessor { class ActionCheckPair { public Action Action { get; set; } public Check QuickCheck { get; set; } } private readonly List processes = new List(); public void AddProcess(Action action) { AddProcess(action, null); } public void AddProcess(Action action, Check quickCheck) { processes.Add( new ActionCheckPair { Action = action, QuickCheck = quickCheck }); } public void Process(Document doc) { // First time, do the quick check foreach( ActionCheckPair process in processes) { if (process.QuickCheck != null && !process.QuickCheck(doc)) Figure 5-3. Document processor with checking 158 | Chapter 5: Composability and Extensibility with Delegates { Console.WriteLine("The process will not succeed."); return; } } // Then perform the action foreach (ActionCheckPair process in processes) { process.Action(doc); } } } There are quite a few new things to look at here. First, we declared a new class inside our DocumentProcessor definition, rather than in the namespace scope. We call this a nested class. We chose to nest the class because it is private to the DocumentProcessor, and we can avoid polluting the namespace with implementation details. Although you can make nested classes publicly accessible, it is unusual to do so and is considered a bad practice. This nested class just associates a pair of delegates: the Action that does the work, and the corresponding Check that performs the quick check. We removed the public property for our list of processes, and replaced it with a pair of AddProcess method overloads. These allow us to add processes to the sequence; one takes both the action and the check, and the other is a convenience overload that allows us to pass the action only. Notice how we had to change the public contract for our class because we initially exposed the list of processes directly. If we’d made the list an implementation detail and provided the single-parameter AddProcess method in the first place, we wouldn’t now need to change our clients as we’d only be extending the class. Our new Process function first iterates the processes and calls on the QuickCheck dele- gate (if it is not null) to see if all is OK. As soon as one of these checks returns false, we return from the method and do no further work. Otherwise, we iterate through the processes again and call the Action delegate. What type is a Check? We need a delegate to a method that returns a Boolean and takes a Document: delegate bool Check(Document doc); We call this type of “check” method a predicate: a function that operates on a set of parameters and returns either true or false for a given input. As you might expect, Generic Actions with Action | 159 given the way things have been going so far, this is a sufficiently useful idea for it to appear in the framework (again, as of .NET 3.5). Generic Predicates with Predicate Unlike the many variants of Action<>, the framework provides us with a single Predicate type, which defines a delegate to a function that takes a single parameter of type T and returns a Boolean. Why only the one parameter? There are good computer-science- philosophical reasons for it. In mathematical logic, a predicate is usually defined as follows: P : X → { true, false } That can be read as “a Predicate of some entity X maps to ‘true’ or ‘false’”. The single parameter in the mathematical expression is an im- portant limitation, allowing us to build more complex systems from the simplest possible building blocks. This formal notion gives rise to the single parameter in the .NET Predicate class, however pragmatically useful it may be to have more than one parameter in your particular application. We can delete our Check delegate (Hurrah! More code removed!), and replace it with a Predicate that takes a Document as its type parameter: Predicate And we can update the DocumentProcessor to make use of Predicate, as shown in Example 5-14. Example 5-14. DocumentProcessor updated to use Predicate class DocumentProcessor { class ActionCheckPair { public Action Action { get; set; } public Predicate QuickCheck { get; set; } } private readonly List processes = new List(); public void AddProcess(Action action) { AddProcess(action, null); } public void AddProcess(Action action, 160 | Chapter 5: Composability and Extensibility with Delegates Predicate quickCheck) { processes.Add( new ActionCheckPair { Action = action, QuickCheck = quickCheck }); } // ... } We can now update our client code to use our new DocumentProcessor API, calling AddProcess now that the list of processes is private (see Example 5-15). Example 5-15. Updating Configure to use modified DocumentProcessor static DocumentProcessor Configure() { DocumentProcessor rc = new DocumentProcessor(); rc.AddProcess(DocumentProcesses.TranslateIntoFrench); rc.AddProcess(DocumentProcesses.Spellcheck); rc.AddProcess(DocumentProcesses.Repaginate); TrademarkFilter trademarkFilter = new TrademarkFilter(); trademarkFilter.Trademarks.Add("Ian"); trademarkFilter.Trademarks.Add("Griffiths"); trademarkFilter.Trademarks.Add("millennium"); rc.AddProcess(trademarkFilter.HighlightTrademarks); return rc; } For the time being, we’re using the overload of AddProcess that doesn’t supply a quickCheck, so if we compile and run, we get the same output as before: Processing document 1 Document traduit. Spellchecked document. Repaginated document. Processing document 2 Document traduit. Spellchecked document. Repaginated document. Highlighting 'millennium' OK, the idea here was to allow our production team to quickly configure a check to see if the process was likely to fail, before embarking on a resource-intensive task. Let’s say DocumentProcesses.TranslateIntoFrench is a very time-consuming function, and they’ve discovered that any document whose text contains a question mark (?) will fail. They’ve raised a bug with the machine translation team, but they don’t want to hold up the entire production process until it is fixed—only 1 in 10 documents suffer from this problem. Generic Predicates with Predicate | 161 They need to add a quick check to go with the TranslateIntoFrench process. It is only one line of code: return !doc.Contains("?"); They could create a static class, with a static utility function to use as their predicate, but the boilerplate code would be about 10 times as long as the actual code itself. That’s a barrier to readability, maintenance, and therefore the general well-being of the de- veloper. C# comes to our rescue with a language feature called the anonymous method. Using Anonymous Methods An anonymous method is just like a regular function, except that it is inlined in the code at the point of use. Let’s update the code in our Configure function to include a delegate to an anonymous method to perform the check: rc.AddProcess( DocumentProcesses.TranslateIntoFrench, delegate(Document doc) { return !doc.Text.Contains("?"); }); The delegate to the anonymous method (i.e., the anonymous delegate) is passed as the second parameter to our AddProcess method. Let’s pull it out so that we can see it a little more clearly (there’s no need to make this change in your code; it is just for clarity): Predicate predicate = delegate(Document doc) { return !doc.Text.Contains("?"); } Written like this, it looks recognizably like a function definition, except that we use the delegate keyword to let the compiler know we are providing a delegate. There’s no need to specify the return type—that is inferred from the context. (In this case, the delegate type is Predicate, so the compiler knows the return type is bool.) Any parameters in our parameter list are accessible only inside the body of the anonymous method itself. Why do we call it an anonymous method? Because it doesn’t have a name that can be referenced elsewhere! The variable that references the delegate to the anonymous method has a name, but not the anonymous delegate type, or the anonymous method itself. If you compile and run the code you’ll see the new output: Processing document 1 The processing will not succeed 162 | Chapter 5: Composability and Extensibility with Delegates Processing document 2 Document traduit. Spellchecked document. Repaginated document. The production team is happy; but is the job done? Not quite; although this inline syntax for an anonymous method is a lot more compact than a static class/function declaration, we can get more compact and expressive still, using lambda expression syntax, which was added in C# 3.0 (anonymous methods having been around since C# 2.0). Creating Delegates with Lambda Expressions In the 1930s (a fertile time for computing theory!) two mathematicians named Church and Kleene devised a formal system for investigating the properties of functions. This was called lambda calculus, and (as further developed by Curry and others) it is still a staple part of computational theory for computer scientists. Fast-forward 70 or so years, and we see just a hint of this theory peeking through in C#’s lambda expressions—only a hint, though, so bear with it. As we saw before, you can think of a function as an expression that maps a set of inputs (the parameters) to an output (the return value). Mathematicians sometimes use a notation similar to this to define a function: (x,y,z) → x + y + z You can read this as defining a function that operates on three parameters (x, y, and z). The result of the function is just the sum of the three parameters, and, by definition, it can have no side effects on the system. The parameters themselves aren’t modified by the function; we just map from the input parameters to a result. Lambda expressions in C# use syntax very similar to this to define functional expres- sions. Here’s the C# equivalent of that mathematical expression we used earlier: (x,y,z) => x + y + z; Notice how it rather cutely uses => as the programming language equiv- alent of →. C++ users should not mistake this for the -> operator—it is quite different! This defines a lambda expression that takes three parameters and returns the sum of those three parameters. Creating Delegates with Lambda Expressions | 163 Some languages enforce the no side effects constraint; but in C# there is nothing to stop you from writing a lambda expression such as this one: (x,y,z) => { SomeStaticClass.CrashAndBurnAndMessWithEverything(); x.ModifyInternalState(); return x + y + z; } (Incidentally, this form of lambda expression, using braces to help define its body, is called a statement-form lambda.) In C#, a lambda is really just a concise way to write an anonymous method. We’re just writing normal code, so we can include operations that have side effects. So, although C# brings along some functional techniques with lambda syntax, it is not a “pure” functional language like ML or F#. Nor does it intend to be. So, what use is a lambda, then? We’ll see some very powerful techniques in Chapter 8 and Chapter 14, where lambdas play an important role in LINQ. Some of the data access features of the .NET Frame- work use the fact that we can convert lambdas into data structures called expression trees, which can be composed to create complex query-like expressions over various types of data. For now, we’re merely going to take advantage of the fact that we can implicitly create a delegate from a lambda, resulting in less cluttered code. How do we write our anonymous delegate as a lambda? Here’s the original: Predicate predicate = delegate(Document doc ) { return !doc.Text.Contains("?"); } And here it is rewritten using a lambda expression: Predicate predicate = doc => !doc.Text.Contains("?"); Compact, isn’t it! For a lot of developers, this syntax takes some getting used to, because it is completely unlike anything they’ve ever seen before. Where are the type declarations? Is this taking advantage of some of these dynamic programming techniques we’ve heard so much about? The short answer is no (but we’ll get to dynamic typing in Chapter 18, don’t worry). One of the nicer features of lambda expression syntax is that it takes care of working out what types the various parameters need to be, based on the context. In this case, the compiler knows that it needs to produce a Predicate, so it can infer that 164 | Chapter 5: Composability and Extensibility with Delegates the parameter type for the lambda must be a Document. You even get full IntelliSense on your lambda parameters in Visual Studio. It is well worth getting used to reading and writing lambdas; you’ll find them to be a very useful and expressive means of defining short func- tions, especially when we look at various aspects of the LINQ technol- ogies and expression composition in later chapters. Most developers, once they get over the initial comprehension hurdles, fall in love with lambdas—I promise! Delegates in Properties The delegates we’ve seen so far have taken one or more parameters, and returned either void (an Action<>) or a bool (a Predicate). But we can define a delegate to any sort of function we like. What if we want to provide a mechanism that allows the client to be notified when each processing step has been completed, and provide the processor with some text to insert into a process log? Our callback delegate might look like this: delegate string LogTextProvider(Document doc); We could add a property to our DocumentProcessor so that we can get and set the callback function (see Example 5-16). Example 5-16. A property that holds a delegate class DocumentProcessor { public LogTextProvider LogTextProvider { get; set; } // ... } And then we could make use of it in our Process method, as shown in Example 5-17. Example 5-17. Using a delegate in a property public void Process(Document doc) { // First time, do the quick check foreach (ActionCheckPair process in processes) { if (process.QuickCheck != null && !process.QuickCheck(doc)) { Console.WriteLine("The process will not succeed."); if (LogTextProvider != null) Delegates in Properties | 165 { Console.WriteLine(LogTextProvider(doc)); } return; } } // Then perform the action foreach (ActionCheckPair process in processes) { process.Action(doc); if (LogTextProvider != null) { Console.WriteLine(LogTextProvider(doc)); } } } Notice that we’re checking that our property is not null, and then we use standard delegate syntax to call the function that it references. Let’s set a callback in our client (see Example 5-18). Example 5-18. Setting a property with a lambda static void Main(string[] args) { // ... DocumentProcessor processor = Configure(); processor.LogTextProvider = (doc => "Some text for the log..."); // ... } Here we used a lambda expression to provide a delegate that takes a Document parameter called doc, and returns a string. In this case, it is just a constant string. Later, we’ll do some work to emit a more useful message. Take a moment to notice again how compact the lambda syntax is, and how the com- piler infers all those parameter types for us. Remember how much code we had to write to do this sort of thing back in the world of abstract base classes? Compile and run, and we see the following output: Processing document 1 The processing will not succeed. Some text for the log... Processing document 2 Document traduit. Some text for the log... Spellchecked document. Some text for the log... 166 | Chapter 5: Composability and Extensibility with Delegates Repaginated document. Some text for the log... Highlighting 'millennium' Some text for the log... That’s an example of a delegate for a function that returns something other than void or a bool. As you might have already guessed, the .NET Framework provides us with a generic type so that we don’t have to declare those delegates by hand, either. Generic Delegates for Functions The .NET Framework exposes a generic class called Func, which you can read as “Func-of T and TResult.” As with Predicate and Action the first type parameter determines the type of the first parameter of the function referenced by the delegate. Unlike Predicate or Action we also get to specify the type of the return value, using the last type parameter: TResult. Just like Action, there is a whole family of Func<> types which take one, two, three, and more parameters. Before .NET 4, Func<> went up to four parameters, but now goes all the way up to 16. So we could replace our custom delegate type with a Func<>. We can delete the delegate declaration: delegate string LogTextProvider(Document doc); and update the property: public Func LogTextProvider { get; set; } We can build and run without any changes to our client code because the new property declaration still expects a delegate for a function with the same signature. And we still get a bit of log text: Processing document 1 The processing will not succeed. Some text for the log... Processing document 2 Document traduit. Some text for the log... Spellchecked document. Some text for the log... Repaginated document. Generic Delegates for Functions | 167 Some text for the log... Highlighting 'millennium' Some text for the log... OK, let’s go back and have a look at that log function. As we noted earlier, it isn’t very useful right now. We can improve it by logging the name of the file we have processed after each output stage, to help the production team diagnose problems. Example 5-19 shows an update to the Main function to do that. Example 5-19. Doing more in our logging callback static void Main(string[] args) { Document doc1 = new Document { Author = "Matthew Adams", DocumentDate = new DateTime(2000, 01, 01), Text = "Am I a year early?" }; Document doc2 = new Document { Author = "Ian Griffiths", DocumentDate = new DateTime(2001, 01, 01), Text = "This is the new millennium, I promise you." }; Document doc3 = new Document { Author = "Matthew Adams", DocumentDate = new DateTime(2002, 01, 01), Text = "Another year, another document." }; string documentBeingProcessed = null; DocumentProcessor processor = Configure(); processor.LogTextProvider = (doc => documentBeingProcessed); documentBeingProcessed = "(Document 1)"; processor.Process(doc1); Console.WriteLine(); documentBeingProcessed = "(Document 2)"; processor.Process(doc2); Console.WriteLine(); documentBeingProcessed = "(Document 3)"; processor.Process(doc3); Console.ReadKey(); } We added a third document to the set, just so that we can see more get processed. Then we set up a local variable called documentBeingProcessed. As we move through the documents we update that variable to reflect our current status. How do we get that information into the lambda expression? Simple: we just use it! 168 | Chapter 5: Composability and Extensibility with Delegates Compile and run that code, and you’ll see the following output: The processing will not succeed. (Document 1) Document traduit. (Document 2) Spellchecked document. (Document 2) Repaginated document. (Document 2) Highlighting 'millennium' (Document 2) Document traduit. (Document 3) Spellchecked document. (Document 3) Repaginated document. (Document 3) We took advantage of the fact that an anonymous method has access to variables declared in its parent scope, in addition to anything in its own scope. For more information about this, see the sidebar below. Closures In general, we call an instance of a function and the set of variables on which it operates a closure. In a pure functional language, a closure is typically implemented by taking a snapshot of the values of the variables at the time at which the closure is created, along with a reference to the function concerned, and those values are immutable. In C#, a similar technique is applied—but the language allows us to modify those variables after the closure has been created. As we see in this chapter, we can use this to our advantage, but we have to be careful to understand and manage the scope of the variables in the closure to avoid peculiar side effects. We’ve seen how to read variables in our containing scope, but what about writing back to them? That works too. Let’s create a process counter that ticks up every time we execute a process, and add it to our logging function (see Example 5-20). Example 5-20. Modifying surrounding variables from a nested method static void Main(string[] args) { // ... (document setup) DocumentProcessor processor = Configure(); Generic Delegates for Functions | 169 string documentBeingProcessed = "(No document set)"; int processCount = 0; processor.LogTextProvider = (doc => { processCount += 1; return documentBeingProcessed; }); documentBeingProcessed = "(Document 1)"; processor.Process(doc1); Console.WriteLine(); documentBeingProcessed = "(Document 2)"; processor.Process(doc2); Console.WriteLine(); documentBeingProcessed = "(Document 3)"; processor.Process(doc3); Console.WriteLine(); Console.WriteLine("Executed " + processCount + " processes."); Console.ReadKey(); } We added a processCount variable at method scope, which we initialized to zero. We’ve switched our lambda expression into the statement form with the braces so that we can write multiple statements in the function body. In addition to returning the name of the document being processed, we also increment our processCount. Finally, at the end of processing, we write out a line that tells us how many processes we’ve executed. So our output looks like this: The processing will not succeed. (Document 1) Document traduit. (Document 2) Spellchecked document. (Document 2) Repaginated document. (Document 2) Highlighting 'millennium' (Document 2) Document traduit. (Document 3) Spellchecked document. (Document 3) Repaginated document. (Document 3) (Document 3) Executed 9 processes. OK, our production team is very happy with all of that, but they have another require- ment. Apparently, they have one team working on some diagnostic components that 170 | Chapter 5: Composability and Extensibility with Delegates are going to track the time taken to execute some of their processes, and another team developing some real-time display of all the processes as they run through the system. They want to know when a process is about to be executed and when it has completed so that these teams can execute some of their own code. Our first thought might be to implement a couple of additional callbacks: one called as processing starts, and the other as it ends; but that won’t quite meet their needs— they have two separate teams who both want, independently, to hook into it. We need a pattern for notifying several clients that something has occurred. The .NET Framework steps up with events. Notifying Clients with Events An event is raised (or sent) by a publisher (or sender) when something of interest occurs (such as an action taking place, or a property changing). Clients can subscribe to the event by providing a suitable delegate, rather like the callbacks we used previously. The method wrapped by the delegate is called the event handler. The neat thing is that more than one client can subscribe to the event. Here’s an example of a couple of events that we can add to the DocumentProcessor to help our production team: class DocumentProcessor { public event EventHandler Processing; public event EventHandler Processed; // ... } Notice that we use the keyword event to indicate that what follows is an event decla- ration. We then specify the delegate type for the event (EventHandler) and the name of the event (using PascalCasing). So, this is just like a declaration for a public field of type EventHandler, but annotated with the event keyword. What does this EventHandler delegate look like? The framework defines it like this: delegate void EventHandler(object sender, EventArgs e); Notice that our delegate takes two parameters. The first is a reference to the publisher of the event so that subscribers can tell who raised it. The second is some data associated with the event. The EventArgs class is defined in the framework, and is a placeholder for events that don’t need any extra information. We’ll see how to customize this later. Almost all events follow this two-argument pattern. Technically, they’re not required to—you can use any delegate type for an event. But in practice, this pattern is almost universal. Notifying Clients with Events | 171 So, how do we raise an event? Well, it really is just like a delegate, so we can use the delegate calling syntax as shown in the OnProcessing and OnProcessed methods in Ex- ample 5-21. Example 5-21. Raising events public void Process(Document doc) { OnProcessing(EventArgs.Empty); // First time, do the quick check foreach (ActionCheckPair process in processes) { if (process.QuickCheck != null && !process.QuickCheck(doc)) { Console.WriteLine("The process will not succeed."); if (LogTextProvider != null) { Console.WriteLine(LogTextProvider(doc)); } OnProcessed(EventArgs.Empty); return; } } // Then perform the action foreach (ActionCheckPair process in processes) { process.Action(doc); if (LogTextProvider != null) { Console.WriteLine(LogTextProvider(doc)); } } OnProcessed(EventArgs.Empty); } private void OnProcessing(EventArgs e) { if (Processing != null) { Processing(this, e); } } private void OnProcessed(EventArgs e) { if (Processed != null) { Processed(this, e); } } Notice how we pulled out the code to check for null and execute the delegate into functions called OnXXX. This isn’t strictly necessary, but it is a very common practice. 172 | Chapter 5: Composability and Extensibility with Delegates If we are designing our class as a base, we often mark this kind of method as a protected virtual so that derived classes can override the event- raising function instead of subscribing to the event. This can be more efficient than going through the event, and it allows us (optionally) to decline to raise the event by not calling on the base implementation. Be careful to document whether derived classes are allowed not to call the base, though! Now we need to subscribe to those events. So let’s create a couple of classes to simulate what the production department would need to do (see Example 5-22). Example 5-22. Subscribing to and unsubscribing from events class ProductionDeptTool1 { public void Subscribe ( DocumentProcessor processor ) { processor.Processing += processor_Processing; processor.Processed += processor_Processed; } public void Unsubscribe(DocumentProcessor processor) { processor.Processing -= processor_Processing; processor.Processed -= processor_Processed; } void processor_Processing(object sender, EventArgs e) { Console.WriteLine("Tool1 has seen processing."); } void processor_Processed(object sender, EventArgs e) { Console.WriteLine("Tool1 has seen that processing is complete."); } } class ProductionDeptTool2 { public void Subscribe( DocumentProcessor processor ) { processor.Processing += (sender, e) => Console.WriteLine("Tool2 has seen processing."); processor.Processed += (sender, e) => Console.WriteLine("Tool2 has seen that processing is complete."); } } Notifying Clients with Events | 173 To subscribe to an event we use the += operator, with a suitable delegate. You can see in ProductionDeptTool1.Subscribe that we used the standard delegate syntax, and in ProductionDeptTool2.Subscribe we used the lambda expression syntax. Of course, you don’t have to subscribe to events in methods called Subscribe—you can do it anywhere you like! When you’re done watching an event for any reason, you can unsubscribe using the -= operator and another delegate to the same method. You can see that in the ProductionDeptTool1.Unsubscribe method. When you subscribe to an event your subscriber implicitly holds a reference to the publisher. This means that the garbage collector won’t be able to collect the publisher if there is still a rooted reference to the subscriber. It is a good idea to provide a means of unsubscribing from events you are no longer actively observing, to avoid growing your working set unnecessarily. Let’s add some code to our Main method to make use of the two new tools, as shown in Example 5-23. Example 5-23. Updated Main method static void Main(string[] args) { // ... ProductionDeptTool1 tool1 = new ProductionDeptTool1(); tool1.Subscribe(processor); ProductionDeptTool2 tool2 = new ProductionDeptTool2(); tool2.Subscribe(processor); documentBeingProcessed = "(Document 1)"; // ... Console.ReadKey(); } If we compile and run, we now see the following output: Tool1 has seen processing. Tool2 has seen processing. The processing will not succeed. (Document 1) Too11 has seen that processing is complete. Tool2 has seen that processing is complete. Tool1 has seen processing. Tool2 has seen processing. 174 | Chapter 5: Composability and Extensibility with Delegates Document traduit. (Document 2) Spellchecked document. (Document 2) Repaginated document. (Document 2) Highlighting 'millennium' (Document 2) Too11 has seen that processing is complete. Tool2 has seen that processing is complete. Tool1 has seen processing. Tool2 has seen processing. Document traduit. (Document 3) Spellchecked document. (Document 3) Repaginated document. (Document 3) (Document 3) Too11 has seen that processing is complete. Tool2 has seen that processing is complete. Executed 9 processes. You might notice that the event handlers have been executed in the order in which we added them. This is not guaranteed to be the case, and you cannot depend on this behavior. If you need deterministic ordering (as we did for our processes, for ex- ample) you should not use an event. Earlier, I alluded to the fact that we can customize the data we send through with the event. We do this by deriving our own class from EventArgs, and adding extra properties or methods to it. Let’s say we want to send the current document through in the event; we can create a class like the one shown in Example 5-24. Example 5-24. Custom event arguments class class ProcessEventArgs : EventArgs { // Handy constructor public ProcessEventArgs(Document document) { Document = document; } // The extra property // We don't want subscribers to be able // to update this property, so we make // it private // (Of course, this doesn't prevent them // from changing the Document itself) Notifying Clients with Events | 175 public Document Document { get; private set; } } We also need to create a suitable delegate for the event, one that takes a ProcessEven tArgs as its second parameter rather than the EventArgs base class. We could do this by hand, sticking to the convention of calling the first parameter sender and the data parameter e: delegate void ProcessEventHandler(object sender, ProcessEventArgs e); Once again, this is such a common thing to need that the framework provides us with a generic type, EventHandler, to save us the boilerplate code. So we can replace the ProcessEventHandler with an EventHandler. Let’s update our event declarations (see Example 5-25). Example 5-25. Updated event members public event EventHandler Processing; public event EventHandler Processed; and then our helper methods which raise the event that will need to take a ProcessE ventArgs (see Example 5-26). Example 5-26. Updated code for raising events private void OnProcessing(ProcessEventArgs e) { if (Processing != null) { Processing(this, e); } } private void OnProcessed(ProcessEventArgs e) { if (Processed != null) { Processed(this, e); } } And finally, our calls to those methods will need to create an appropriate ProcessEven tArgs object, as shown in Example 5-27. Example 5-27. Creating the event arguments object public void Process(Document doc) { ProcessEventArgs e = new ProcessEventArgs(doc); OnProcessing(e); 176 | Chapter 5: Composability and Extensibility with Delegates // First time, do the quick check foreach (ActionCheckPair process in processes) { if (process.QuickCheck != null && !process.QuickCheck(doc)) { Console.WriteLine("The process will not succeed."); if (LogTextProvider != null) { Console.WriteLine(LogTextProvider(doc)); } OnProcessed(e); return; } } // Then perform the action foreach (ActionCheckPair process in processes) { process.Action(doc); if (LogTextProvider != null) { Console.WriteLine(LogTextProvider(doc)); } } OnProcessed(e); } Notice how we happen to reuse the same event data for each event we raise. That’s safe to do because our event argument instance cannot be modified—its only property has a private setter. If it were possible for event handlers to change the event argument object, it would be risky to use the same one for both events. We could offer our colleagues on the production team another facility using these events. We already saw how they need to perform a quick check before each individual process to determine whether they should abort processing. We can take advantage of our Processing event to give them the option of canceling the whole process before it even gets off the ground. The framework defines a class called CancelEventArgs which adds a Boolean property called Cancel to the basic EventArgs. Subscribers can set the property to True, and the publisher is expected to abort the operation. Let’s add a new EventArgs class for that (see Example 5-28). Example 5-28. A cancelable event argument class class ProcessCancelEventArgs : CancelEventArgs { public ProcessCancelEventArgs(Document document) { Document = document; } public Document Document Notifying Clients with Events | 177 { get; private set; } } We’ll update the declaration of our Processing event, and its corresponding helper, as shown in Example 5-29 (but we’ll leave the Processed event as it is—if the document has already been processed, it’s too late to cancel it). Example 5-29. A cancelable event public event EventHandler Processing; private void OnProcessing(ProcessCancelEventArgs e) { if (Processing != null) { Processing(this, e); } } Finally, we need to update the Process method to create the right kind of event argu- ment object, and to honor requests for cancellation (see Example 5-30). Example 5-30. Supporting cancellation public void Process(Document doc) { ProcessEventArgs e = new ProcessEventArgs(doc); ProcessCancelEventArgs ce = new ProcessCancelEventArgs(doc); OnProcessing(ce); if (ce.Cancel) { Console.WriteLine("Process canceled."); if (LogTextProvider != null) { Console.WriteLine(LogTextProvider(doc)); } return; } // ... } Now we’ll make use of this in one of our production tools, as shown in Example 5-31. Example 5-31. Taking advantage of cancelability class ProductionDeptTool1 { public void Subscribe(DocumentProcessor processor) { processor.Processing += processor_Processing; processor.Processed += processor_Processed; } 178 | Chapter 5: Composability and Extensibility with Delegates public void Unsubscribe(DocumentProcessor processor) { processor.Processing -= processor_Processing; processor.Processed -= processor_Processed; } void processor_Processing(object sender, ProcessCancelEventArgs e) { Console.WriteLine("Tool1 has seen processing, and not canceled."); } void processor_Processed(object sender, EventArgs e) { Console.WriteLine("Tool1 has seen that processing is complete."); } } class ProductionDeptTool2 { public void Subscribe(DocumentProcessor processor) { processor.Processing += (sender, e) => { Console.WriteLine("Tool2 has seen processing and canceled it"); if(e.Document.Text.Contains("document")) { e.Cancel = true; } }; processor.Processed += (sender, e) => Console.WriteLine("Tool2 has seen that processing is complete."); } } Notice how we don’t have to update the event data parameter—we can take advantage of polymorphism and just refer to it through its base type, unless we want to take advantage of its new features. In the lambda expression syntax, of course, the new type parameter is inferred and we don’t have to change anything; we can just update the handler in ProductionDeptTool2 to cancel if it sees the text "document". If we compile and run, we now see the following output: The process will not succeed. (Document 1) Tool1 has seen that processing is complete. Tool2 has seen that processing is complete. Tool1 has seen processing, and not canceled. Tool2 has seen processing, and not canceled. Document traduit. (Document 2) Spellchecked document. (Document 2) Repaginated document. Notifying Clients with Events | 179 (Document 2) Highlighting 'millennium' (Document 2) Tool1 has seen that processing is complete. Tool2 has seen that processing is complete. Tool1 has seen processing, and not canceled. Tool2 has seen processing and canceled. Process canceled. (Document 3) Executed 6 processes. So we have our cancellation behavior, but we have to be very careful. Notice that Tool1 happened to see the event first, and it happily executed its handler, before Tool2 got in and canceled the whole thing. When you write handlers for cancelable events, you must ensure that it doesn’t matter if some or all of those handlers never get called and that they behave correctly if the action they expect never actually occurs. Cancelable events need very careful documentation to indicate how they relate to the actions around them, and the exact semantics of cancellation. It is therefore (in general) a bad idea to do what we have just done, and convert a noncancelable event into a cancelable one, if your code has already shipped; you stand a very good chance of breaking any clients that just recompile successfully against the new version. Exposing Large Numbers of Events Some classes (particularly those related to user interactions) need to expose a very large number of events. If you use the normal event syntax shown in the preceding examples, storage is allocated for every single event you declare, even if the events have no sub- scribers. This means that objects of this type can get very large, very quickly. To avoid this situation, C# provides you with the ability to manage storage for the events yourself, using syntax similar to a property getter and setter, with your own backing storage: public event EventHandler MyEvent { add { // Code to add handler here } remove { // Code to remove handler here } } Typically, you use a Dictionary to create the backing storage for the event only when it gets its first subscriber. (Dictionaries are described in Chapter 9.) 180 | Chapter 5: Composability and Extensibility with Delegates Example 5-32 updates the DocumentProcessor we’re developing in this chapter to use a dictionary for the backing storage for its events. Example 5-32. Custom event storage class DocumentProcessor { private Dictionary events; public event EventHandler Processing { add { Delegate theDelegate = EnsureEvent("Processing"); events["Processing"] = ((EventHandler) theDelegate) + value; } remove { Delegate theDelegate = EnsureEvent("Processing"); events["Processing"] = ((EventHandler) theDelegate) - value; } } public event EventHandler Processed { add { Delegate theDelegate = EnsureEvent("Processed"); events["Processed"] = ((EventHandler) theDelegate) + value; } remove { Delegate theDelegate = EnsureEvent("Processed"); events["Processed"] = ((EventHandler) theDelegate) - value; } } private Delegate EnsureEvent(string eventName) { // Construct the dictionary if it doesn't already // exist if (events == null) { Notifying Clients with Events | 181 events = new Dictionary(); } // Add a placeholder for the delegate if we don't // have it already Delegate theDelegate = null; if (!events.TryGetValue( eventName, out theDelegate)) { events.Add(eventName, null); } return theDelegate; } private void OnProcessing(ProcessCancelEventArgs e) { Delegate eh = null; if( events != null && events.TryGetValue("Processing", out eh) ) { EventHandler pceh = eh as EventHandler; if (pceh != null) { pceh(this, e); } } } private void OnProcessed(ProcessEventArgs e) { Delegate eh = null; if (events != null && events.TryGetValue("Processed", out eh)) { EventHandler pceh = eh as EventHandler; if (pceh != null) { pceh(this, e); } } } // ... } Obviously, that’s a lot more complex than the automatic method, and you would not normally use it for a class that exposes just a couple of events, but it can save a lot of working set for classes that are either large in number, or publish a large number of events but have few subscribers. 182 | Chapter 5: Composability and Extensibility with Delegates Summary In this chapter, we saw how functional techniques provide powerful reuse and exten- sibility mechanisms for our programs, in ways that can be more flexible and yet simpler than class-based approaches. We also saw how events enabled a publisher-to-multiple- subscribers relationship. In the next chapter, we’ll look at how we deal with unexpected situations: errors, failures, and exceptions. Summary | 183 CHAPTER 6 Dealing with Errors Errors happen all the time; they’re a fact of life: • Despite the best efforts of Microsoft Word, an army of highly skilled reviewers and editors, and even your authors, it would be surprising if there wasn’t a typograph- ical error in a book of this length. • Although they are relatively few and far between, there are bugs in the .NET Framework—hence the need for occasional service packs. • You might type your credit card number for an online transaction and accidentally transpose two digits; or forget to type in the expiration date. Like it or not, we’re going to have to face up to the fact that there are going to be errors of all kinds to deal with in our software too. In this chapter, we’ll look at various types of errors, the tools that C# and the .NET Framework give us to deal with them, and some strategies for applying those tools. First, we need to recognize that all errors are not made the same. We’ve classified a few of the more common ones in Table 6-1. Table 6-1. A far-from-exhaustive list of some common errors Error Description/example Bug A failure to implement a contract according to its documentation. Unexpected behavior A failure to document a contract properly for all expected input. Unexpected input A client passes data to a method that is outside some expected range. Unexpected data type A client passes data to a method that is not of the expected type. Unexpected data format A client passes data to a method in a format that is not recognized. Unexpected result A client receives information from a method that it did not expect for the given input. Unexpected method call The class wasn’t expecting you to call a particular method at that time—you hadn’t performed some required initialization, for example. Unavailable resource A method tried to access a resource of some kind and it was not present—a hardware device was not plugged in, for instance. 185 Error Description/example Contended resource A method tried to access a scarce resource of some kind (memory or a hardware device that cannot be shared), and it was not available because someone else was using it. Although bugs are probably the most obvious type of error, we won’t actually be dealing with them directly in this chapter. We will, however, look at how our error-handling techniques can make it easier (or harder!) to find the bugs that are often the cause of the other, better defined issues. Let’s get started with an example we can use to look at error-handling techniques. We’re going to branch out into the world of robotics for this one, and build a turtle-controlling application. The real-world turtle is a rectangular piece of board on which are mounted two motors that can drive two wheels. The wheels are located in the middle of the left and right edges of the board, and there are nondriven castor wheels at the front and back to give it a bit of stability. We can drive the two motors independently: we can move forward, move backward, or stop. And by moving the wheels in different direc- tions, or moving one wheel at time, we can steer it about a bit like a tank. Let’s create a class to model our turtle (see Example 6-1). Example 6-1. The Turtle class class Turtle { // The width of the platform public double PlatformWidth { get; set; } // The height of the platform public double PlatformHeight { get; set; } // The speed at which the motors drive the wheels, // in meters per second. For ease, we assume that takes account // of the distance traveled by the tires in contact // with the ground, and any slipping public double MotorSpeed { get; set; } // The state of the left motor public MotorState LeftMotorState { get; set; } 186 | Chapter 6: Dealing with Errors // The state of the right motor public MotorState RightMotorState { get; set; } // The current position of the turtle public Point CurrentPosition { get; private set; } // The current orientation of the turtle public double CurrentOrientation { get; private set; } } // The current state of a motor enum MotorState { Stopped, Running, Reversed } In addition to the motor control, we can define the size of the platform and the speed at which the motors rotate the wheels. We also have a couple of properties that tell us where the turtle is right now, relative to its point of origin, and the direction in which it is currently pointing. To make our turtle simulator actually do something, we can add a method which makes time pass. This looks at the state of the different motors and applies an appropriate algorithm to calculate the new position of the turtle. Example 6-2 shows our first, somewhat naive, go at it. Example 6-2. Simulating turtle motion // Run the turtle for the specified duration public void RunFor(double duration) { if (LeftMotorState == MotorState.Stopped && RightMotorState == MotorState.Stopped) { // If we are at a full stop, nothing will happen return; } // The motors are both running in the same direction // then we just drive if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Running) || (LeftMotorState == MotorState.Reversed && Dealing with Errors | 187 RightMotorState == MotorState.Reversed)) { Drive(duration); return; } // The motors are running in opposite directions, // so we don't move, we just rotate about the // center of the rig if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Reversed) || (LeftMotorState == MotorState.Reversed && RightMotorState == MotorState.Running)) { Rotate(duration); return; } } If both wheels are pointing in the same direction (forward or reverse), we drive (or reverse) in the direction we are pointing. If they are driving in opposite directions, we rotate about our center. If both are stopped, we will remain stationary. Example 6-3 shows the implementations of Drive and Rotate. They use a little bit of trigonometry to get the job done. Example 6-3. Simulating rotation and movement private void Rotate(double duration) { // This is the total circumference of turning circle double circum = Math.PI * PlatformWidth; // This is the total distance traveled double d = duration * MotorSpeed; if (LeftMotorState == MotorState.Reversed) { // And we're going backwards if the motors are reversed d *= -1.0; } // So we've driven it this proportion of the way round double proportionOfWholeCircle = d / circum; // Once round is 360 degrees (or 2pi radians), so we have traveled // this far: CurrentOrientation = CurrentOrientation + (Math.PI * 2.0 * proportionOfWholeCircle); } private void Drive(double duration) { // This is the total distance traveled double d = duration * MotorSpeed; if (LeftMotorState == MotorState.Reversed) { // And we're going backwards if the motors are reversed d *= -1.0; 188 | Chapter 6: Dealing with Errors } // Bit of trigonometry for the change in the x,y coordinates double deltaX = d * Math.Sin(CurrentOrientation); double deltaY = d * Math.Cos(CurrentOrientation); // And update the position CurrentPosition = new Point(CurrentPosition.X + deltaX, CurrentPosition.Y + deltaY); } Let’s write a quick test program to see whether the code we’ve written actually does what we expect (see Example 6-4). Example 6-4. Testing the turtle static void Main(string[] args) { // Here's our turtle Turtle arthurTheTurtle = new Turtle {PlatformWidth = 10.0, PlatformHeight = 10.0, MotorSpeed = 5.0}; ShowPosition(arthurTheTurtle); // We want to proceed forwards arthurTheTurtle.LeftMotorState = MotorState.Running; arthurTheTurtle.RightMotorState = MotorState.Running; // For two seconds arthurTheTurtle.RunFor(2.0); ShowPosition(arthurTheTurtle); // Now, let's rotate clockwise for a bit arthurTheTurtle.RightMotorState = MotorState.Reversed; // PI / 2 seconds should do the trick arthurTheTurtle.RunFor(Math.PI / 2.0); ShowPosition(arthurTheTurtle); // And let's go into reverse arthurTheTurtle.RightMotorState = MotorState.Reversed; arthurTheTurtle.LeftMotorState = MotorState.Reversed; // And run for 5 seconds arthurTheTurtle.RunFor(5); ShowPosition(arthurTheTurtle); // Then rotate back the other way arthurTheTurtle.RightMotorState = MotorState.Running; // And run for PI/4 seconds to give us 45 degrees arthurTheTurtle.RunFor(Math.PI / 4.0); ShowPosition(arthurTheTurtle); Dealing with Errors | 189 // And finally drive backwards for a bit arthurTheTurtle.RightMotorState = MotorState.Reversed; arthurTheTurtle.LeftMotorState = MotorState.Reversed; arthurTheTurtle.RunFor(Math.Cos(Math.PI / 4.0)); ShowPosition(arthurTheTurtle); Console.ReadKey(); } private static void ShowPosition(Turtle arthurTheTurtle) { Console.WriteLine( "Arthur is at ({0}) and is pointing at angle {1:0.00} radians.", arthurTheTurtle.CurrentPosition, arthurTheTurtle.CurrentOrientation); } We chose the times for which to run quite carefully so that we end up going through relatively readable distances and angles. (Hey, someone could design a more usable facade over this API!) If we compile and run, we see the following output: Arthur is at (0,0) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 1.57 radians. Arthur is at (-25,10) and is pointing at angle 1.57 radians. Arthur is at (-25,10) and is pointing at angle 0.79 radians. Arthur is at (-27.5,7.5) and is pointing at angle 0.79 radians. OK, that seems fine for basic operation. But what happens if we change the width of the platform to zero? Turtle arthurTheTurtle = new Turtle { PlatformWidth = 0.0, PlatformHeight = 10.0, MotorSpeed = 5.0 }; Not only does that not make much sense, but the output is not very useful either; clearly we have divide-by-zero problems: Arthur is at (0,0) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle Infinity radians. Arthur is at (NaN,NaN) and is pointing at angle Infinity radians. Arthur is at (NaN,NaN) and is pointing at angle NaN radians. Arthur is at (NaN,NaN) and is pointing at angle NaN radians. Clearly, our real-world turtle could go badly wrong if we told it to rotate through an infinite angle. At the very least, we’d get bored waiting for it to finish. We should prevent the user from running it if the PlatformWidth is less than or equal to zero. Previously, we used the following code: // Run the turtle for the specified duration public void RunFor(double duration) { 190 | Chapter 6: Dealing with Errors if (PlatformWidth <= 0.0) { // What to do here? } // ... } That detects the problem, but what should we do if our particular turtle is not set up correctly? Previously, we silently ignored the problem, and returned as though every- thing was just fine. Is that really what we want to do? For this application it might be perfectly safe, but what if another developer uses our turtle with a paintbrush strapped to its back, to paint the lines on a tennis court? The developer added a few extra moves at the beginning of his sequence, and he didn’t notice that he had inadvertently done so before he initialized the PlatformWidth. We could have a squiggly paint disaster on our hands! Not a Number? The System.Double type defines a number of constant values that are used to represent some very interesting doubles: • Double.NaN is the result of dividing zero by zero (e.g., 0.0/0.0). • Double.NegativeInfinity is the result of dividing a negative number by zero (e.g., −1.0/0.0). • Double.PositiveInfinity is the result of dividing a positive number by zero (e.g., 1.0/0.0). They also behave in interesting ways. For example, you can’t compare one of these special values with another (e.g., (0.0/0.0 != Double.NaN)). Instead, you have to use helper methods such as these: • Double.IsNaN(0.0/0.0) • Double.IsPositiveInfinity(1.0/0.0) • Double.IsNegativeInfinity(-1.0/0.0) If you don’t care whether it is a positive or a negative infinity, just some sort of infinity, you can use this helper: Double.IsInfinity(1.0/0.0). Be very careful when playing with infinities, as you can easily get into trouble! When and How to Fail Choosing when and how to fail is one of the big debates in software development. There is a lot of consensus about what we do, but things are much less clear-cut when it comes to failures. When and How to Fail | 191 You have a number of choices: 1. Try to plow on regardless. 2. Try to make sense of what has happened and work around it. 3. Return an error of some kind to your caller, and hope the caller knows what to do with it. 4. Stop. At the moment, we’re using option 1: try to plow on regardless; and you can see that this might or might not be dangerous. The difficulty is that we can be sure it is safe only if we know why our client is calling us. Given that we can’t possibly have knowledge of the continuum of all possible clients (and their clients, and their clients’ clients), plugging on regardless is, in general, not safe. We might be exposing ourselves to all sorts of security problems and data integrity issues of which we cannot be aware at this time. What about option 2? Well, that is really an extension of the contract: we’re saying that particular types of data outside the range we previously defined are valid, it is just that we’ll special-case them to other values. This is quite common with range proper- ties, where we clamp values outside the range to the minimum and maximum permitted values. Example 6-5 shows how we could implement that. Example 6-5. Range checking class Turtle { // The width of the platform must be between 1.0 and 10.0 inclusive // Values outside this range will be coerced into the range. private double platformWidth; public double PlatformWidth { get { return platformWidth; } set { platformWidth = value; EnsurePlatformSize(); } } // The height of the platform must be between 1.0 and 10.0 inclusive // Values outside this range will be coerced into the range. private double platformHeight; public double PlatformHeight { get { return platformHeight; } set { platformHeight = value; EnsurePlatformSize(); } } 192 | Chapter 6: Dealing with Errors // The new constructor initializes the platform size appropriately public Turtle() { EnsurePlatformSize(); } // This method enforces the newly documented constraint // we added to the contract private void EnsurePlatformSize() { if (PlatformWidth < 1.0) { PlatformWidth = 1.0; } if (PlatformWidth > 10.0) { PlatformWidth = 10.0; } if (PlatformHeight < 1.0) { PlatformHeight = 1.0; } if (PlatformHeight > 10.0) { PlatformHeight = 10.0; } } // ... } Here we documented a constraint in our contract, and enforced that constraint first at construction, and then whenever clients attempt to modify the value. We chose to enforce that constraint at the point when the value can be changed because that makes the effect of the constraint directly visible. If users set an out-of-bounds value and read it back they can immediately see the effect of the constraint on the property. That’s not the only choice, of course. We could have done it just before we used it—but if we changed the implementation, or added features, we might have to add lots of calls to EnsurePlatformSize, and you can be certain that we’d forget one somewhere. When we run the application again, we see the following output: Arthur is at (0,0) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 15.71 radians. Arthur is at (-1.53075794227797E-14,35) and is pointing at angle 15.71 radians. Arthur is at (-1.53075794227797E-14,35) and is pointing at angle 7.85 radians. Arthur is at (-3.53553390593275,35) and is pointing at angle 7.85 radians. Although this is a very useful technique, and it has clearly banished those less-than- useful NaNs, we have to consider: is this the right solution for this particular problem? Let’s think about our tennis-court-painting robot again. Would we really want it to When and How to Fail | 193 paint the court as though it were a 1-meter-wide robot, just because we forgot to ini- tialize it? Looking at the distances traveled and the angles through which it has turned, the answer is clearly no! Constraints such as this are useful in lots of cases. We might want to ensure that some UI element not extend off the screen, or grow too big or small, for example. But equally, an online banking application that doesn’t permit transactions less than $10 shouldn’t just clamp the amount the user entered from $1 to $10 and carry on happily! So let’s backtrack a little and look at another option: returning a value that signifies an error. Returning Error Values For many years, programmers have written methods that detect errors, and which re- port those errors by returning an error code. Typically, this is a Boolean value of some kind, with True representing success and False failure. Or you might use either an int or an enum if you need to distinguish lots of different types of errors. Before we add an error return value, we should remove the code we just added that silently enforces the constraints. We can delete EnsurePlat formSize and any references to it. (Or if you’re following along in Visual Studio and don’t want to delete the code, just comment out all the rel- evant lines.) So where are we going to return the error from? Our first instinct might be to put it in the RunFor method, where we suggested earlier; but look at the code—there’s nothing substantive there. The problem actually occurs in Rotate. What happens if we change the Rotate method later so that it depends on different properties? Do we also update RunFor to check the new constraints? Will we remember? It is Rotate that actually uses the properties, so as a rule of thumb we should do the checking there. It will also make the debugging easier later—we can put breakpoints near the origin of the error and see what is going wrong. Let’s change the Rotate method and see what happens (see Example 6-6). Example 6-6. Indicating errors through the return value private bool Rotate(double duration) { if (PlatformWidth <= 0.0) { return false; } 194 | Chapter 6: Dealing with Errors // This is the total circumference of turning circle double circum = Math.PI * PlatformWidth; // This is the total distance traveled double d = duration * MotorSpeed; if (LeftMotorState == MotorState.Reversed) { // And we're going backwards if the motors are reversed d *= -1.0; } // So we've driven it this proportion of the way round double proportionOfWholeCircle = d / circum; // Once round is 360 degrees (or 2pi radians), so we have traveled CurrentOrientation = CurrentOrientation + (Math.PI * 2.0 * proportionOfWholeCircle); return true; } If we compile and run with our all-new error checking added, we see the following output: Arthur is at (0,0) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 0.00 radians. Arthur is at (0,-15) and is pointing at angle 0.00 radians. Arthur is at (0,-15) and is pointing at angle 0.00 radians. Arthur is at (0,-18.5355339059327) and is pointing at angle 0.00 radians. Hmmm; that’s not very good. Rotate has indeed failed, but we’ve carried on driving the turtle up and down that line because we didn’t do anything with the return value. This is the great benefit, and great downside, of error return values: you can just ignore them. Let’s look at where we call Rotate and see what we can do with that error (see Exam- ple 6-7). Example 6-7. Detecting failure and then wondering what to do with it // Run the turtle for the specified duration public void RunFor(double duration) { // ... // The motors are running in opposite directions, // so we don't move, we just rotate about the // center of the rig if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Reversed) || (LeftMotorState == MotorState.Reversed && RightMotorState == MotorState.Running)) { Returning Error Values | 195 if (!Rotate(duration)) { // It failed, so what now? } return; } } It is simple enough to check to see if it failed, but what are we actually going to do about it? Is there any action we can take? Not surprisingly, the answer is no—we know no more about the needs of our caller than we did when we were discussing the other options. So we are going to have to pass the error on up. Example 6-8 shows an im- plementation of Run that does that. Example 6-8. Passing the buck // Run the turtle for the specified duration // Returns false if there was a failure // Or true if the run succeeded public bool RunFor(double duration) { if (LeftMotorState == MotorState.Stopped && RightMotorState == MotorState.Stopped) { // If we are at a full stop, nothing will happen return true; } // The motors are both running in the same direction // then we just drive if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Running) || (LeftMotorState == MotorState.Reversed && RightMotorState == MotorState.Reversed)) { Drive(duration); return true; } // The motors are running in opposite directions, // so we don't move, we just rotate about the // center of the rig if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Reversed) || (LeftMotorState == MotorState.Reversed && RightMotorState == MotorState.Running)) { return Rotate(duration); } // We didn't expect to get here return false; } 196 | Chapter 6: Dealing with Errors Notice that we updated our documentation for the public method as we changed the contract. We also have to return values from all of the exit points of our method. That has exposed another problem with our implementation: we never supported one motor at the stop condition, and the other at the driving or reversing condition. Well, that’s fine—we can return an error if we hit those conditions now. One problem with this contract is that we can’t tell why our error occurred. Was it due to the state of the motors, or a problem with Rotate? We could create an enum that lets us distinguish between these error types: enum TurtleError { OK, RotateError, MotorStateError } Then we could use the enum as shown in Example 6-9. Example 6-9. Indicating errors with an enum // Run the turtle for the specified duration // Returns one of the TurtleError values if a failure // occurs, or TurtleError.OK if it succeeds public TurtleError RunFor(double duration) { if (LeftMotorState == MotorState.Stopped && RightMotorState == MotorState.Stopped) { // If we are at a full stop, nothing will happen return TurtleError.OK; } // The motors are both running in the same direction // then we just drive if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Running) || (LeftMotorState == MotorState.Reversed && RightMotorState == MotorState.Reversed)) { Drive(duration); return TurtleError.OK; } // The motors are running in opposite directions, // so we don't move, we just rotate about the // center of the rig if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Reversed) || (LeftMotorState == MotorState.Reversed && RightMotorState == MotorState.Running)) { if (!Rotate(duration)) { Returning Error Values | 197 return TurtleError.RotateError; } } return TurtleError.MotorStateError; } OK so far, although it is starting to get a bit tortuous, and we’re going up only one call in the stack. But let’s build and run anyway: Arthur is at (0,0) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 0.00 radians. Arthur is at (0,-15) and is pointing at angle 0.00 radians. Arthur is at (0,-15) and is pointing at angle 0.00 radians. Arthur is at (0,-18.5355339059327) and is pointing at angle 0.00 radians. Yup; we’re no better off than before, because all we’ve done is to pass the responsibility up to the client, and they are still free to ignore our pleadings. Given that the problem is a result of our oversight in the first place, what is the likelihood that we’ll remember to check the error message? Things would be even worse if this was in a library; we could recompile against this new version, and everything would seem fine, while in the background everything would quietly be going horribly wrong. It is probably about time we did something with the error message, so let’s see what happens in our client code (see Example 6-10). Example 6-10. Handling an error static void Main(string[] args) { Turtle arthurTheTurtle = new Turtle { PlatformWidth = 0.0, PlatformHeight = 10.0, MotorSpeed = 5.0 }; ShowPosition(arthurTheTurtle); // We want to proceed forwards arthurTheTurtle.LeftMotorState = MotorState.Running; arthurTheTurtle.RightMotorState = MotorState.Running; // For two seconds TurtleError result = arthurTheTurtle.RunFor(2.0); if (result != TurtleError.OK) { HandleError(result); return; } ShowPosition(arthurTheTurtle); 198 | Chapter 6: Dealing with Errors // Now, let's rotate clockwise for a bit arthurTheTurtle.RightMotorState = MotorState.Reversed; // PI / 2 seconds should do the trick result = arthurTheTurtle.RunFor(Math.PI / 2.0); if (result != TurtleError.OK) { HandleError(result); return; } ShowPosition(arthurTheTurtle); // And let's go into reverse arthurTheTurtle.RightMotorState = MotorState.Reversed; arthurTheTurtle.LeftMotorState = MotorState.Reversed; // And run for 5 seconds result = arthurTheTurtle.RunFor(5); if (result != TurtleError.OK) { HandleError(result); return; } ShowPosition(arthurTheTurtle); // Then rotate back the other way arthurTheTurtle.RightMotorState = MotorState.Running; // And run for PI/4 seconds to give us 45 degrees result = arthurTheTurtle.RunFor(Math.PI / 4.0); if (result != TurtleError.OK) { HandleError(result); return; } ShowPosition(arthurTheTurtle); // And finally drive backwards for a bit arthurTheTurtle.RightMotorState = MotorState.Reversed; arthurTheTurtle.LeftMotorState = MotorState.Reversed; result = arthurTheTurtle.RunFor(Math.Cos(Math.PI / 4.0)); if (result != TurtleError.OK) { HandleError(result); return; } ShowPosition(arthurTheTurtle); Console.ReadKey(); } private static void HandleError(TurtleError result) Returning Error Values | 199 { Console.WriteLine("We hit turtle error {0}", result); Console.ReadKey(); } Every time we call the RunFor method, we have to stash away the error message that is returned, check it for problems, and then decide what we’re going to do. In this instance, we decided to quit the application, after showing an error message to the user, because it isn’t safe to continue. If we compile and run, here’s the output: Arthur is at (0,0) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 0.00 radians. We hit turtle error RotateError From an application point of view, this is much better behavior than before: we were able to stop when we hit our first problem. Unfortunately, we had to write quite a lot of boilerplate code to achieve that end, and our code is much less readable than it was before. We also created a huge number of potential exit points out of our function, which decreases its maintainability. So while it is better, I’m not totally happy with it; this is catching just one potential error from one function, and we have almost as many lines of code dealing with that as we do our success scenario, scattered throughout our whole program! Debugging with Return Values So we finally spotted the problem, and stopped it from causing trouble. How do we find out what is wrong? Well, first we should take a look at the error message. That tells us that it has something to do with rotating the turtle, which gives us a bit of a clue. The easiest way to see what is really going on, though, might be to set a breakpoint in our error handler and see what state the system is in when the error occurs. To set a breakpoint, we can put the cursor on the line where we want to break into the debugger, and press F9. Figure 6-1 shows the code with a breakpoint set. Figure 6-1. Code with a breakpoint set If we run this now, the application will break into the debugger when we hit our error handler. If we press Ctrl-Alt-C, we can inspect the call stack to see where we went wrong, as shown in Figure 6-2. 200 | Chapter 6: Dealing with Errors As you can see, there’s not an awful lot to help us; we lost context in which the error occurred because we returned out of the method that had the actual problem, and wound back up to our calling function. It isn’t completely useless—we now know which call had the problem (this time), so we can put a breakpoint on the relevant line and run again; but what if this was a hard- to-reproduce, intermittent error? We may have lost our one chance this week to identify and fix the problem! These are not the only problems with a return-value-based approach to error handling. What if we already need to use the return value on the method? We’re heading into the realm of “magic” values that mean an error has occurred, or we could add out or ref parameters to allow our method to return both a useful output and an error code. And what about property setters; we don’t have the option of a return value, but we might well like to return an error of some kind if the value is out of range. If you’re thinking “surely there has to be a better way,” you’re right. C# (like most modern languages) supports an alternative means of signaling errors: exceptions. Exceptions Rather than return an error code from a method, we can instead throw an instance of any type derived from Exception. Let’s rewrite our Rotate method to do that (see Ex- ample 6-11). Example 6-11. Indicating an error with an exception private void Rotate(double duration) { if (PlatformWidth <= 0.0) { throw new InvalidOperationException( "The PlatformWidth must be initialized to a value > 0.0"); } // This is the total circumference of turning circle double circum = Math.PI * PlatformWidth; // This is the total distance traveled double d = duration * MotorSpeed; if (LeftMotorState == MotorState.Reversed) Figure 6-2. Call stack, broken in the error handler Exceptions | 201 { // And we're going backwards if the motors are reversed d *= −1.0; } // So we've driven it this proportion of the way round double proportionOfWholeCircle = d / circum; // Once round is 360 degrees (or 2pi radians), so we have traveled CurrentOrientation = CurrentOrientation + (Math.PI * 2.0 * proportionOfWholeCircle); // return true; (This is now redunant, so you can delete it) } Notice that we changed the return specification back to void, and removed the unnec- essary return at the end. The interesting bit, though, is in our test at the beginning of the method. Pre- and Post Conditions: Design by Contract The quick tests at the beginning of the method are sometimes called “guard clauses” or “guards.” Unless performance is more important to your application than correct operation (and it usually isn’t), it is a good idea to check these preconditions before you attempt to execute the method. Sometimes you will also want a similar set of post-condition tests on exit from the method, to verify that your state is still valid at the end of the operation. The design-by-contract development philosophy requires you to specify these pre- and post conditions as a part of your method contract, and some languages such as Eiffel support declarative specification of these conditions. Microsoft Research is working on an extension of C# called Spec# which includes some of these design-by-contract features. You can read about it at http://research.mi crosoft.com/en-us/projects/specsharp/. Rather than return an instance of an enum, we throw an instance of the InvalidOpera tionException class. InvalidOperationException is one of several types derived from Exception. It is intended to be used when an operation fails because the current state of the object itself doesn’t allow the method to succeed (rather than, say, because a parameter passed in to the method was incorrect). That seems to fit this case quite nicely, so we can make use of it. Back before C# 3.0, you could throw an instance of any type you liked (e.g., a string). In C# 3.0, a constraint was added that only types derived from Exception can be thrown. 202 | Chapter 6: Dealing with Errors If we take a look at the Exception class (see http://msdn.microsoft.com/library/system .exception) we’ll see that it has a Message property. That’s what we’re setting with the string we pass to the constructor, and it can be any text we like—preferably something that will help us (or one of our clients) debug the problem in the future. There’s also a property called Data. This is a dictionary of key/value pairs that lets us associate more information with the exception, and it can be extremely useful for de- bugging or logging purposes. Replacing the return value with an exception, we will need to perform a bit of surgery on our application to get it to compile. First, let’s change the Turtle.RunFor method so that it no longer returns a value, and delete the TurtleError enumeration (see Example 6-12). Example 6-12. Passing the buck is no longer required // Run the turtle for the specified duration public void RunFor(double duration) { if (LeftMotorState == MotorState.Stopped && RightMotorState == MotorState.Stopped) { // If we are at a full stop, nothing will happen return; } // The motors are both running in the same direction // then we just drive if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Running) || (LeftMotorState == MotorState.Reversed && RightMotorState == MotorState.Reversed)) { Drive(duration); } // The motors are running in opposite directions, // so we don't move, we just rotate about the // center of the rig if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Reversed) || (LeftMotorState == MotorState.Reversed && RightMotorState == MotorState.Running)) { Rotate(duration); } } Then, we can update the calling program, and strip out the code that deals with the return value (see Example 6-13). Exceptions | 203 Example 6-13. Main no longer checking explicitly for errors static void Main(string[] args) { Turtle arthurTheTurtle = new Turtle { PlatformWidth = 0.0, PlatformHeight = 10.0, MotorSpeed = 5.0 }; ShowPosition(arthurTheTurtle); // We want to proceed forwards arthurTheTurtle.LeftMotorState = MotorState.Running; arthurTheTurtle.RightMotorState = MotorState.Running; // For two seconds arthurTheTurtle.RunFor(2.0); ShowPosition(arthurTheTurtle); // Now, let's rotate clockwise for a bit arthurTheTurtle.RightMotorState = MotorState.Reversed; // PI / 2 seconds should do the trick arthurTheTurtle.RunFor(Math.PI / 2.0); ShowPosition(arthurTheTurtle); // And let's go into reverse arthurTheTurtle.RightMotorState = MotorState.Reversed; arthurTheTurtle.LeftMotorState = MotorState.Reversed; // And run for 5 seconds arthurTheTurtle.RunFor(5); ShowPosition(arthurTheTurtle); // Then rotate back the other way arthurTheTurtle.RightMotorState = MotorState.Running; // And run for PI/4 seconds to give us 45 degrees arthurTheTurtle.RunFor(Math.PI / 4.0); ShowPosition(arthurTheTurtle); // And finally drive backwards for a bit arthurTheTurtle.RightMotorState = MotorState.Reversed; arthurTheTurtle.LeftMotorState = MotorState.Reversed; arthurTheTurtle.RunFor(Math.Cos(Math.PI / 4.0)); ShowPosition(arthurTheTurtle); Console.ReadKey(); } Finally, we can delete the HandleError method. 204 | Chapter 6: Dealing with Errors OK, what happens if you compile and run (make sure you press F5 or choose Debug→Start Debugging so that you run in the debugger)? Well, you drop very rapidly into the debugger, as you can see in Figure 6-3. Figure 6-3. An unhandled exception in the debugger As the debugger implies, we’ve broken in here because the exception is unhandled; but notice that we’ve broken right at the point at which the exception actually occurred. Even if we hadn’t provided some nice descriptive error text, we can clearly see why we failed, unlike with error codes, where by the time the debugger got involved, we had already lost track of the root cause of the failure. If we want an even closer look, we can click the View Detail link on the callout. This produces a dialog containing a property grid view of the exception object that was thrown. We can examine this to help us debug the problem. (You can see the Message and Data properties that we previously looked at, and I’ve popped open the StackTrace for the exception, in the example in Figure 6-4.) That’s already a huge improvement over the return value approach; but are there any obvious downsides to throwing an exception? Exceptions | 205 Figure 6-4. Exception detail with stack trace open Well, the first downside is that throwing an exception is way more expensive than simply returning a value. You don’t really want to throw an exception just to manage your normal flow of control. Passing parameters and looking at internal state is always going to be a better choice for anything you might call “the success path.” That being said, expense is, of course, relative, and as usual, you should use the best tool for the job. Plus, exceptions are actually lower cost than return values if you don’t actually throw them. In our previous example, we allocated and copied return values even if everything was OK. With the exception model, the success path is basically free. The debate about when to use exceptions versus return values continues to rage in our industry. I don’t expect it to let up anytime soon. As I said at the beginning of the chapter, it is almost like we’re not really on top of the whole error-handling situation. We’ve seen what happens if we don’t handle an exception that we throw (i.e., it per- colates up until eventually we crash); and while that behavior is far more satisfactory than the situation when we ignored a return value, we would like to do much better. We want to handle the error and exit gracefully, as we did before. 206 | Chapter 6: Dealing with Errors As you might expect, C# provides us with language features to do just that: try, finally, and catch. Handling Exceptions When we handled our return values, we had to propagate them up the call stack by hand, adding appropriate return values to each and every method, checking the result, and either passing it up or transforming and passing it as we go. Exceptions, on the other hand, propagate up the stack automatically. If we don’t want to add a handler, we don’t have to, and the next call site up gets its chance instead, until eventually we pop out at the top of Main and either break into the debugger or Windows Error Handling steps in. This means we can take a more structured approach to error handling—identifying points in our application control flow where we want to handle particular types of exceptions, and gathering our error-handling code into easily identified blocks. The try, catch, and finally keywords help us to define those blocks (along with the ubiquitous braces). In our example, we have no need to handle the potential errors from each and every call to RunFor separately. Instead, we can wrap the whole set into a single set of try, catch, and finally blocks, as shown in Example 6-14. Example 6-14. Handling exceptions static void Main(string[] args) { Turtle arthurTheTurtle = new Turtle { PlatformWidth = 0.0, PlatformHeight = 10.0, MotorSpeed = 5.0 }; ShowPosition(arthurTheTurtle); try { // We want to proceed forwards arthurTheTurtle.LeftMotorState = MotorState.Running; arthurTheTurtle.RightMotorState = MotorState.Running; // For two seconds arthurTheTurtle.RunFor(2.0); ShowPosition(arthurTheTurtle); // Now, let's rotate clockwise for a bit arthurTheTurtle.RightMotorState = MotorState.Reversed; // PI / 2 seconds should do the trick arthurTheTurtle.RunFor(Math.PI / 2.0); ShowPosition(arthurTheTurtle); Exceptions | 207 // And let's go into reverse arthurTheTurtle.RightMotorState = MotorState.Reversed; arthurTheTurtle.LeftMotorState = MotorState.Reversed; // And run for 5 seconds arthurTheTurtle.RunFor(5); ShowPosition(arthurTheTurtle); // Then rotate back the other way arthurTheTurtle.RightMotorState = MotorState.Running; // And run for PI/4 seconds to give us 45 degrees arthurTheTurtle.RunFor(Math.PI / 4.0); ShowPosition(arthurTheTurtle); // And finally drive backwards for a bit arthurTheTurtle.RightMotorState = MotorState.Reversed; arthurTheTurtle.LeftMotorState = MotorState.Reversed; arthurTheTurtle.RunFor(Math.Cos(Math.PI / 4.0)); ShowPosition(arthurTheTurtle); } catch (InvalidOperationException e) { Console.WriteLine("Error running turtle:"); Console.WriteLine(e.Message); } finally { Console.ReadKey(); } } If any of the code in the try block throws an exception, the runtime looks to see if there are any catch blocks whose exception type matches the type of that exception. It matches successfully if the catch parameter is either of the same type, or of a less-derived (base) type than that of the exception. You can have any number of catch blocks for different types of exceptions, and it will look through them in the order they are defined; the first one that matches wins (even if there is a “better” match farther down). If it doesn’t find a suitable match, the exception will be propagated on up the call stack, just as though there was no try block. To see how this works in practice, let’s quickly modify the code in Example 6-14 to catch Exception as well, as shown in Example 6-15. 208 | Chapter 6: Dealing with Errors Example 6-15. Poorly placed catch block try { ... } catch (Exception e2) { Console.WriteLine("Caught generic exception..."); } catch (InvalidOperationException e) { Console.WriteLine("Error running turtle:"); Console.WriteLine(e.Message); } finally { Console.WriteLine("Waiting in the finally block..."); Console.ReadKey(); } If you try to compile this, you’ll see the following error: A previous catch clause already catches all exceptions of this or of a super type ('System.Exception') This occurs because Exception is an ancestor of InvalidOperationException, and the clause appears first in the list of catch blocks. If we switch those around, we compile successfully, as shown in Example 6-16. Example 6-16. Catching exceptions in the right order try { ... } catch (InvalidOperationException e) { Console.WriteLine("Error running turtle:"); Console.WriteLine(e.Message); } catch (Exception e2) { Console.WriteLine("Caught generic exception..."); } finally { Console.WriteLine("Waiting in the finally block..."); Console.ReadKey(); } Exceptions | 209 Catching Too Much You should consider very carefully whether you want to catch instances of the base Exception type. When you do that you are saying “something bad happened, but I don’t really know what it was.” That degree of uncertainty tends to imply that the app has lost control of its own internal consistency. It is a common practice to catch Exception in your top-level exception handlers (e.g., in Main or, as we’ll see later, in threading worker functions); and when you do, you normally need to terminate the application (or at least restart some subsystem). Of course, you might know perfectly well what might go wrong, and you’re catching Exception because you can’t be bothered to list the half dozen exception types you intend to handle in the same way. (Maybe F1 wasn’t working so well that day and you couldn’t inspect the docs; and someone was pressing you to check in your changes.) Beware! What happens if an implementation detail changes in another class on which you de- pend, such that a new exception is thrown? Your handler will swallow it up and carry on. That might be OK, and your game of tic-tac-toe will continue happily. Or it might have unintended consequences, such as data loss or the start of WWIII. You just can’t know in advance. When the flow of control leaves the try block successfully, or the flow of control exits the last catch block if an exception occurred in the try block, the code in the finally block is executed. In other words, the code in the finally block is always executed, regardless of whether there was an exception. If you designed your exception-handling code nicely, you’ll almost certainly use far more finally blocks than you do catch blocks. The finally block is a good place for cleaning up your resources, or winding back internal state if an error occurs, to ensure that your pre- and post conditions are still valid, whereas a catch block allows you to deal with an error condition you, as a client, understand in some way—even if it is only to present a message to the user (as in this case). If we compile and run our code again, we’ll see the following output: Arthur is at (0,0) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 0.00 radians. Error running turtle: The PlatformWidth must be initialized to a value > 0.0 Waiting in the finally block... Notice how the error-handling code is now consolidated neatly into clearly defined blocks, rather than scattered throughout our code, and we’ve been able to cut down substantially on the number of points of return from our method. 210 | Chapter 6: Dealing with Errors At the moment, we’re not handling any exceptions in our Turtle itself. Let’s imagine that our Turtle is being provided to clients in a library, and we (as the leading vendors of turtle simulators) want the library to do some internal logging when errors occur: maybe we have an opt-in customer experience program that sends telemetry back to our team. We still want the errors to propagate up to the client for them to deal with; we just want to see them on the way past. C# gives us the ability to catch, and then transparently rethrow, an exception, as shown in Example 6-17. Example 6-17. Rethrowing an exception // Run the turtle for the specified duration public void RunFor(double duration) { try { if (LeftMotorState == MotorState.Stopped && RightMotorState == MotorState.Stopped) { // If we are at a full stop, nothing will happen return; } // The motors are both running in the same direction // then we just drive if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Running) || (LeftMotorState == MotorState.Reversed && RightMotorState == MotorState.Reversed)) { Drive(duration); } // The motors are running in opposite directions, // so we don't move, we just rotate about the // center of the rig if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Reversed) || (LeftMotorState == MotorState.Reversed && RightMotorState == MotorState.Running)) { Rotate(duration); } } catch (Exception ex) { Console.WriteLine("Log message: " + ex.Message); // Rethrow throw; Exceptions | 211 } } The first thing to notice is that we caught the base Exception type, having just said that we almost never do that. We want to log every exception, and because we’re rethrowing rather than eating it, we won’t simply ignore exceptions we weren’t expecting. After we execute our handler code (just writing the message to the console in this case), we use the throw keyword, without any object, to rethrow the exception we just caught. If you compile and run that, you’ll see the following output: Arthur is at (0,0) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 0.00 radians. Log error: The PlatformWidth must be initialized to a value > 0.0 Error running turtle: The PlatformWidth must be initialized to a value > 0.0 Waiting in the finally block Notice that we get the output from both of the exception handlers. That’s not the only way to throw from a catch block: it is perfectly reasonable to throw any exception from our exception handler! We often do this to wrap up an exception that comes from our implementation in another exception type that is more appropriate for our context. The original exception is not thrown away, but stashed away in the InnerException property of the new one, as shown in Example 6-18. Example 6-18. Wrapping one exception in another // Run the turtle for the specified duration public void RunFor(double duration) { try { if (LeftMotorState == MotorState.Stopped && RightMotorState == MotorState.Stopped) { // If we are at a full stop, nothing will happen return; } // The motors are both running in the same direction // then we just drive if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Running) || (LeftMotorState == MotorState.Reversed && RightMotorState == MotorState.Reversed)) { Drive(duration); } // The motors are running in opposite directions, // so we don't move, we just rotate about the // center of the rig 212 | Chapter 6: Dealing with Errors if ((LeftMotorState == MotorState.Running && RightMotorState == MotorState.Reversed) || (LeftMotorState == MotorState.Reversed && RightMotorState == MotorState.Running)) { Rotate(duration); } } catch (InvalidOperationException iox) { throw new Exception("Some problem with the turtle...", iox); } catch (Exception ex) { // Log here Console.WriteLine("Log message: " + ex.Message); // Rethrow throw; } } Notice how we passed the exception to be wrapped as a parameter to the new exception when we constructed it. Let’s make a quick modification to the exception handler in Main to take advantage of this new feature (see Example 6-19). Example 6-19. Reporting an InnerException static void Main(string[] args) { Turtle arthurTheTurtle = new Turtle { PlatformWidth = 0.0, PlatformHeight = 10.0, MotorSpeed = 5.0 }; ShowPosition(arthurTheTurtle); try { // ... } catch (InvalidOperationException e) { Console.WriteLine("Error running turtle:"); Console.WriteLine(e.Message); } catch (Exception e1) { // Loop through the inner exceptions, printing their messages Exception current = e1; while (current != null) { Console.WriteLine(current.Message); current = current.InnerException; } } finally Exceptions | 213 { Console.WriteLine("Waiting in the finally block"); Console.ReadKey(); } } If we compile and run again, we can see the following output, including the messages from both the outer and inner exceptions: Arthur is at (0,0) and is pointing at angle 0.00 radians. Arthur is at (0,10) and is pointing at angle 0.00 radians. Some problem with the turtle has occurred The PlatformWidth must be initialized to a value > 0.0 Waiting in the finally block Clearly, wrapping an implementation-detail exception with something explicitly docu- mented in our public contract can simplify the range of exception handlers you require. It also helps to encapsulate implementation details, as the exceptions you throw can be considered part of your contract. On the other hand, are there any disadvantages to throwing a wrapped exception (or indeed rethrowing the original exception explicitly, rather than implicitly with throw;)? As programming tends to be a series of compromises, the answer is, as you might expect, yes. If you explicitly (re)throw an exception, the call stack in the exception handler starts at the new throw statement, losing the original context in the debugger (although you can still inspect it in the inner exception in the object browser). This makes debugging noticeably less productive. Because of this, you should consider carefully whether you need to wrap the exception, and always ensure that you implicitly (rather than explicitly) rethrow exceptions that you have caught and then wish to pass through. When Do finally Blocks Run? It is worth clarifying exactly when the finally block gets executed, under a few edge conditions. First, let’s see what happens if we run our example application outside the debugger. If we do that (by pressing Ctrl-F5) we’ll see that Windows Error Handling* materializes, and presents the user with an error dialog before we actually hit our finally block at all! It is like the runtime has inserted an extra catch block in our own (top-level) ex- ception handler, rather than percolating up another level (and hence out of our scope, invoking the code in the finally block). And what happens when exceptions are thrown out of the exception handlers? * Or “Dr. Watson” as the crash handler was more colorfully named on older versions of Windows. 214 | Chapter 6: Dealing with Errors Let’s add a finally block to our RunFor method (see Example 6-20). Example 6-20. Seeing when finally blocks run // Run the turtle for the specified duration public void RunFor(double duration) { try { // ... } catch (InvalidOperationException iox) { throw new Exception("Some problem with the turtle has occurred", iox); } catch (Exception ex) { // Log here Console.WriteLine("Log error: " + ex.Message); // Rethrow throw; } finally { Console.WriteLine("In the Turtle finally block"); } } If you compile and run this code, you’ll see the following output: Arthur is at (0,0) and is pointing at angle 0.00 radians. In the Turtle finally block Arthur is at (0,10) and is pointing at angle 0.00 radians. In the Turtle finally block Some problem with the turtle has occurred The PlatformWidth must be initialized to a value > 0.0 Waiting in the finally block So our finally block executes after the exception is thrown, but before it executes the exception handlers farther up the stack. Deciding What to Catch One important question remains: how did we know what exception type to catch from our code? Unlike some other languages (e.g., Java) there is no keyword which allows us to specify that a method can throw a particular exception. We have to rely on good developer documentation. The MSDN documentation for the framework itself care- fully documents all the exceptions that can be thrown from its methods (and proper- ties), and we should endeavor to do the same. The .NET Framework provides a wide variety of exception types that you can catch (and often use). Let’s revisit Table 6-1 (the common error types) and see what is avail- able for those situations (see Table 6-2). Exceptions | 215 Table 6-2. Some common errors and their exception types Error Description Examples Unexpected input A client passes data to a method that is outside some expected range. ArgumentException ArgumentNullException ArgumentOutOfRangeException Unexpected data type A client passes data to a method that is not of the expected type. InvalidCastException Unexpected data format A client passes data to a method in a format that is not recognized. FormatException Unexpected result A client receives information from a method that it did not expect for the given input (e.g., null). NullReferenceException Unexpected method call The class wasn’t expecting you to call a particular method at that time; you hadn’t performed some required initiali- zation, for example. InvalidOperationException Unavailable resource A method tried to access a resource of some kind and it failed to respond in a timely fashion; a hardware device was not plugged in, for instance. TimeoutException Contended resource A method tried to access a scarce resource of some kind (memory or a hardware device that cannot be shared) and it was not available because someone else was using it. OutOfMemoryException TimeoutException Obviously, that’s a much abbreviated list, but it contains some of the most common exceptions you’ll see in real applications. One of the most useful that you’ll throw yourself is the ArgumentException. You can use that when parameters passed to your methods fail to validate. Let’s make use of that in our RunFor method. Say that a “feature” of our turtle hardware is that it crashes and becomes unresponsive if we try to run it for zero seconds. We can work around this in our software by checking for this condition in the RunFor method, and throwing an exception if clients try this, as shown in Example 6-21. Example 6-21. Throwing an exception when arguments are bad public void RunFor(double duration) { if (duration <= double.Epsilon) { throw new ArgumentException( "Must provide a duration greater than 0", "duration"); } try { // ... } catch (InvalidOperationException iox) 216 | Chapter 6: Dealing with Errors { throw new Exception("Some problem with the turtle has occurred", iox); } catch (Exception ex) { // Log here Console.WriteLine("Log error: " + ex.Message); // Rethrow throw; } finally { Console.WriteLine("In the Turtle finally block"); } } The second parameter in this constructor should match the name of the parameter that is in error. The first represents the exception message. When you come to use ArgumentNullException (which you throw when you are erroneously passed a null argument) you’ll find that the error message and parameter arguments are swapped around in the construc- tor. This irritating inconsistency has been with us since .NET 1.0, and too much code depends on it to fix it now. The code in Example 6-22 updates Main, to sneak in an attempt to run it for zero seconds. Example 6-22. Testing for the expected exception static void Main(string[] args) { Turtle arthurTheTurtle = new Turtle { PlatformWidth = 0.0, PlatformHeight = 10.0, MotorSpeed = 5.0 }; ShowPosition(arthurTheTurtle); try { arthurTheTurtle.RunFor(0.0); // ... } catch (InvalidOperationException e) { Console.WriteLine("Error running turtle:"); Console.WriteLine(e.Message); } catch (Exception e1) { // Loop through the inner exceptions, printing their messages Exception current = e1; while (current != null) Exceptions | 217 { Console.WriteLine(current.Message); current = current.InnerException; } } finally { Console.WriteLine("Waiting in the finally block"); Console.ReadKey(); } } If we compile and run, we’ll see the following output: Arthur is at (0,0) and is pointing at angle 0.00 radians. Must provide a duration greater than 0 Parameter name: duration Waiting in the finally block Notice how the error message automatically includes the details of the problem parameter. Custom Exceptions You might want to create your own exceptions for a couple of reasons: • My exception is a special snowflake. • I want to group my exceptions together for layered exception handling. The first of these is the most problematic. You should think very carefully about whether your exception is really special, or whether you can just reuse an existing exception type. When you introduce new exceptions, you’re asking clients to understand and deal with a new type of problem, and you’re expecting them to handle it in a special way. There are occasional instances of this, but more often the differences are in the context (i.e., that it was thrown from your code) rather than the exception itself (i.e., something was out of range, invalid, null, or unavailable, or it timed out). Slightly more often, you provide custom exception types when you want to provide a convenient API over some additional information that comes along with the exception. The Exception.Data property we discussed earlier might be a better solution—it gives you somewhere to put information without needing to add a new kind of exception. But the convenience of a dedicated property might outweigh the costs of introducing a custom exception. Finally, you might wish to create a custom exception class to allow you to conceptually group some subsystem’s exceptions together. DbException is an example of this in the .NET Framework; it represents the various errors that can occur when using a database. There are various specialized errors that derive from this, such as the SqlException thrown by the SQL Server subsystem, but the common base class 218 | Chapter 6: Dealing with Errors enables you to write a single catch for all database errors, rather than having to handle provider-specific errors. Again, you should think carefully about this before doing it: what client exception- handling scenarios are you enabling, and why do you need the custom type? However, having been through all of this, creating your own exception type is very simple. Let’s create a TurtleException for our exception wrapper (see Example 6-23). Whether we really want a TurtleException is another matter. I’m not sure I really would in these circumstances, but your mileage may vary. Example 6-23. A custom exception [Serializable] class TurtleException : Exception { public TurtleException() {} public TurtleException(string message) : base(message) { } public TurtleException(string message, Exception innerException) : base(message, innerException) {} // For serialization support protected TurtleException(SerializationInfo info, StreamingContext context) : base(info, context) {} } The first thing to notice is that we derive from Exception. If you’ve plowed through the MSDN documentation you might have noticed the ApplicationException type, which derives from Exception, and was provided as a base class for application-defined exceptions. Why, you might ask, are we not deriving from ApplicationException? Well, ApplicationException adds no functionality to Exception, and the .NET designers could not come up with a scenario where it was useful to catch ApplicationException (as opposed to Exception). Sadly, they only realized this after .NET 1.0 had shipped, so it is in the library, but it is now deprecated. You should neither derive from nor catch ApplicationException. Also, we provide a bunch of standard constructors: a default parameterless constructor, one that takes a message, and one that takes a message and an inner exception. Even Exceptions | 219 if you add more properties to your own exception that you wish to initialize in the constructor, you should still provide constructors that follow this pattern (including the default, parameterless one). The final constructor supports serialization. We do this because Exception itself is marked as a serializable class, which means that derived classes have to be too. This enables exceptions to cross appdomain boundaries.† We’re just calling the base class’s constructor here. Because there is no constructor inheritance in C#, we need to provide a matching constructor which calls the one in our base. If we didn’t do this, any code that polymorphically used our TurtleException as its base Exception might break. Summary In this chapter, we reviewed the various types of errors that might occur in our software and looked at several strategies for handling them. These include ignoring the problem, aborting the application, returning errors, and throwing exceptions. We also saw some of the benefits and pitfalls of returning errors, and how exceptions can often provide a more robust and flexible means of alerting your clients to problems. We saw how we can handle exceptions in layers, sometimes catching, using, and then rethrowing an exception, sometimes wrapping an implementation exception in a public exception type, and sometimes allowing exceptions to bubble up to the next layer of handlers. We saw what happens when an unhandled exception pops out at the top of the stack, and how we can use finally blocks at each layer to ensure that application state remains consistent, and resources can be released, whether exceptions occur or not. We then took a quick review of some of the most common exceptions provided by the frame- work, and how we might use them. Finally, we looked at creating our own exception types and why we might (and might not) wish to do so. We’ve come a long way in the past few chapters, covering all of the everyday C# pro- gramming concepts you’ll need. In the next few chapters, we’ll look at features of the .NET Framework in more detail, and how we can best use them in C#; starting with the collection classes. † An appdomain is a kind of process within a process. We’ll talk about them a little more in Chapter 11 and Chapter 15, but they’re mainly used by systems that need to host code, such as ASP.NET. 220 | Chapter 6: Dealing with Errors CHAPTER 7 Arrays and Lists Most programs have to deal with multiple pieces of information. Payroll systems need to calculate the salary of every employee in a company; a space battle game has to track the position of all the ships and missiles; a social networking website needs to be able to show all of the user’s acquaintances. Dealing with large numbers of items is a task at which computers excel, so it’s no surprise that C# has a range of features dedicated to working with collections of information. Sets of information crop up so often that we’ve already seen some of what C# has to offer here. So we’ll start with a more detailed look at the collection-based features we’ve already seen, and in the next chapter we’ll look at the powerful LINQ (Language In- tegrated Query) feature that C# offers for finding and processing information in po- tentially large sets of information. Arrays The ability to work with collections is so important that the .NET Framework’s type system has a feature just for this purpose: the array. This is a special kind of object that can hold multiple items, without needing to declare a field for each individual item. Example 7-1 creates an array of strings, with one entry for each event coming up in one of the authors’ calendars over the next few days. You may notice a theme here (although one misfit appears to be a refugee from an earlier chapter’s theme, but that’s just how the author’s weekend panned out; real data is never tidy). Example 7-1. An array of strings string[] eventNames = { "Swing Dancing at the South Bank", "Saturday Night Swing", "Formula 1 German Grand Prix", "Swing Dance Picnic", "Stompin' at the 100 Club" }; 221 Look at the variable declaration on the first line. The square brackets after string in- dicate that eventNames is not just a single string; it’s an array of string values. These square brackets tie in with the syntax for accessing individual elements in the array. Example 7-2 prints the first and fifth items in the array. (So this will print out the text “Swing Dancing at the South Bank”, followed by “Stompin’ at the 100 Club”.) Example 7-2. Using elements in an array Console.WriteLine(eventNames[0]); Console.WriteLine(eventNames[4]); The number inside the square brackets is called the index, and as you can see, C# starts counting array elements from zero. As you may recall from Chapter 2, the index says how far into the array we’d like C# to look—to access the very first element, we don’t have to go any distance into the array at all, so its index is 0. Likewise, an index of 4 jumps past the first four items to arrive at the fifth. To modify an array element you just put the same syntax on the lefthand side of an assignment. For example, noticing that I got one of the event names slightly wrong, I can update it, like so: eventNames[1] = "Saturday Night Swing Club"; While you can change any element of an array like this, the number of elements is fixed for the lifetime of the array. (As we’ll see later, this limitation is less drastic than it first sounds.) If you try to use too high an index or a negative index when accessing an array element, the code will throw an IndexOutOfRangeException. Since elements are numbered from zero, the highest acceptable index is one less than the number of elements. For example, eventNames[4] is the last item in our five-item array, so trying to read or write eventNames[5] would throw an exception. The .NET Framework supports arrays whose first element is numbered from something other than zero. This is to support languages such as Visual Basic that have historically offered such a construct. However, you cannot use such arrays with the C# index syntax shown here—you would need to use the Array class’s GetValue and SetValue helper meth- ods to use such an array. Since the size of an array is fixed at the moment it is constructed, let’s look at the construction process in more detail. Construction and Initialization There are two ways you can create a new array in C#. Example 7-1 showed the most straightforward approach—the array variable declaration was followed by a list of the 222 | Chapter 7: Arrays and Lists array’s contents enclosed in braces, which is called an initializer list. But this requires you to know exactly what you want in the array when you write the code. You will often work with information that your program discovers at runtime, perhaps from a database or a web service. So C# offers an alternative mechanism that lets you choose the array’s size at runtime. For example, suppose I decide I’d like to display the events in my calendar as a num- bered list. I already have an array of event names, but I’d like to build a new string array that adds a number to the event text. Example 7-3 shows how to do this. Example 7-3. Creating an array dynamically static string[] AddNumbers(string[] names) { string[] numberedNames = new string[names.Length]; for (int i = 0; i < names.Length ; ++i) { numberedNames[i] = string.Format("{0}: {1}", i, names[i]); } return numberedNames; } This AddNumbers method doesn’t know up front what will be in the array it creates— it’s building a modified copy of an existing array. So instead of creating a fixed list of items, it uses this syntax: new ElementType[arrayLength]. This specifies the two things that are fixed when you create a new array: the type and number of elements. When you create an array with this minimal syntax, the elements all start out with their default values. With the string type used here, the default is null; an array of numbers created this way would contain all zeros. So Example 7-3 immediately goes on to pop- ulate the newly created array with some useful values. In fact, that’s also what happened in Example 7-1—when you provide an array with a list of initial contents, the C# compiler turns it into the sort of code shown in Example 7-4. Example 7-4. How initializer lists work string[] eventNames = new string[5]; eventNames[0] = "Swing Dancing at the South Bank"; eventNames[1] = "Saturday Night Swing"; eventNames[2] = "Formula 1 German Grand Prix"; eventNames[3] = "Swing Dance Picnic"; eventNames[4] = "Stompin' at the 100 Club"; The array initialization syntax in Example 7-1 is really just convenient shorthand— the .NET Framework itself always expects to be told the number and type of elements, so it’s just a matter of whether the C# compiler works out the element count for you. The Example 7-1 shorthand works only at the point at which you declare a variable. If your program decides to put a new array into an existing variable, you’ll find that the syntax no longer works (see Example 7-5). Arrays | 223 Example 7-5. Where array initializers fail // Won't compile! eventNames = { "Dean Collins Shim Sham Lesson", "Intermediate Lindy Hop Lesson", "Wild Times - Social Dancing at Wild Court" }; The reasons for this are somewhat arcane. In general, C# cannot always work out what element type you need for an array, because it may have more than one choice. For example, a list of strings doesn’t necessarily have to live in an array of type string[]. An array of type object[] is equally capable of holding the same data. And as we’ll see later, initializer lists don’t necessarily have to initialize arrays—this list of strings could initialize a List, for example. As it happens, only one of those choices would work in Example 7-5—we’re assigning into the eventNames variable, which is of type string[], so you’d think the compiler would know what we want. But since there are some situations which really are am- biguous, Microsoft decided to require you to specify the element type everywhere ex- cept for the one special case of initializing a newly declared array variable. The upshot is not so bad—if you specify the element type, you still get to use the initializer list syntax and have C# count the elements for you. Example 7-6 modifies Example 7-5 by explicitly stating the type of array we’d like before providing its con- tents. By the way, we could also have added this explicit array type in Example 7-1— it would have worked, it’s just more verbose than necessary in that particular case. Example 7-6. Combining an explicit element type with an initializer list // Will compile! eventNames = new string[] { "Dean Collins Shim Sham Lesson", "Intermediate Lindy Hop Lesson", "Wild Times - Social Dancing at Wild Court" }; This syntax works anywhere you need an array-typed expression. For example, we could use this to pass in an array to the AddNumbers method in Example 7-3, as shown in Example 7-7. Example 7-7. Inline array initializer string[] result = AddNumbers(new string[] { "The Jazz Devil", "Jitterbugs" }); This inline array technique can occasionally be useful if you need to call a method that demands to be passed an array, and you happen not to have one handy. The String class’s Split method illustrates an interesting twist on this. 224 | Chapter 7: Arrays and Lists Array arguments and the params keyword The String.Split method breaks a string into multiple strings based on separator characters. You tell it which characters to treat as separators by passing a char array. Example 7-8 splits on spaces, commas, and periods. Example 7-8. Array arguments in the class library string[] items = inputString.Split( new char[] { ' ', ',', '.' }, StringSplitOptions.RemoveEmptyEntries); If inputString contained "One,Two Three, Four. Five.", this would put a five-element array into items containing the strings "One", "Two", "Three", "Four", and "Five". Example 7-8 asks Split to ignore empty items so that when we get both a period and a space in succession we don’t get an empty string in the results to represent the fact that there were two separators. If you don’t need to skip such things, there’s a simpler overload of Split that illustrates yet another way to initialize an array: string[] items = inputString.Split(' ', ',', '.'); It looks like we’ve passed three char arguments to this method. But there’s no such overload of Split—this ends up calling an overload that looks like this: public string[] Split(params char[] separator) ... That params keyword is significant. When an argument is marked with this keyword, C# lets you use syntax that makes it look like a series of individual arguments, and it will create an array from these for you. (You’re free to provide the array explicitly if you prefer.) The params keyword can be used on only the very last argument of a method, to avoid potential ambiguity about which values go into arrays and which become arguments in their own right. That’s why Example 7-8 had to create the array explicitly. The examples so far contain nothing but strings. This is a poor way to represent events in a calendar—it would be useful to know when each event occurs. We could add a second array of type DateTimeOffset[] whose elements correspond to the event names in the original array. But spreading related data across multiple arrays can make code awkward to write and hard to maintain. Fortunately, there’s a better way. Custom Types in Arrays You can create an array using any type for the element type—you’re not limited to types provided by the .NET Framework class library. You can use a class defined in the way shown in Chapter 3, such as the calendar event type in Example 7-9. Example 7-9. Custom class to represent events in a calendar class CalendarEvent { public string Title { get; set; } Arrays | 225 public DateTimeOffset StartTime { get; set; } public TimeSpan Duration { get; set; } } This class holds the event’s title, start time, and duration in a single object. We can create an array of these objects, as shown in Example 7-10. Example 7-10. Creating an array with a custom element type CalendarEvent[] events = { new CalendarEvent { Title = "Swing Dancing at the South Bank", StartTime = new DateTimeOffset (2009, 7, 11, 15, 00, 00, TimeSpan.Zero), Duration = TimeSpan.FromHours(4) }, new CalendarEvent { Title = "Saturday Night Swing", StartTime = new DateTimeOffset (2009, 7, 11, 19, 30, 00, TimeSpan.Zero), Duration = TimeSpan.FromHours(6.5) }, new CalendarEvent { Title = "Formula 1 German Grand Prix", StartTime = new DateTimeOffset (2009, 7, 12, 12, 10, 00, TimeSpan.Zero), Duration = TimeSpan.FromHours(3) }, new CalendarEvent { Title = "Swing Dance Picnic", StartTime = new DateTimeOffset (2009, 7, 12, 15, 00, 00, TimeSpan.Zero), Duration = TimeSpan.FromHours(4) }, new CalendarEvent { Title = "Stompin' at the 100 Club", StartTime = new DateTimeOffset (2009, 7, 13, 19, 45, 00, TimeSpan.Zero), Duration = TimeSpan.FromHours(5) } }; Notice that Example 7-10 uses the new keyword to initialize each object. This highlights an important point about arrays: individual array elements are similar to variables of the same type. Recall from Chapter 3 that a custom type defined with the class key- word, such as the CalendarEvent type in Example 7-9, is a reference type. This means that when you declare a variable of that type, the variable does not represent a particular object—it’s a storage location that can refer to an object. And the same is true of each element in an array if the element type is a reference type. Figure 7-1 shows the objects that Example 7-10 creates: five CalendarEvent objects (shown on the right), and an array 226 | Chapter 7: Arrays and Lists object of type CalendarEvent[] (shown on the left) where each element in the array refers to one of the event objects. Figure 7-1. An array with reference type elements As you saw in Chapter 3, with reference types multiple different variables can all refer to the same object. Since elements in an array behave in a similar way to local variables of the element type, we could create an array where all the elements refer to the same object, as shown in Example 7-11. Example 7-11. Multiple elements referring to the same object CalendarEvent theOnlyEvent = new CalendarEvent { Title = "Swing Dancing at the South Bank", StartTime = new DateTimeOffset (2009, 7, 11, 15, 00, 00, TimeSpan.Zero), Duration = TimeSpan.FromHours(4) }; CalendarEvent[] events = { theOnlyEvent, theOnlyEvent, theOnlyEvent, theOnlyEvent, theOnlyEvent }; Figure 7-2 illustrates the result. While this particular example is not brilliantly useful, in some situations it’s helpful for multiple elements to refer to one object. For example, imagine a feature for booking meeting rooms or other shared facilities—this could be a useful addition to a calendar program. An array might describe how the room will be used today, where each element represents a one-hour slot for a particular room. If Arrays | 227 the same individual had booked the same room for two different slots, the two corre- sponding array elements would both refer to the same person. Figure 7-2. An array where all of the elements refer to the same object Another feature that reference type array elements have in common with reference type variables and arguments is support for polymorphism. As you saw in Chapter 4, a variable declared as some particular reference type can refer to any object of that type, or of any type derived from the variable’s declared type. This works for arrays too— using the examples from Chapter 4, if an array’s type is FirefighterBase[], each ele- ment could refer to a Firefighter, or TraineeFirefighter, or anything else that derives from FirefighterBase. (And each element is allowed to refer to an object of a different type, as long as the objects are all compatible with the element type.) Likewise, you can declare an array of any interface type—for example, INamedPerson[], in which case each element can refer to any object of any type that implements that interface. Taking this to extremes, an array of type object[] has elements that can refer to any object of any reference type, or any boxed value. As you will remember from Chapter 3, the alternative to a reference type is a value type. With value types, each variable holds its own copy of the value, rather than a reference to some potentially shared object. As you would expect, this behavior carries over to arrays when the element type is a value type. Consider the array shown in Example 7-12. Example 7-12. An array of integer values int[] numbers = { 2, 3, 5, 7, 11 }; Like all the numeric types, int is a value type, so we end up with a rather different structure. As Figure 7-3 shows, the array elements are the values themselves, rather than references to values. Why would you need to care about where exactly the value lives? Well, there’s a sig- nificant difference in behavior. Given the numbers array in Example 7-12, consider this code: 228 | Chapter 7: Arrays and Lists int thirdElementInArray = numbers[2]; thirdElementInArray += 1; Console.WriteLine("Variable: " + thirdElementInArray); Console.WriteLine("Array element: " + numbers[2]); which would print out the following: Variable: 6 Array element: 5 Figure 7-3. An array with value type elements Because we are dealing with a value type, the thirdElementInArray local variable gets a copy of the value in the array. This means that the code can change the local variable without altering the element in the array. Compare that with similar code working on the array from Example 7-10: CalendarEvent thirdElementInArray = events[2]; thirdElementInArray.Title = "Modified title"; Console.WriteLine("Variable: " + thirdElementInArray.Title); Console.WriteLine("Array element: " + events[2].Title); This would print out the following: Variable: Modified title Array element: Modified title This shows that we’ve modified the event’s title both from the point of view of the local variable and from the point of view of the array element. That’s because both refer to the same CalendarEvent object—with a reference type, when the first line gets an ele- ment from the array we don’t get a copy of the object, we get a copy of the reference to that object. The object itself is not copied. The distinction between the reference and the object being referred to means that there’s sometimes scope for ambiguity—what exactly does it mean to change an ele- ment in an array? For value types, there’s no ambiguity, because the element is the value. The only way to change an entry in the numbers array in Example 7-12 is to assign a new value into an element: numbers[2] = 42; Arrays | 229 But as you’ve seen, with reference types the array element is just a reference, and we may be able to modify the object it refers to without changing the array element itself. Of course, we can also change the element, it just means something slightly different— we’re asking to change which object that particular element refers to. For example, this: events[2] = events[0]; causes the third element to refer to the same object as the first. This doesn’t modify the object that element previously referenced. (It might cause the object to become inac- cessible, though—if nothing else has a reference to that object, overwriting the array element that referred to it means the program no longer has any way of getting hold of that object, and so the .NET Framework can reclaim the memory it occupies during the next garbage collection cycle.) It’s often tempting to talk in terms of “the fourth object in the array,” and in a lot of cases, that’s a perfectly reasonable approximation in practice. As long as you’re aware that with reference types, array elements contain references, not objects, and that what you really mean is “the object referred to by the fourth element in the array” you won’t get any nasty surprises. Regardless of what element type you choose for an array, all arrays provide various useful methods and properties. Array Members An array is an object in its own right; distinct from any objects its elements may refer to. And like any object, it has a type—as you’ve already seen, we write an array type as SomeType[]. Whatever type SomeType may be, its corresponding array type, Some Type[], will derive from a standard built-in type called Array, defined in the System namespace. The Array base class provides a variety of services for working with arrays. It can help you find interesting items in an array. It can reorder the elements, or move information between arrays. And there are methods for working with the array’s size. Finding elements Suppose we want to find out if an array of calendar items contains any events that start on a particular date. An obvious way to do this would be to write a loop that iterates through all of the elements in the array, looking at each date in turn (see Example 7-13). Example 7-13. Finding elements with a loop DateTime dateOfInterest = new DateTime (2009, 7, 12); foreach (CalendarEvent item in events) { if (item.StartTime.Date == dateOfInterest) { Console.WriteLine(item.Title + ": " + item.StartTime); 230 | Chapter 7: Arrays and Lists } } Example 7-13 relies on a useful feature of the DateTimeOffset type that makes it easy to work out whether two DateTimeOffset values fall on the same day, regardless of the exact time. The Date property returns a DateTime in which the year, month, and day are copied over, but the time of day is set to the default time of midnight. Although Example 7-13 works just fine, the Array class provides an alternative: its FindAll method builds a new array containing only those elements in the original array that match whatever criteria you specify. Example 7-14 uses this method to do the same job as Example 7-13. Example 7-14. Finding elements with FindAll DateTime dateOfInterest = new DateTime (2009, 7, 12); CalendarEvent[] itemsOnDateOfInterest = Array.FindAll(events, e => e.StartTime.Date == dateOfInterest); foreach (CalendarEvent item in itemsOnDateOfInterest) { Console.WriteLine(item.Title + ": " + item.StartTime); } Notice that we’re using a lambda expression to tell FindAll which items match. That’s not mandatory—FindAll requires a delegate here, so you can use any of the alternatives discussed in Chapter 5, including lambda expressions, anonymous methods, method names, or any expression that returns a suitable delegate. The delegate type here is Predicate, where T is the array element type (Predicate in this case). We also discussed predicate delegates in Chapter 5, but in case your memory needs refreshing, we just need to supply a function that takes a CalendarEvent and returns true if it matches, and false if it does not. Example 7-14 uses the same expression as the if statement in Example 7-13. This may not seem like an improvement on Example 7-13. We’ve not written any less code, and we’ve ended up using a somewhat more advanced language feature—lambda expressions—to get the job done. However, notice that in Example 7-14, we’ve already done all the work of finding the items of interest before we get to the loop. Whereas the loop in Example 7-13 is a mixture of code that works out what items we need and code that does something with those items, Example 7-14 keeps those tasks neatly separated. And if we were doing more complex work with the matching items, that separation could become a bigger advantage—code tends to be easier to understand and maintain when it’s not trying to do too many things at once. The FindAll method becomes even more useful if you want to pass the set of matching items on to some other piece of code, because you can just pass the array of matches Arrays | 231 it returns as an argument to some method in your code. But how would you do that with the approach in Example 7-13, where the match-finding code is intermingled with the processing code? While the simple foreach loop in Example 7-13 is fine for trivial examples, FindAll and similar techniques (such as LINQ, which we’ll get to in the next chapter) are better at managing the more complicated scenarios likely to arise in real code. This is an important principle that is not limited to arrays or collections. In general, you should try to construct your programs by combining small pieces, each of which does one well-defined job. Code written this way tends to be easier to maintain and to contain fewer bugs than code written as one big, sprawling mass of complexity. Separating code that selects information from code that processes information is just one example of this idea. The Array class offers a few variations on the FindAll theme. If you happen to be in- terested only in finding the first matching item, you can just call Find. Conversely, FindLast returns the very last matching item. Sometimes it can be useful to know where in the array a matching item was found. So as an alternative to Find and FindLast, Array also offers FindIndex and FindLastIndex, which work in the same way except they return a number indicating the position of the first or last match, rather than returning the matching item itself. Finally, one special case for finding the index of an item turns out to crop up fairly often: the case where you know exactly which object you’re interested in, and just need to know where it is in the array. You could do this with a suitable predicate, for example: int index = Array.FindIndex(events, e => e == someParticularEvent); But Array offers the more specialized IndexOf and LastIndexOf, so you only have to write this: int index = Array.IndexOf(events, someParticularEvent); Ordering elements Sometimes it’s useful to modify the order in which entries appear in an array. For example, with a calendar, some events will be planned long in advance while others may be last-minute additions. Any calendar application will need to be able to ensure that events are displayed in chronological order, regardless of how they were added, so we need some way of getting items into the right order. The Array class makes this easy with its Sort method. We just need to tell it how we want the events ordered—it can’t really guess, because it doesn’t have any way of knowing whether we consider our events to be ordered by the Title, StartTime, or Duration property. This is a perfect job for a delegate: we can provide a tiny bit of code 232 | Chapter 7: Arrays and Lists that looks at two CalendarEvent objects and says whether one should appear before the other, and pass that code into the Sort method (see Example 7-15). Example 7-15. Sorting an array Array.Sort(events, (event1, event2) => event1.StartTime.CompareTo(event2.StartTime)); The Sort method’s first argument, events, is just the array we’d like to reorder. (We defined that back in Example 7-10.) The second argument is a delegate, and for con- venience we again used the lambda syntax introduced in Chapter 5. The Sort method wants to be able to know, for any two events, whether one should appear before the other, It requires a delegate of type Comparison, a function which takes two argu- ments—we called them event1 and event2 here—and which returns a number. If event1 is before event2, the number must be negative, and if it’s after, the number must be positive. We return zero to indicate that the two are equal. Example 7-15 just defers to the StartTime property—that’s a DateTimeOffset, which provides a handy CompareTo method that does exactly what we need. It turns out that Example 7-15 isn’t changing anything here, because the events array created in Example 7-10 happens to be in ascending order of date and time already. So just to illustrate that we can sort on any criteria, let’s order them by duration instead: Array.Sort(events, (event1, event2) => event1.Duration.CompareTo(event2.Duration)); This illustrates how the use of delegates enables us to plug in any number of different ordering criteria, leaving the Array class to get on with the tedious job of shuffling the array contents around to match the specified order. Some data types such as dates or numbers have an intrinsic ordering. It would be irri- tating to have to tell Array.Sort how to work out whether one number comes before or after another. And in fact we don’t have to—we can pass an array of numbers to a simpler overload of the Sort method, as shown in Example 7-16. Example 7-16. Sorting intrinsically ordered data int[] numbers = { 4, 1, 2, 5, 3 }; Array.Sort(numbers); As you would expect, this arranges the numbers into ascending order. We would pro- vide a comparison delegate here only if we wanted to sort the numbers into some other order. You might be wondering what would happen if we tried this simpler method with an array of CalendarEvent objects: Array.Sort(events); // Blam! Arrays | 233 If you try this, you’ll find that the method throws an InvalidOperationException, be- cause Array.Sort has no way of working out what order we need. It works only for types that have an intrinsic order. And should we want to, we could make Calen darEvent self-ordering. We just have to implement an interface called IComparable, which provides a single method, CompareTo. Example 7-17 implements this, and defers to the DateTimeOffset value in StartTime—the DateTimeOffset type implements IComparable. So all we’re really doing here is passing the responsibility on to the property we want to use for ordering, just like we did in Ex- ample 7-15. The one extra bit of work we do is to check for comparison with null— the IComparable interface documentation states that a non-null object should always compare as greater than null, so we return a positive number in that case. Without this check, our code would crash with a NullReferenceException if null were passed to CompareTo. Example 7-17. Making a type comparable class CalendarEvent : IComparable { public string Title { get; set; } public DateTimeOffset StartTime { get; set; } public TimeSpan Duration { get; set; } public int CompareTo(CalendarEvent other) { if (other == null) { return 1; } return StartTime.CompareTo(other.StartTime); } } Now that our CalendarEvent class has declared an intrinsic ordering for itself, we are free to use the simplest Sort overload: Array.Sort(events); // Works, now that CalendarEvent is IComparable Getting your array contents in order isn’t the only reason for relocating elements, so Array offers some slightly less specialized methods for moving data around. Moving or copying elements Suppose you want to build a calendar application that works with multiple sources of information—maybe you use several different websites with calendar features and would like to aggregate all the events into a single list. Example 7-18 shows a method that takes two arrays of CalendarEvent objects, and returns one array containing all the elements from both. Example 7-18. Copying elements from two arrays into one big one static CalendarEvent[] CombineEvents(CalendarEvent[] events1, CalendarEvent[] events2) { 234 | Chapter 7: Arrays and Lists CalendarEvent[] combinedEvents = new CalendarEvent[events1.Length + events2.Length]; events1.CopyTo(combinedEvents, 0); events2.CopyTo(combinedEvents, events1.Length); return combinedEvents; } This example uses the CopyTo method, which makes a complete copy of all the elements of the source array into the target passed as the first argument. The second argument says where to start copying elements into the target—Example 7-18 puts the first array’s elements at the start (offset zero), and then copies the second array’s elements directly after that. (So the ordering won’t be very useful—you’d probably want to sort the results after doing this.) You might sometimes want to be a bit more selective—you might want to copy only certain elements from the source into the target. For example, suppose you want to remove the first event. Arrays cannot be resized in .NET, but you could create a new array that’s one element shorter, and which contains all but the first element of the original array. The CopyTo method can’t help here as it copies the whole array, but you can use the more flexible Array.Copy method instead, as Example 7-19 shows. Example 7-19. Copying less than the whole array static CalendarEvent[] RemoveFirstEvent(CalendarEvent[] events) { CalendarEvent[] croppedEvents = new CalendarEvent[events.Length - 1]; Array.Copy( events, // Array from which to copy 1, // Starting point in source array croppedEvents, // Array into which to copy 0, // Starting point in destination array events.Length - 1 // Number of elements to copy ); return croppedEvents; } The key here is that we get to specify the index from which we want to start copying— 1 in this case, skipping over the first element, which has an index of 0. In practice, you would rarely do this—if you need to be able to add or remove items from a collection, you would normally use the List type that we’ll be looking at later in this chapter, rather than a plain array. And even if you are working with arrays, there’s an Array.Resize helper function that you would typically use in reality— it calls Array.Copy for you. However, you often have to copy data be- tween arrays, even if it might not be strictly necessary in this simple example. A more complex example would have obscured the essential simplicity of Array.Copy. Arrays | 235 The topic of array sizes is a little more complex than it first appears, so let’s look at that in more detail. Array Size Arrays know how many elements they contain—several of the previous examples have used the Length property to discover the size of an existing array. This read-only prop- erty is defined by the base Array class, so it’s always present.* That may sound like enough to cover the simple task of knowing an array’s size, but arrays don’t have to be simple sequential lists. You may need to work with multidimensional data, and .NET supports two different styles of arrays for that: jagged and rectangular arrays. Arrays of arrays (or jagged arrays) As we said earlier, you can make an array using any type as the element type. And since arrays themselves have types, it follows that you can have an array of arrays. For ex- ample, suppose we wanted to create a list of forthcoming events over the next five days, grouped by day. We could represent this as an array with one entry per day, and since each day may have multiple events, each entry needs to be an array. Example 7-20 creates just such an array. Example 7-20. Building an array of arrays static CalendarEvent[][] GetEventsByDay(CalendarEvent[] allEvents, DateTime firstDay, int numberOfDays) { CalendarEvent[][] eventsByDay = new CalendarEvent[numberOfDays][]; for (int day = 0; day < numberOfDays; ++day) { DateTime dateOfInterest = (firstDay + TimeSpan.FromDays(day)).Date; CalendarEvent[] itemsOnDateOfInterest = Array.FindAll(allEvents, e => e.StartTime.Date == dateOfInterest); eventsByDay[day] = itemsOnDateOfInterest; } return eventsByDay; } * There’s also a LongLength, which is a 64-bit version of the property, which theoretically allows for larger arrays than the 32-bit Length property. However, .NET currently imposes an upper limit on the size of any single array: it cannot use more than 2 GB of memory, even in a 64-bit process. So in practice, LongLength isn’t very useful in the current version of .NET (4). (You can use a lot more than 2 GB of memory in total in a 64-bit process—the 2 GB limit applies only to individual arrays.) 236 | Chapter 7: Arrays and Lists We’ll look at this one piece at a time. First, there’s the method declaration: static CalendarEvent[][] GetEventsByDay(CalendarEvent[] allEvents, DateTime firstDay, int numberOfDays) { The return type—CalendarEvent[][]—is an array of arrays, denoted by two pairs of square brackets. You’re free to go as deep as you like, by the way—it’s perfectly possible to have an array of arrays of arrays of arrays of anything. The method’s arguments are fairly straightforward. This method expects to be passed a simple array containing an unstructured list of all the events. The method also needs to know which day we’d like to start from, and how many days we’re interested in. The very first thing the method does is construct the array that it will eventually return: CalendarEvent[][] eventsByDay = new CalendarEvent[numberOfDays][]; Just as new CalendarEvent[5] would create an array capable of containing five CalendarEvent elements, new CalendarEvent[5][] would create an array capable of containing five arrays of CalendarEvent objects. Since our method lets the caller specify the number of days, we pass that argument in as the size of the top-level array. Remember that arrays are reference types, and that whenever you create a new array whose element type is a reference type, all the elements are initially null. So although our new eventsByDay array is capable of referring to an array for each day, what it holds right now is a null for each day. So the next bit of code is a loop that will populate the array: for (int day = 0; day < numberOfDays; ++day) { ... } Inside this loop, the first couple of lines are similar to the start of Example 7-14: DateTime dateOfInterest = (firstDay + TimeSpan.FromDays(day)).Date; CalendarEvent[] itemsOnDateOfInterest = Array.FindAll(allEvents, e => e.StartTime.Date == dateOfInterest); The only difference is that this example calculates which date to look at as we progress through the loop. So Array.FindAll will return an array containing all the events that fall on the day for the current loop iteration. The final piece of code in the loop puts that into our array of arrays: eventsByDay[day] = itemsOnDateOfInterest; Once the loop is complete, we return the array: return eventsByDay; } Each element will contain an array with the events that fall on the relevant day. Arrays | 237 Code that uses such an array can use the normal element access syntax, for example: Console.WriteLine("Number of events on first day: " + eventsByDay[0].Length); Notice that this code uses just a single index—this means we want to retrieve one of the arrays from our array of arrays. In this case, we’re looking at the size of the first of those arrays. Or we can dig further by providing multiple indexes: Console.WriteLine("First day, second event: " + eventsByDay[0][1].Title); This syntax, with its multiple sets of square brackets, fits right in with the syntax used to declare and construct the array of arrays. So why is an array of arrays sometimes called a jagged array? Figure 7-4 shows the various objects you would end up with if you called the method in Example 7-20, passing the events from Example 7-10, asking for five days of events starting from July 11. The figure is laid out to show each child array as a row, and as you can see, the rows are not all the same length—the first couple of days have two items per row, the third day has one, and the last two are empty (i.e., they are zero-length arrays). So rather than looking like a neat rectangle of objects, the rows form a shape with a somewhat uneven or “jagged” righthand edge. This jaggedness can be either a benefit or a problem, depending on your goals. In this example, it’s helpful—we used it to handle the fact that the number of events in our calendar may be different every day, and some days may have no events at all. But if you’re working with information that naturally fits into a rectangular structure (e.g., pixels in an image), rows of differing lengths would constitute an error—it would be better to use a data structure that doesn’t support such things, so you don’t have to work out how to handle such an error. Moreover, jagged arrays end up with a relatively complicated structure—there are a lot of objects in Figure 7-4. Each array is an object distinct from the objects its element refers to, so we’ve ended up with 11 objects: the five events, the five per-day arrays (including two zero-length arrays), and then one array to hold those five arrays. In situations where you just don’t need this flexibility, there’s a simpler way to represent multiple rows: a rectangular array. Rectangular arrays A rectangular array† lets you store multidimensional data in a single array, rather than needing to create arrays of arrays. They are more regular in form than jagged arrays— in a two-dimensional rectangular array, every row has the same width. † Rectangular arrays are also sometimes called multidimensional arrays, but that’s a slightly confusing name, because jagged arrays also hold multidimensional data. 238 | Chapter 7: Arrays and Lists Rectangular arrays are not limited to two dimensions, by the way. Just as you can have arrays of arrays of arrays, so you can have any number of dimensions in a “rectangular” array, although the name starts to sound a bit wrong. With three dimensions, it’s a cuboid rather than a rectangle, and more generally the shape of these arrays is always an orthotope. Presumably the designers of C# and the .NET Framework felt that this “proper” name was too obscure (as does the spellchecker in Word) and that rectangular was more usefully descriptive, despite not being technically correct. Pragmatism beat pedantry here because C# is fundamentally a practical language. Figure 7-4. A jagged array Arrays | 239 Rectangular arrays tend to suit different problems than jagged arrays, so we need to switch temporarily to a different example. Suppose you were writing a simple game in which a character runs around a maze. And rather than going for a typical modern 3D game rendered from the point of view of the player, imagine something a bit more retro—a basic rendering of a top-down view, and where the walls of the maze all fit neatly onto a grid. If you’re too young to remember this sort of thing, Figure 7-5 gives a rough idea of what passed for high-tech entertainment back when your authors were at school. Figure 7-5. Retro gaming—3D is for wimps We don’t want to get too hung up on the details of the game play, so let’s just assume that our code needs to know where the walls are in order to work out where the player can or can’t move next, and whether she has a clean shot to take out the baddies chasing her through the maze. We could represent this as an array of numbers, where 0 repre- sents a gap and 1 represents a wall, as Example 7-21 shows. (We could also have used bool instead of int as the element type, as there are only two possible options: a wall or no wall. However, using true and false would have prevented each row of data from fitting on a single row in this book, making it much harder to see how Example 7-21 reflects the map in Figure 7-5. Moreover, using numbers leaves open the option to add exciting game features such as unlockable doors, squares of instant death, and other classics.) 240 | Chapter 7: Arrays and Lists Example 7-21. A multidimensional rectangular array int[,] walls = new int[,] { { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 }, { 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1 }, { 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1 }, { 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1 }, { 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1 }, { 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0 }, { 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1 }, { 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1 }, { 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1 }, { 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1 }, { 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1 }, { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } }; There are a couple of differences between this and previous examples. First, notice that the array type has a comma between the square brackets. The number of commas indicates how many dimensions we want—no commas at all would mean a one- dimensional array, which is what we’ve been using so far, but the single comma here specifies a two-dimensional array. We could represent a cuboid layout with int[,,], and so on, into as many dimensions as your application requires. The second thing to notice here is that we’ve not had to use the new keyword for each row in the initializer list—new appears only once, and that’s because this really is just a single object despite being multidimensional. As Figure 7-6 illustrates, this kind of array has a much simpler structure than the two-dimensional jagged array in Figure 7-4. While Figure 7-6 is accurate in the sense that just one object holds all the values here, the grid-like layout of the numbers is not a literal rep- resentation of how the numbers are really stored, any more than the position of the various objects in Figure 7-4 is a literal representation of what you’d see if you peered into your computer’s memory chips with a scanning electron microscope. In reality, multidimensional arrays store their elements as a sequential list just like the simple array in Figure 7-3, because computer memory itself is just a big sequence of storage locations. But the programming model C# presents makes it look like the array really is multidimensional. The syntax for accessing elements in a rectangular array is slightly different from that of a jagged array. But like a jagged array, the access syntax is consistent with the dec- laration syntax—as Example 7-22 shows, we use a single pair of square brackets, pass- ing in an index for each dimension, separated by commas. Arrays | 241 Figure 7-6. A two-dimensional rectangular array Example 7-22. Accessing an element in a rectangular array static bool CanCharacterMoveDown(int x, int y, int[,] walls) { int newY = y + 1; // Can't move off the bottom of the map if (newY == walls.GetLength(0)) { return false; } // Can only move down if there's no wall in the way return walls[newY, x] == 0; } If you pass in the wrong number of indexes, the C# compiler will com- plain. The number of dimensions (or rank, to use the official term) is considered to be part of the type: int[,] is a different type than int[,,], and C# checks that the number of indexes you supply matches the array type’s rank. 242 | Chapter 7: Arrays and Lists Example 7-22 performs two checks: before it looks to see if there’s a wall in the way of the game character, it first checks to see if the character is up against the edge of the map. To do this, it needs to know how big the map is. And rather than assuming a fixed-size grid, it asks the array for its size. But it can’t just use the Length property we saw earlier—that returns the total number of elements. Since this is a 12 × 12 array, Length will be 144. But we want to know the length in the vertical dimension. So instead, we use the GetLength method, which takes a single argument indicating which dimen- sion you want—0 would be the vertical dimension and 1 in this case is horizontal. Arrays don’t really have any concept of horizontal and vertical. They simply have as many dimensions as you ask for, and it’s up to your program to decide what each dimension is for. This particular program has chosen to use the first dimension to represent the vertical position in the maze, and the second dimension for the horizontal position. This rectangular example has used a two-dimensional array of integers, and since int is a value type, the values get to live inside the array. You can also create multidimen- sional rectangular arrays with reference type elements. In that case, you’ll still get a single object containing all the elements of the array in all their dimensions, but these individual elements will be null references—you’ll need to create objects for them to refer to, just like you would with a single-dimensional array. While jagged and rectangular multidimensional arrays give us flexibility in terms of how to specify the size of an array, we have not yet dealt with an irritating sizing problem mentioned back at the start of the chapter: an array’s size is fixed. We saw that it’s possible to work around this by creating new arrays and copying some or all of the old data across, or by getting the Array.Resize method to do that work for us. But these are inconvenient solutions, so in practice, we rarely work directly with arrays in C#. There’s a far easier way to work with changing collection sizes, thanks to the List class. List The List class, defined in the System.Collections.Generic namespace, is effectively a resizable array. Strictly speaking, it’s just a generic class provided by the .NET Frame- work class library, and unlike arrays, List does not get any special treatment from the type system or the CLR. But from a C# developer’s perspective, it feels very similar—you can do most of the things you could do with an array, but without the restriction of a fixed size. List | 243 Generics List is an example of a generic type. You do not use a generic type directly; you use it to build new types. For example, List is a list of integers, and List is a list of strings. These are two types in their own right, built by passing different type arguments to List. Plugging in type arguments to form a new type is called instan- tiating the generic type. Generics were added in C# 2.0 mainly to support collection classes such as List. Before this, we had to use the ArrayList class (which you should no longer use; it’s not present in Silverlight, and may eventually be deprecated in the full .NET Framework). ArrayList was also a resizable array, but it represented all items as object. This meant it could hold anything, but every time you read an element, you were obliged to cast to the type you were expecting, which was messy. With generics, we can write code that has one or more placeholder type names—the T in List, for example. We call these type parameters. (The distinction between parameters and arguments is the same here as it is for methods: a parameter is a named placeholder, whereas an argument is a specific value or type provided for that parameter at the point at which you use the code.) So you can write code like this: public class Wrapper { public Wrapper(T v) { Value = v; } public T Value { get; private set; } } This code doesn’t need to know what type T is—and in fact T can be any type. If we want a wrapper for an int, we can write Wrapper, and that generates a class exactly like the example, except with the T replaced by int throughout. Some classes take multiple type parameters. Dictionary collections (which are descri- bed in Chapter 9) require both a key and a value type, so you would specify, say, Dictionary. An instantiated generic type is a type in its own right, so you can use one as an argument for another generic type, for example, Diction ary>. You can also specify a type parameter list for a method. For example, .NET defines an extension method for all collections called OfType. If you have a List that happens to contain a mixture of different kinds of objects, you can retrieve just the items that are of type string by calling myList.OfType(). You may be wondering why .NET offers arrays when List appears to be more useful. The answer is that it wouldn’t be possible for List to exist if there were no arrays: List uses an array internally to hold its elements. As you add elements, it allocates new, larger arrays as necessary, copying the old contents over. It employs various tricks to minimize how often it needs to do this. 244 | Chapter 7: Arrays and Lists List is one of the most useful types in the .NET Framework. If you’re dealing with multiple pieces of information, as programs often do, it’s very common to need some flexibility around the amount of information—fixed-size lists are the exception rather than the rule. (An individual’s calendar tends to change over time, for example.) So have we just wasted your time with the first half of this chapter? Not at all—not only do arrays crop up a lot in APIs, but List collections are very similar in use to arrays. We could migrate most of the examples seen so far in this chapter from arrays to lists. Returning to our earlier, nonrectangular example, we would need to modify only the first line of Example 7-10, which creates an array of CalendarEvent objects. That line currently reads: CalendarEvent[] events = It is followed by the list of objects to add to the array, contained within a pair of braces. If you change that line to this: List events = new List the initializer list can remain the same. Notice that besides changing the variable dec- laration to use the List type (with the generic type argument T set to the element type CalendarEvent, of course) we also need an explicit call to the constructor. (Nor- mally, you’d expect parentheses after the type name when invoking a constructor, but those are optional when using an initializer list.) As you saw earlier, the use of new is optional when assigning a value to a newly declared array, but C# does not extend that courtesy to other collection types. While we can initialize the list in much the same way as we would an array, the differ- ence is that we are free to add and remove elements later. To add a new element, we can use the Add method: CalendarEvent newEvent = new CalendarEvent { Title = "Dean Collins Shim Sham Lesson", StartTime = new DateTimeOffset (2009, 7, 14, 19, 15, 00, TimeSpan.Zero), Duration = TimeSpan.FromHours(1) }; events.Add(newEvent); This appends the element to the end of the list. If you want to put the new element somewhere other than at the end, you can use Insert: events.Insert(2, newEvent); The first argument indicates the index at which you’d like the new item to appear— any items at or after this index will be moved down to make space. You can also remove items, either by index, using the RemoveAt method, or by passing the value you’d like to remove to the Remove method (which will remove the first element it finds that con- tains the specified value). List | 245 List does not have a Length property, and instead offers a Count. This may seem like pointless inconsistency with arrays, but there’s a reason. An array’s Length property is guaranteed not to change. A List can- not make that guarantee, and so the behavior of its Count property is necessarily different from an array’s Length. The use of different names signals the fact that the semantics are subtly different. List also offers AddRange, which lets you add multiple elements in a single step. This makes it much easier to concatenate lists—remember that with arrays we ended up writing the CombineEvents method in Example 7-18 to concatenate a couple of arrays. But with lists, it becomes as simple as the code shown in Example 7-23. Example 7-23. Adding elements from one list to another events1.AddRange(events2); The one possible downside of List is that this kind of operation modifies the first list. Example 7-18 built a brand-new array, leaving the two input arrays unmodified, so if any code happened still to be using those original arrays, it would carry on working. But Example 7-23 modifies the first list by adding in the events from the second list. You would need to be confident that nothing in your code was relying on the first list containing only its original content. Of course, you could always build a brand-new new List from the contents of two existing lists. (There are various ways to do this, but one straightforward ap- proach is to construct a new List and then call AddRange twice, once for each list.) You access elements in a List with exactly the same syntax as for an array. For example: Console.WriteLine("List element: " + events[2].Title); As with arrays, a List will throw an IndexOutOfRangeException if you use too high an index, or a negative index. This applies for writes as well as reads—a List will not automatically grow if you write to an index that does not yet exist. There is a subtle difference between array element access and list element access that can cause problems with custom value types (structs). You may recall that Chapter 3 warned that when writing a custom value type, it’s best to make it immutable if you plan to use it in a collection. To understand why, you need to know how List makes the square bracket syntax for element access work. 246 | Chapter 7: Arrays and Lists Custom Indexers Arrays are an integral part of the .NET type system, so C# knows exactly what to do when you access an array element using the square bracket syntax. However, as List demonstrates, it’s also possible to use this same syntax with some objects that are not arrays. For this to work, the object’s type needs to help C# out by defining the behavior for this syntax. This takes the form of a slightly unusual-looking property, as shown in Example 7-24. Example 7-24. A custom indexer class Indexable { public string this[int index] { get { return "Item " + index; } set { Console.WriteLine("You set item " + index + " to " + value); } } } This has the get and set parts we’d expect in a normal property, but the definition line is a little unusual: it starts with the accessibility and type as normal, but where we’d expect to see the property name we instead have this[int index]. The this keyword signifies that this property won’t be accessed by any name. It is followed by a parameter list enclosed in square brackets, signifying that this is an indexer property, defining what should happen if we use the square bracket element access syntax with objects of this type. For example, look at the code in Example 7-25. Example 7-25. Using a custom indexer Indexable ix = new Indexable(); Console.WriteLine(ix[10]); ix[42] = "Xyzzy"; After constructing the object, the next line uses the same element access syntax you’d use to read an element from an array. But this is not an array, so the C# compiler will look for a property of the kind shown in Example 7-24. If you try this on a type that doesn’t provide an indexer, you’ll get a compiler error, but since this type has one, that ix[10] expression ends up calling the indexer’s get accessor. Similarly, the third line has the element access syntax on the lefthand side of an assignment, so C# will use the indexer’s set accessor. List | 247 If you want to support the multidimensional rectangular array style of index (e.g., ix[10, 20]), you can specify multiple parameters between the square brackets in your indexer. Note that the List class does not do this—while it covers most of the same ground as the built-in array types, it does not offer rectangular multidimensional behavior. You’re free to create a jagged list of lists, though. For example, List> is a list of lists of integers, and is similar in use to an int[][]. The indexer in Example 7-24 doesn’t really contain any elements at all—it just makes up a value in the get, and prints out the value passed into set without storing it any- where. So if you run this code, you’ll see this output: Item 10 You set item 42 to Xyzzy It may seem a bit odd to provide array-like syntax but to discard whatever values are “written,” but this is allowed—there’s no rule that says that indexers are required to behave in an array-like fashion. In practice, most do—the reason C# supports indexers is to make it possible to write classes such as List that feel like arrays without nec- essarily having to be arrays. So while Example 7-24 illustrates that you’re free to do whatever you like in a custom indexer, it’s not a paragon of good coding style. What does any of this have to do with value types and immutability, though? Look at Example 7-26. It has a public field with an array and also an indexer that provides access to the array. Example 7-26. Arrays versus indexers // This class's purpose is to illustrate a difference between // arrays and indexers. Do not use this in real code! class ArrayAndIndexer { public T[] TheArray = new T[100]; public T this[int index] { get { return TheArray[index]; } set { TheArray[index] = value; } } } 248 | Chapter 7: Arrays and Lists You might think that it would make no difference whether we use this class’s indexer, or go directly for the array. And some of the time that’s true, as it is in this example: ArrayAndIndexer aai = new ArrayAndIndexer(); aai.TheArray[10] = 42; Console.WriteLine(aai[10]); aai[20] = 99; Console.WriteLine(aai.TheArray[20]); This swaps freely between using the array and the indexer, and as the output shows, items set through one mechanism are visible through the other: 42 99 However, things are a little different if we make this class store a mutable value type. Here’s a very simple modifiable value type: struct CanChange { public int Number { get; set; } public string Name { get; set; } } The Number and Name properties both have setters, so this is clearly not an immutable type. This might not seem like a problem—we can do more or less exactly the same with this type as we did with int just a moment ago: ArrayAndIndexer aai = new ArrayAndIndexer(); aai.TheArray[10] = new CanChange { Number = 42 }; Console.WriteLine(aai[10].Number); aai[20] = new CanChange { Number = 99, Name = "My item" }; Console.WriteLine(aai.TheArray[20].Number); That works fine. The problem arises when we try to modify a property of one of the values already inside the array. We can do it with the array: aai.TheArray[10].Number = 123; Console.WriteLine(aai.TheArray[10].Number); That works—it prints out 123 as you’d expect. But this does not work: aai[20].Number = 456; If you try this, you’ll find that the C# compiler reports the following error: error CS1612: Cannot modify the return value of 'ArrayAndIndexer.this[int]' because it is not a variable That’s a slightly cryptic message. But the problem becomes clear when we think about what we just asked the compiler to do. The intent of this code: aai[20].Number = 456; seems clear—we want to modify the Number property of the item whose index is 20. And remember, this line of code is using our ArrayAndIndexer class’s indexer. Look- ing at Example 7-26, which of the two accessors would you expect it to use here? Since List | 249 we’re modifying the value, you might expect set to be used, but a set accessor is an all or nothing proposition: calling set means you want to replace the whole element. But we’re not trying to do that here—we just want to modify the Number property of the value, leaving its Name property unmodified. If you look at the set code in Exam- ple 7-26, it simply doesn’t offer that as an option—it will completely replace the element at the specified index in the array. The set accessor can come into play only when we’re providing a whole new value for the element, as in: aai[20] = new CanChange { Number = 456 }; That compiles, but we end up losing the Name property that the element in that location previously had, because we overwrote the entire value of the element. Since set doesn’t work, that leaves get. The C# compiler could interpret this code: aai[20].Number = 456; as being equivalent to the code in Example 7-27. Example 7-27. What the compiler might have done CanChange elem = aai[20]; elem.Number = 456; And in fact, that’s what it would have done if we were using a reference type. However, it has noticed that CanChange is a value type, and has therefore rejected the code. (The error message says nothing about value types, but you can verify that this is the heart of the problem by changing the CanChange type from a struct to a class. That removes the compiler error, and you’ll find that the code aai[20].Number = 456 works as expected.) Why has the compiler rejected this seemingly obvious solution? Well, remember that the crucial difference between reference types and value types is that values usually involve copies—if you retrieve a value from an indexer, the indexer returns a copy. So in Example 7-27 the elem variable holds a copy of the item at index 20. Setting elem.Number to 456 has an effect on only that copy—the original item in the array remains unchanged. This makes clear why the compiler rejected our code—the only thing it can do with this: aai[20].Number = 456; is to call the get accessor, and then set the Number property on the copy returned by the array, leaving the original value unaltered. Since the copy would then immediately be discarded, the compiler has wisely determined that this is almost certainly not what we meant. (If we really want that copy-then-modify behavior, we can always write the code in Example 7-27 ourselves, making the fact that there’s a copy explicit. Putting the copy into a named variable also gives us the opportunity to go on and do something with the copy, meaning that setting a property on the copy might no longer be a waste of effort.) 250 | Chapter 7: Arrays and Lists You might be thinking that the compiler could read and modify a copy like Example 7-27, and then write that value back using the set indexer accessor. However, as Example 7-24 showed, indexer accessors are not required to work in the obvious way, and more generally, accessors can have side effects. So the C# compiler cannot assume that such a get- modify-set sequence is necessarily safe. This problem doesn’t arise with reference types, because in that case, the get accessor returns a reference rather than a value—no copying occurs because that reference refers to the same object that the corresponding array entry refers to. But why does this work when we use the array directly? Recall that the compiler didn’t have a problem with this code: aai.TheArray[10].Number = 123; It lets that through because it’s able to make that behave like we expect. This will in fact modify the Number property of the element in the array. And this is the rather subtle difference between an array and an indexer. With an array you really can work directly with the element inside the array—no copying occurs in this example. This works because the C# compiler knows what an array is, and is able to generate code that deals directly with array elements in situ. But there’s no way to write a custom indexer that offers the same flexibility. (There are reasons for this, but to explain them would require an exploration of the .NET Framework’s type safety rules, which would be lengthy and quite outside the scope of this chapter.) Having established the root of the problem, let’s look at what this means for List. Immutability and List The List class gets no special privileges—it may be part of the .NET Framework class library, but it is subject to the same restrictions as your code. And so it has the same problem just described—the following code will produce the same compiler error you saw in the preceding section: List numbers = new List { new CanChange() }; numbers[0].Number = 42; // Will not compile One way of dealing with this would be to avoid using custom value types in a collection class such as List, preferring custom reference types instead. And that’s not a bad rule of thumb—reference types are a reasonable default choice for most data types. However, value types do offer one compelling feature if you happen to be dealing with very large volumes of data. As Figure 7-1 showed earlier, an array with reference type elements results in an object for the array itself, and one object for each element in the array. But when an array has value type elements, you end up with just one object— the values live inside the array, as Figure 7-3 illustrates. List has similar character- istics because it uses an array internally. List | 251 For an array with hundreds of thousands of elements, the simpler structure of Figure 7-3 can have a noticeable impact on performance. For example, I just ran a quick test on my computer to see how long it would take to create a List with 500,000 entries, and then run through the list, adding the Number values together. Example 7-28 shows the code—it uses the Stopwatch class from the System.Diagnos tics namespace, which provides a handy way to see how long things are taking. Example 7-28. Microbenchmarking values versus references in lists Stopwatch sw = new Stopwatch(); sw.Start(); int itemCount = 500000; List items = new List(itemCount); for (int i = 0; i < itemCount; ++i) { items.Add(new CanChange { Number = i }); } sw.Stop(); Console.WriteLine("Creation: " + sw.ElapsedTicks); sw.Reset(); sw.Start(); int total = 0; for (int i = 0; i < itemCount; ++i) { total += items[i].Number; } sw.Stop(); Console.WriteLine("Total: " + total); Console.WriteLine("Sum: " + sw.ElapsedTicks); With CanChange as a value type, it takes about 150 ms on my machine to populate the list, and then about 40 ms to run through all the numbers, adding them together. But if I change CanChange from a struct to a class (i.e., make it a reference type) the numbers become more like 600 ms and 50 ms, respectively. So that’s about 25 percent longer to perform the calculations but a staggering four times longer to create the collection in the first place. And that’s because with CanChange as a reference type, we now need to ask the .NET Framework to create half a million objects for us instead of just one object when we initialize the list. From the perspective of an end user, this is the difference between a tiny hiatus and an annoyingly long delay—when an application freezes for more than half a second, users begin to wonder if it has hung, which is very disruptive. 252 | Chapter 7: Arrays and Lists Please don’t take away the message that value types are four times faster than reference types—they aren’t. A micro benchmark like this should always be taken with a very strong pinch of salt. All we’ve really meas- ured here is how long it takes to do something contrived in an isolated and artificial experiment. This example is illuminating only insofar as it demonstrates that the choice between value types and reference types can sometimes have a profound effect. It would be a mistake to draw a generalized conclusion from this. Notice that even in this example we see significant variation: the first part of the code slowed down by a factor of four, but in the second part, the impact was much smaller. In some scenarios, there will be no meas- urable difference, and as it happens there are situations in which value types can be shown to be slower than reference types. The bottom line is this: the only important performance measurements are ones you make yourself on the system you are building. If you think your code might get a useful speedup by using a value type instead of a reference type in a large collection, measure the effect of that change, rather than doing it just because some book said it would be faster. Since the use of value types in a collection can sometimes offer very useful performance benefits, the rule of thumb we suggested earlier—always use reference types—looks too restrictive in practice. So this is where immutability comes into play. As we saw earlier in this section, the fact that a get accessor can only return a copy of a value type causes problems if you ever need to modify a value already in a collection. But if your value types are immutable, you will never hit this problem. And as we’ll see in Chap- ter 16, there are other benefits to immutable types. So we now know how List is able to make itself resemble an array. Having under- stood some of the subtle differences between array element access and custom indexers, let’s get back to some of the other functionality of List. Finding and Sorting Earlier we saw that the Array class offers a variety of helper methods for finding elements in arrays. If you try to use these directly on a List, it won’t work. The following code from Example 7-14 will not compile if events is a List, for example: DateTime dateOfInterest = new DateTime (2009, 7, 12); CalendarEvent[] itemsOnDateOfInterest = Array.FindAll(events, e => e.StartTime.Date == dateOfInterest); List | 253 This will cause an error, because Array.FindAll expects an array, and we’re now giving it a List. However, all the finding and sorting functionality we saw earlier is still available; you just have to use the methods provided by List instead of Array: DateTime dateOfInterest = new DateTime(2009, 7, 12); List itemsOnDateOfInterest = events.FindAll( e => e.StartTime.Date == dateOfInterest); Notice a slight stylistic difference—whereas with arrays, FindAll is a static method provided by the Array class, List chooses to make its FindAll method an instance member—so we invoke it as events.FindAll. Style aside, it works in exactly the same way. As you might expect, it returns its results as another List rather than as an array. This same stylistic difference exists with all the other techniques we looked at before. List provides Find, FindLast, FindIndex, FindLastIndex, IndexOf, LastIndexOf, and Sort methods that all work in almost exactly the same way as the array equivalents we looked at earlier, but again, they’re instance methods rather than static methods. Since List offers almost everything you’re likely to want from an array and more besides, List will usually be your first choice to represent a collection of data. (The only common exception is if you need a rectangular array.) Unfortunately, you will sometimes come up against APIs that simply require you to provide an array. In fact, we already wrote some code that does this: the AddNumbers method back in Exam- ple 7-3 requires its input to be in the form of an array. But even this is easy to deal with: List provides a handy ToArray() method for just this eventuality, building a copy of the list’s contents in array form. But wouldn’t it be better if we could write our code in such a way that it didn’t care whether incoming information was in an array, a List, or some other kind of col- lection? It is possible to do exactly this, using the polymorphism techniques discussed in Chapter 4. Collections and Polymorphism Polymorphic code is code that is able to work on a variety of different forms of data. The foreach keyword has this characteristic. For example: foreach (CalendarEvent ev in events) { Console.WriteLine(ev.Title); } This code works if events is an array—CalendarEvent[]—but it works equally well if events is a List. And in fact, there are many more specialized collection types in the .NET Framework class library that we’ll look at in a later chapter that foreach can work with. You can even arrange for it to work with custom collection classes you may have written yourself. All this is possible because the .NET Framework 254 | Chapter 7: Arrays and Lists defines some standard interfaces for representing collections of things. The foreach construct depends on a pair of interfaces: IEnumerable and IEnumerator. These derive from a couple of nongeneric base interfaces, IEnumerable and IEnumerator. These interfaces are defined in the class library, and they are reproduced in Example 7-29. Example 7-29. Enumeration interfaces namespace System.Collections.Generic { public interface IEnumerable : IEnumerable { new IEnumerator GetEnumerator(); } public interface IEnumerator : IDisposable, IEnumerator { new T Current { get; } } } namespace System.Collections { public interface IEnumerable { IEnumerator GetEnumerator(); } public interface IEnumerator { bool MoveNext(); object Current { get; } void Reset(); } } The split between the generic and nongeneric interfaces here is a historical artifact. Versions 1.0 and 1.1 of .NET did not support generics, so only the base IEnumerable and IEnumerator interfaces existed. When .NET 2.0 shipped in 2005, generics were introduced, making it possible to provide versions of these interfaces that were explicit about what type of objects a collection contains, but in order to maintain backward compatibility the old version 1.x interfaces had to remain. You will normally use the generic versions, because they are easier to work with. Conceptually, if a type implements IEnumerable it is declaring that it contains a sequence of items of type T. To get hold of the items, you can call the GetEnumerator method, which will return an IEnumerator. An enumerator is an object that lets you work through the objects in an enumerable collection one at a time.‡ The split between enumerables and enumerators makes it possible to have different parts of your program ‡ If you’re familiar with C++ and its Standard Template Library, an enumerator is broadly similar in concept to an iterator in the STL. Collections and Polymorphism | 255 working their way through the same collection at the same time, without all of them needing to be in the same place. This can be useful in multithreaded applications (al- though as we’ll see in a later chapter, you have to be extremely careful about letting multiple threads use the same data structure simultaneously). Some enumerable collections, such as List, can be modified. (.NET defines an IList interface to represent the abstract idea of a modifi- able, ordered collection. List is just one implementation IList.) You should avoid modifying a collection while you’re in the process of iterating through it. For example, do not call Add on a List in the middle of a foreach loop that uses that list. List detects when this happens, and throws an exception. Note that unlike IList, IEnumerable does not provide any meth- ods for modifying the sequence. While this provides less flexibility to the consumer of a sequence, it broadens the range of data that can be wrapped as an IEnumerable. For some sources of data it doesn’t make sense to provide consumers of that data with the ability to reorder it. These interfaces make it possible to write a function that uses a collection without having any idea of the collection’s real type—you only need to know what type of elements it contains. We could rewrite Example 7-3 so that it works with any IEnumer able rather than just an array of strings, as shown in Example 7-30. Example 7-30. Using IEnumerable and IEnumerator static string[] AddNumbers(IEnumerable names) { List numberedNames = new List(); using (IEnumerator enumerator = names.GetEnumerator()) { int i = 0; while (enumerator.MoveNext()) { string currentName = enumerator.Current; numberedNames.Add(string.Format("{0}: {1}", i, currentName)); i += 1; } } return numberedNames.ToArray(); } Since List and arrays both implement IEnumerable, this modified code in Ex- ample 7-30 will now work with List, as well as arrays, or any other collection class that implements IEnumerable. For more information on the subtleties of type compatibility and enumerations, see the sidebar on the next page. 256 | Chapter 7: Arrays and Lists Enumerations and Variance Suppose you’ve written a function that uses an enumeration of elements of some base type, perhaps an IEnumerable. (Chapter 4 defined FirefighterBase as a base class of various types representing firefighters.) For example: static void ShowNames(IEnumerable people) { foreach (FirefighterBase person in people) { Console.WriteLine(person.Name); } } What would you expect to happen if you tried to pass this method an IEnumera ble, where TraineeFirefighter derives from FirefighterBase? It seems like it should work—ShowNames expects to get a sequence of FirefighterBase objects, and since TraineeFirefighter derives from FirefighterBase, an IEnumera ble will return a sequence of objects that are all of type Firefight erBase (as well as being of type TraineeFirefighter). In C# 4.0, this works as you’d expect. But it didn’t in previous versions. In general, it’s not safe to assume that types are necessarily compatible just because their type argu- ments happen to be compatible. For example, there’s an IList interface which de- fines an Add method. IList cannot safely be converted to IList, because the latter’s Add method would allow anything derived from FirefighterBase (e.g., Firefighter, TraineeFirefighter) to be added, but in prac- tice the implementer of IList might not allow that—it might ac- cept only the TraineeFirefighter type. IEnumerable works here because the T type only ever comes out of an enumeration; there’s no way to pass instances of T into IEnumerable. The interface definition states this—as Example 7-29 shows, the type argument is prefixed with the out keyword. In the official terminology, this means that IEnumerable is covariant with T. This means that if type D derives from type B (or is otherwise type-compatible—maybe B is an in- terface that D implements), IEnumerable is type-compatible with IEnumerable. Generic arguments can also be prefixed with the in keyword, meaning that the type is only ever passed in, and will never be returned. The IComparable interface we saw earlier happens to work this way. In this case, we say that IComparable is contra- variant with T—it works the other way around. You cannot pass an IComparable to a method expecting an IComparable, because that method might pass in a different kind of FirefighterBase, such as Firefighter. But you can pass an IComparable to a method expecting an ICompara ble (even though you cannot pass a FirefighterBase to a method expecting a TraineeFirefighter). An IComparable is capable of being compared to any FirefighterBase, and is therefore able to be compared with a TraineeFirefighter. By default, generic arguments are neither covariant nor contravariant. C# 4.0 intro- duced support for variance because the absence of variance with collection interfaces just seemed wrong—IEnumerable now works like most developers would expect. Collections and Polymorphism | 257 Example 7-30 works much harder than it needs to—it creates the enumerator explicitly, and walks through the objects by calling MoveNext in a loop, retrieving the Current value each time around. (A newly created enumerator needs us to call MoveNext before first reading Current. It doesn’t automatically start on the first item because there might not be one—collections can be empty.) As it happens, that’s exactly what foreach does, so we can get that to do the work for us. Example 7-31 does the same thing as Exam- ple 7-30, but lets the C# compiler generate the code. Example 7-31. Using an IEnumerable with foreach static string[] AddNumbers(IEnumerable names) { List numberedNames = new List(); int i = 0; foreach (string currentName in names) { numberedNames.Add(string.Format("{0}: {1}", i, currentName)); i += 1; } return numberedNames.ToArray(); } This example only half enters into the spirit of things—it can accept any IEnumera ble, but it stubbornly continues to return an array. This isn’t necessarily a problem; after all, arrays implement IEnumerable. However, our code is a little in- elegant in the way that it creates a List and then converts that into an array at the end. There’s a better way—C# makes it very easy to provide a sequence of objects directly as an IEnumerable. Creating Your Own IEnumerable Before version 2 of C# (which shipped with Visual Studio 2005), writing your own enumerable types was tedious—you had to write a class that implemented IEnumera tor, and that would usually be a separate class from the one that implemented IEnumerable, because multiple enumerators can be active simultaneously for any single collection. It wasn’t hugely tricky, but it was enough of a hassle to put most people off. But C# 2 made it extremely easy to provide enumerations. Example 7-32 shows yet another reworking of the AddNumbers method. Example 7-32. Implementing IEnumerable with yield return static IEnumerable AddNumbers(IEnumerable names) { int i = 0; foreach (string currentName in names) { yield return string.Format("{0}: {1}", i, currentName); i += 1; } } 258 | Chapter 7: Arrays and Lists Instead of using the normal return statement, this method uses yield return. This special form of return statement can only be used inside a method that returns either an enumerable or an enumerator object—you’ll get a compiler error if you try to use it anywhere else. It works rather differently from a normal return. A normal return state- ment indicates that the method has finished, and would like to return control to the caller (returning a value, if the method’s return type was not void). But yield return effectively says: “I want to return this value as an item in the collection, but I might not be done yet—I could have more values to return.” The yield return in Example 7-32 is in the middle of a foreach loop. Whereas a normal return would break out of the loop, in this case the loop is still running, even though the method has returned a value. This leads to some slightly surprising flow of execu- tion. Let’s look at the order in which this code runs. Example 7-33 modifies the AddNumbers method from Example 7-32 by adding a few calls to Console.Writeline, so we can see exactly how the code runs. It also includes a Main method with a foreach loop iterating over the collection returned by AddNumbers, again with some Con sole.WriteLine calls to keep track of what’s going on. Example 7-33. Exploring yield return class Program { static IEnumerable AddNumbers(IEnumerable names) { Console.WriteLine("Starting AddNumbers"); int i = 0; foreach (string currentName in names) { Console.WriteLine("In AddNumbers: " + currentName); yield return string.Format("{0}: {1}", i, currentName); i += 1; } Console.WriteLine("Leaving AddNumbers"); } static void Main(string[] args) { string[] eventNames = { "Swing Dancing at the South Bank", "Saturday Night Swing", "Formula 1 German Grand Prix", "Swing Dance Picnic", "Stompin' at the 100 Club" }; Console.WriteLine("Calling AddNumbers"); IEnumerable numberedNames = AddNumbers(eventNames); Console.WriteLine("Starting main loop"); foreach (string numberedName in numberedNames) { Console.WriteLine("In main loop: " + numberedName); Collections and Polymorphism | 259 } Console.WriteLine("Leaving main loop"); } } Here’s the output: Calling AddNumbers Starting main loop Starting AddNumbers In AddNumbers: Swing Dancing at the South Bank In main loop: 0: Swing Dancing at the South Bank In AddNumbers: Saturday Night Swing In main loop: 1: Saturday Night Swing In AddNumbers: Formula 1 German Grand Prix In main loop: 2: Formula 1 German Grand Prix In AddNumbers: Swing Dance Picnic In main loop: 3: Swing Dance Picnic In AddNumbers: Stompin' at the 100 Club In main loop: 4: Stompin' at the 100 Club Leaving AddNumbers Leaving main loop Even though the main method calls AddNumbers only once, before the start of the loop, you can see from the output that the code flits back and forth between the main loop and AddNumbers for each item in the list. That’s how yield return works—it returns from the method temporarily. Execution will continue from after the yield return as soon as the code consuming the collection asks for the next element. (More precisely, it will happen when the client code calls MoveNext on the enumerator.) C# generates some code that remembers where it had got to on the last yield return so that it can carry on from where it left off. You might be wondering what happens if the consumer abandons the loop halfway through. If that happens, execution will not continue from the yield return. However, as you saw in Example 7-30, code that con- sumes an enumeration should have a using statement to ensure that the enumerator is always disposed of—a foreach loop will always do this for you. The enumerator generated by C# to implement yield return relies on this to ensure that any using or finally blocks inside your enumerator method run correctly even when the enumeration is aban- doned halfway through. This causes a slight wrinkle in the story regarding exception handling. You’ll find that you cannot use yield return inside a try block that is followed by a catch block, for example, because it’s not possible for the C# compiler to guarantee that exceptions will be handled consistently in situations where enumerations are abandoned. 260 | Chapter 7: Arrays and Lists This ability to continue from where we left off as the consumer iterates through the loop illustrates a subtler benefit of yield return: it doesn’t just make the code slightly neater; it lets the code be lazy. Lazy collections The AddNumbers method in Example 7-31 creates all of its output before it returns any- thing. We could describe it as being eager—it does all the work it might need to do right up front. But the modified version in Example 7-32, which uses yield return, is not so eager: it generates items only when it is asked for them, as you can see from the output of Example 7-33. This approach of not doing work until absolutely necessary is often referred to as a lazy style. In fact, if you look closely at the output you’ll see that the AddNumbers method in Example 7-33 is so lazy, it doesn’t seem to run any code at all until we start asking it for items—the Starting AddNumbers message printed out at the beginning of the AddNumbers method (before it starts its foreach loop) doesn’t appear when we call AddNumbers—as you can see, the Starting main loop message appears first, even though Main doesn’t print that out until after AddNumbers returns. This illustrates that none of the code in AddNumbers runs at the point when we call AddNumbers. Nothing happens until we start retrieving elements. Support for lazy collections is the reason that IEnumerable does not provide a Count property. The only way to find out how many items are in an enumeration is to enumerate the whole lot and see how many come out. Enumerable sequences don’t necessarily know how many items they contain until you’ve asked for all the items. Lazy enumeration has some benefits, particularly if you are dealing with very large quantities of information. Lazy enumeration makes it possible to start processing data as soon as the first item becomes available. Example 7-34 illustrates this. Its GetAllFilesInDirectory returns an enumeration that returns all the files in a folder, including all those in any subdirectories. The Main method here uses this to enumerate all the files on the C: drive. (In fact, the Directory class can save us from writing all this code—there’s an overload of Directory.EnumerateFiles that will do a lazy, recursive search for you. But writing our own version is a good way to see how lazy enumeration works.) Example 7-34. Lazy enumeration of a large, slow data set class Program { static IEnumerable GetAllFilesInDirectory(string directoryPath) { IEnumerable files = null; IEnumerable subdirectories = null; try { Collections and Polymorphism | 261 files = Directory.EnumerateFiles(directoryPath); subdirectories = Directory.EnumerateDirectories(directoryPath); } catch (UnauthorizedAccessException) { Console.WriteLine("No permission to access " + directoryPath); } if (files != null) { foreach (string file in files) { yield return file; } } if (subdirectories != null) { foreach (string subdirectory in subdirectories) { foreach (string file in GetAllFilesInDirectory(subdirectory)) { yield return file; } } } } static void Main(string[] args) { foreach (string file in GetAllFilesInDirectory(@"c:\")) { Console.WriteLine(file); } } } If you run this, you’ll find it starts printing out filenames immediately, even though it clearly won’t have had time to discover every single file on the hard disk. (That’s why we’re not using the overload of Directory.GetFiles that recursively searches subdir- ectories for us. As you’ll see in Chapter 8, the Directory class can save us from writing all this code, but it insists on finding all the files before starting to return any of them.) It’s possible to chain enumerations together. For example, we can combine Exam- ple 7-34 with the AddNumbers function, as shown in Example 7-35. Example 7-35. Chaining lazy enumerators together IEnumerable allFiles = GetAllFilesInDirectory(@"c:\"); IEnumerable numberedFiles = AddNumbers(allFiles); foreach (string file in numberedFiles) { Console.WriteLine(file); } 262 | Chapter 7: Arrays and Lists If we’re using the version of AddNumbers from Example 7-32—the one that uses yield return—this will start printing out filenames (with added numbers) immediately. However, if you try it with the version from Example 7-31, you’ll see something quite different. The program will sit there for as many minutes as it takes to find all the filenames on the hard disk—it might print out some messages to indicate that you don’t have permission to access certain folders, but it won’t print out any filenames until it has all of them. And it ends up consuming quite a lot of memory—on my system it uses more than 130 MB of memory, as it builds up a huge List containing all of the filenames, whereas the lazy version makes do with a rather more frugal 7 MB. So in its eagerness to do all of the necessary work up front, Example 7-31 actually slowed us down. It didn’t return any information until it had collected all of the information. Ironically, the lazy version in Example 7-32 enabled us to get to work much faster, and to work more efficiently. This style of enumeration, in which work is done no sooner than nec- essary, is sometimes called deferred execution. While that’s more of a mouthful, it’s probably more fitting in cases where the effect is the op- posite of what lazy suggests. Lazy enumeration also permits an interesting technique whereby infinite loops aren’t necessarily a problem. A method can yield an infinite collection, leaving it up to the caller to decide when to stop. Example 7-36 returns an enumeration of numbers in the Fibonacci series. That’s an infinite series, and since this example uses the BigInteger type introduced in .NET 4, the quantity of numbers it can return is limited only by space and time—the amount of memory in the computer, and the impending heat death of the universe, respectively (or your computer’s next reboot, whichever comes sooner). Example 7-36. An infinite sequence using System.Numerics; // Required for BigInteger ... static IEnumerable Fibonacci() { BigInteger current = 1; BigInteger previous = 1; yield return 1; while (true) { yield return current; BigInteger next = current + previous; previous = current; current = next; } } Collections and Polymorphism | 263 Because consumers of enumerations are free to stop enumerating at any time, in prac- tice this sort of enumeration will just keep going until the calling code decides to stop. We’ll see some slightly more practical uses for this when we explore parallel execution and multithreading later in the book. The concept of chaining lazy enumerations together shown in Example 7-35 is a very useful technique—it’s the basis of the most powerful feature that was added in version 3 of C#: LINQ. LINQ is such an important topic that the next chapter is devoted to it. But before we move on, let’s review what we’ve seen so far. Summary The .NET Framework’s type system has intrinsic support for collections of items in the form of arrays. You can make arrays out of any type. They can be either simple single- dimensional lists, nested arrays of arrays, or multidimensional “rectangular” arrays. The size of an array is fixed at the moment you create it, so when we need a bit more flexibility we use the List generic collection class instead. This works more or less like an array, except we can add and remove items at will. (It uses arrays internally, dynamically allocating new arrays and copying elements across as necessary.) Both arrays and lists offer various services for finding and sorting elements. Thanks to the IEnumerable interface, it’s possible to write polymorphic code that can work with any kind of collection. And as we’re about to see, LINQ takes that idea to a whole new level. 264 | Chapter 7: Arrays and Lists CHAPTER 8 LINQ LINQ, short for Language Integrated Query, provides a powerful set of mechanisms for working with collections of information, along with a convenient syntax. You can use LINQ with the arrays and lists we saw in the previous chapter—anything that implements IEnumerable can be used with LINQ, and there are LINQ providers for databases and XML documents. And even if you have to deal with data that doesn’t fit into any of these categories, LINQ is extensible, so in principle, a provider could be written for more or less any information source that can be accessed from .NET. This chapter will focus mainly on LINQ to Objects—the provider for running queries against objects and collections—but the techniques shown here are applicable to other LINQ sources. Collections of data are ubiquitous, so LINQ can have a profound effect on how you program. Both of your authors have found that LINQ has changed how we write C# in ways we did not anticipate. Pre-LINQ versions of C# now feel like a different and significantly less powerful language. It may take a little while to get your head around how to use LINQ, but it’s absolutely worth the effort. LINQ is not a single language feature—it’s the culmination of several elements that were added to version 3.0 of the C# language and version 3.5 of the .NET Framework. (Despite the different version numbers, these did in fact ship at the same time—they were both part of the Visual Studio 2008 release.) So as well as exploring the most visible aspect of LINQ—the query syntax—we’ll also examine the other associated language and framework features that contribute to LINQ. Query Expressions C# 3.0 added query expressions to the language—these look superficially similar to SQL queries in some respects, but they do not necessarily involve a database. For ex- ample, we could use the data returned by the GetAllFilesInDirectory code from the preceding chapter, reproduced here in Example 8-1. This returns an IEnumera ble containing the filenames of all the files found by recursively searching the 265 specified directory. In fact, as we mentioned in the last chapter, it wasn’t strictly nec- essary to work that hard. We implemented the function by hand to illustrate some details of how lazy evaluation works, but as Example 8-1 shows, we can get the .NET Framework class library to do the work for us. The Directory.EnumerateFiles method still enumerates the files in a lazy fashion when used in this recursive search mode—it works in much the same way as the example we wrote in the previous chapter. Example 8-1. Enumerating filenames static IEnumerable GetAllFilesInDirectory(string directoryPath) { return Directory.EnumerateFiles(directoryPath, "*", SearchOption.AllDirectories); } Since a LINQ query can work with any enumeration of objects, we can write a query that just returns the files larger than, say, 10 million bytes, as shown in Example 8-2. Example 8-2. Using LINQ with an enumeration var bigFiles = from file in GetAllFilesInDirectory(@"c:\") where new FileInfo(file).Length > 10000000 select file; foreach (string file in bigFiles) { Console.WriteLine(file); } As long as the C# file has a using System.Linq; directive at the top (and Visual Studio adds this to new C# files by default) this code will work just fine. Notice that we’ve done nothing special to enable the use of a query here—the GetAllFilesInDirectory method just returns the lazy enumeration provided by the Directory class. And more generally, this sort of query works with anything that implements IEnumerable. Let’s look at the query in more detail. It’s common to assign LINQ query expressions into variables declared with the var keyword, as Example 8-2 does: var bigFiles = ... This tells the compiler that we want it to deduce that variable’s type for us. As it hap- pens, it will be an IEnumerable, and we could have written that explicitly, but as you’ll see shortly, queries sometimes end up using anonymous types, at which point the use of var becomes mandatory. The first part of the query expression itself is always a from clause. This describes the source of information that we want to query, and also defines a so-called range variable: from file in GetAllFilesInDirectory(@"c:\") The source appears on the right, after the in keyword—this query runs on the files returned by the GetAllFilesInDirectory method. The range variable, which appears 266 | Chapter 8: LINQ between the from and in keywords, chooses the name by which we’ll refer to source items in the rest of the query—file in this example. It’s similar to the iteration variable in a foreach loop. The next line in Example 8-2 is a where clause: where new FileInfo(file).Length > 10000000 This is an optional, although very common, LINQ query feature. It acts as a filter— only items for which the expression is true will be present in the results of the query. This clause constructs a FileInfo object for the file, and then looks at its Length property so that the query only returns files that are larger than the specified size. The final part of the query describes what information we want to come out of the query, and it must be either a select or a group clause. Example 8-2 uses a select clause: select file; This is a trivial select clause—it just selects the range variable, which contains the filename. That’s why this particular query ends up producing an IEnumera ble. But we can put other expressions in here—for example, we could write: select File.ReadAllLines(file).Length; This uses the File class (defined in System.IO) to read the file’s text into an array with one element per line, and then retrieves that array’s Length. This would make the query return an IEnumerable, containing the number of lines in each file. You may be wondering exactly how this works. The code in a LINQ query expression looks quite different from most other C# code—it is, by design, somewhat reminiscent of database queries. But it turns out that all that syntax turns into straightforward method calls. Query Expressions Versus Method Calls The C# language specification defines a process by which all LINQ query expressions are converted into method invocations. Example 8-3 shows what the query expression in Example 8-2 turns into. Incidentally, C# ignores whitespace on either side of the . syntax for member access, so the fact that this example has been split across multiple lines to fit on the page doesn’t stop it from compiling. Example 8-3. LINQ query as method calls var bigFiles = GetAllFilesInDirectory(@"c:\"). Where(file => new FileInfo(file).Length > 10000000); Let’s compare this with the components of the original query: var bigFiles = from file in GetAllFilesInDirectory(@"c:\") where new FileInfo(file).Length > 10000000 select file; Query Expressions | 267 The source, which follows the in keyword in the query expression, becomes the starting point—that’s the enumeration returned by GetAllFilesInDirectory in this case. The next step is determined by the presence of the where clause—this turns into a call to the Where method on the source enumeration. As you can see, the condition in the where clause has turned into a lambda expression, passed as an argument to the Where method. The final select clause has turned into...nothing! That’s because it’s a trivial select— it just selects the range variable and nothing else, in which case there’s no need to do any further processing of the information that comes out of the Where method. If we’d had a slightly more interesting expression in the select clause, for example: var bigFiles = from file in GetAllFilesInDirectory(@"c:\") where new FileInfo(file).Length > 10000000 select "File: " + file; we would have seen a corresponding Select method in the equivalent function calls, as Example 8-4 shows. Example 8-4. Where and Select as methods var bigFiles = GetAllFilesInDirectory(@"c:\"). Where(file => new FileInfo(file).Length > 10000000). Select(file => "File: " + file); A question remains, though: where did the Where and Select methods here come from? GetAllFilesInDirectory returns an IEnumerable, and if you examine this in- terface (which we showed in the preceding chapter) you’ll see that it doesn’t define a Where method. And yet if you try these method-based equivalents of the query expres- sions, you’ll find that they compile just fine as long as you have a using System.Linq; directive at the top of the file, and a project reference to the System.Core library. What’s going on? The answer is that Where and Select in these examples are extension methods. Extension Methods and LINQ One of the language features added to C# 3.0 for LINQ is support for extension meth- ods. These are methods bolted onto a type by some other type. You can add new meth- ods to an existing type, even if you can’t change that type—perhaps it’s a type built into the .NET Framework. For example, the built-in string type is not something we get to change, and it’s sealed, so we cannot derive from it either, but that doesn’t stop us from adding new methods. Example 8-5 adds a new and not very useful Backwards method that returns a copy of the string with the characters in reverse order.* * This is even less useful than it sounds. If the string in question contains characters that are required to be used in strict sequence, such as combining characters or surrogates, naively reversing the character order will have peculiar results. But the point here is to illustrate how to add new methods to an existing type, not to explain why it’s surprisingly difficult to reverse a Unicode string. 268 | Chapter 8: LINQ Example 8-5. Adding an extension method to string static class StringAdditions { // Naive implementation for illustrative purposes. // DO NOT USE in real code! public static string Backwards(this string input) { char[] characters = input.ToCharArray(); Array.Reverse(characters); return new string(characters); } } Notice the this keyword in front of the first argument—that indicates that Backwards is an extension method. Also notice that the class is marked as static—you can only define extension methods in static classes. As long as this class is in a namespace that’s in scope (either because of a using directive, or because it’s in the same namespace as the code that wants to use it) you can call this method as though it were a normal member of the string class: string stationName = "Finsbury Park"; Console.WriteLine(stationName.Backwards()); The Where and Select methods used in Example 8-4 are extension methods. The System.Linq namespace defines a static class called Enumerable which defines these and numerous other extension methods for IEnumerable. Here’s the signature for one of the Where overloads: public static IEnumerable Where( this IEnumerable source, Func predicate) Notice that this is a generic method—the method itself takes a type argument, called TSource here, and passes that through as the type argument T for the first parameter’s IEnumerable. The result is that this method extends IEnumerable, whatever T may be. In other words, as long as the System.Linq namespace is in scope, all IEnumera ble implementations appear to offer a Where method. Select and Where are examples of LINQ operators—standard methods that are available wherever LINQ is supported. The Enumerable class in System.Linq provides all the LINQ operators for IEnumerable, but is not the only LINQ provider—it just provides query support for collections in memory, and is sometimes referred to as LINQ to Objects. In later chapters, we’ll see sources that support LINQ queries against data- bases and XML documents. Anyone can write a new provider, because C# neither knows nor cares what the source is or how it works—it just mechanically translates query expressions into method calls, and as long as the relevant LINQ operators are available, it will use them. This leaves different data sources free to implement the various operators in whatever way they see fit. Example 8-6 shows how you could exploit this to provide custom implementations of the Select and Where operators. Query Expressions | 269 Example 8-6. Custom implementation of some LINQ operators public class Foo { public string Name { get; set; } public Foo Where(Func predicate) { return this; } public TResult Select(Func selector) { return selector(this); } } These are normal methods rather than extension methods—we’re writing a custom type, so we can add LINQ operators directly to that type. Since C# just converts LINQ queries into method calls, it doesn’t matter whether LINQ operators are normal meth- ods or extension methods. So with these methods in place, we could write the code shown in Example 8-7. Example 8-7. Confusing but technically permissible use of a LINQ query Foo source = new Foo { Name = "Fred" }; var result = from f in source where f.Name == "Fred" select f.Name; C# will follow the rules for translating query expressions into method calls, just as it would for any query, so it will turn Example 8-7 into this: Foo source = new Foo { Name = "Fred" }; var result = source.Where(f => f.Name == "Fred").Select(f => f.Name); Since the Foo class provides the Where and Select operators that C# expects, this will compile and run. It won’t be particularly useful, because our Where implementation completely ignores the predicate. And it’s also a slightly bizarre thing to do—our Foo class doesn’t appear to represent any kind of collection, so it’s rather misleading to use syntax that’s intended to be used with collections. In fact, Example 8-7 has the same effect as: var result = source.Name; So you’d never write code like Example 8-6 and Example 8-7 for a type as simple as Foo in practice—the purpose of these examples is to illustrate that the C# compiler blindly translates query expressions into method calls, and has no understanding or expectation of what those calls might do. The real functionality of LINQ lives entirely in the class library. Query expressions are just a convenient syntax. 270 | Chapter 8: LINQ let Clauses Query expressions can contain let clauses. This is an interesting kind of clause in that unlike most of the rest of a query, it doesn’t correspond directly to any particular LINQ operator. It’s just a way of making it easier to structure your query. You would use a let clause when you need to use the same information in more than one place in a query. For example, suppose we want to modify the query in Exam- ple 8-2 to return a FileInfo object, rather than a filename. We could do this: var bigFiles = from file in GetAllFilesInDirectory(@"c:\") where new FileInfo(file).Length > 10000000 select new FileInfo(file); But this code repeats itself—it creates a FileInfo object in the where clause and then creates another one in the select clause. We can avoid this repetition with a let clause: var bigFiles = from file in GetAllFilesInDirectory(@"c:\") let info = new FileInfo(file) where info.Length > 10000000 select info; The C# compiler jumps through some significant hoops to make this work. There’s no need to know the details to make use of a let clause, but if you’re curious to know how it works, here’s what happens. Under the covers it generates a class containing two properties called file and info, and ends up generating two queries: var temp = from file in GetAllFilesInDirectory(@"c:\") select new CompilerGeneratedType(file, new FileInfo(file)); var bigFiles = from item in temp where item.info.Length > 10000000 select item.info; The purpose of the first query is to produce a sequence in which the range variable is wrapped in the compiler-generated type, alongside any variables declared with a let clause. (It’s not actually called CompilerGeneratedType, of course—the compiler gen- erates a unique, meaningless name.) This allows all these variables to be available in all the clauses of the query. LINQ Concepts and Techniques Before we look in detail at the services LINQ offers, there are some features that apply across all of LINQ that you should be aware of. Delegates and Lambdas LINQ query syntax makes implicit use of lambdas. The expressions that appear in where, select, or most other clauses are written as ordinary expressions, but as you’ve seen, the C# compiler turns queries into a series of method calls, and the expressions become lambda expressions. LINQ Concepts and Techniques | 271 Most of the time, you can just write the expressions you need and they work. But you need to be wary of code that has side effects. For example, it would be a bad idea to write the sort of query shown in Example 8-8. Example 8-8. Unhelpful side effects in a query int x = 10000; var bigFiles = from file in GetAllFilesInDirectory(@"c:\") where new FileInfo(file).Length > x++ select file; The where clause here increments a variable declared outside the scope of the query. This is allowed (although it’s a bad idea) in LINQ to Objects. Some LINQ providers, such as the ones you would use with databases, will reject such a query at runtime. This will have the potentially surprising result that the query could return different files every time it runs, even if the underlying data has not changed. Remember, the ex- pression in the where clause gets converted into an anonymous method, which will be invoked once for every item in the query’s source. The first time this runs, the local x variable will be incremented once for every file on the disk. If the query is executed again, that’ll happen again—nothing will reset x to its original state. Moreover, queries are often executed sometime after the point at which they are cre- ated, which can make code with side effects very hard to follow—looking at the code in Example 8-8 it’s not possible to say exactly when x will be modified. We’d need more context to know that—when exactly is the bigFiles query evaluated? How many times? In practice, it is important to avoid side effects in queries. This extends beyond simple things such as the ++ operator—you also need to be careful about invoking methods from within a query expression. You’ll want to avoid methods that change the state of your application. It’s usually OK for expressions in a query to read variables from the surrounding scope, though. A small modification to Example 8-8 illustrates one way you could exploit this (see Example 8-9). Example 8-9. Using a local variable in a query int minSize = 10000; var bigFiles = from file in GetAllFilesInDirectory(@"c:\") where new FileInfo(file).Length > minSize select file; var filesOver10k = bigFiles.ToArray(); minSize = 100000; var filesOver100k = bigFiles.ToArray(); minSize = 1000000; 272 | Chapter 8: LINQ var filesOver1MB = bigFiles.ToArray(); minSize = 10000000; var filesOver10MB = bigFiles.ToArray(); This query makes use of a local variable as before, but this query simply reads the value rather than modifying it. By changing the value of that variable, we can modify how the query behaves the next time it is evaluated. (The call to ToArray() executes the query and puts the results into an array. This is one way of forcing an immediate execution of the query.) Functional Style and Composition LINQ operators all share a common characteristic: they do not modify the data they work on. For example, you can get LINQ to sort the results of a query, but unlike Array.Sort or List.Sort, which both modify the order of an existing collection, sorting in LINQ works by producing a new IEnumerable which returns objects in the specified order. The original collection is not modified. This is similar in style to .NET’s string type. The string class provides various methods that look like they will modify the string, such as Trim, ToUpper, and Replace. But strings are immutable, so all of these methods work by building a new string—you get a modi- fied copy, leaving the original intact. LINQ never tries to modify sources, so it’s able to work with immutable sources. LINQ to Objects relies on IEnumerable, which does not provide any mechanism for mod- ifying the contents or order of the underlying collection. Of course, LINQ does not require sources to be immutable. IEnumera ble can be implemented by modifiable and immutable classes alike. The point is that LINQ will never attempt to modify its source collections. This approach is sometimes described as a functional style. Functional programming languages such as F# tend to have this characteristic—just as mathematical functions such as addition, multiplication, and trigonometric functions do not modify their in- puts, neither does purely functional code. Instead, it generates new information based on its inputs—new enumerations layered on top of input enumerations in the case of LINQ. C# is not a purely functional language—it’s possible and indeed common to write code that modifies things—but that doesn’t stop you from using a functional style, as LINQ shows. Functional code is often highly composable—it tends to lead to APIs whose features can easily be combined in all sorts of different ways. This in turn can lead to more maintainable code—small, simple features are easier to design, develop, and test than LINQ Concepts and Techniques | 273 complex, monolithic chunks of code, but you can still tackle complex problems by combining smaller features. Since LINQ works by passing a sequence to a method that transforms its input into a new sequence, you can plug together as many LINQ oper- ators as you like. The fact that these operators never modify their inputs simplifies things. If multiple pieces of code are all vying to modify some data, it can become difficult to ensure that your program behaves correctly. But with a functional style, once data is produced it never changes—new calculations yield new data instead of modifying existing data. If you can be sure that some piece of data will never change, it becomes much easier to understand your code’s behavior, and you’ll have a better chance of making it work. This is especially important with multithreaded code. Deferred Execution Chapter 7 introduced the idea of lazy enumeration (or deferred execution, as it’s also sometimes called). As we saw, iterating over an enumeration such as the one returned by GetAllFilesInDirectory does the necessary work one element at a time, rather than processing everything up front. The query in Example 8-2 preserves this characteristic—if you run the code, you won’t have to wait for GetAllFilesInDirec tory to finish before you see any results; it will start printing filenames immediately. (Well, almost immediately—it depends on how far it has to look before finding a file large enough to get through the where clause.) And in general, LINQ queries will defer work as much as possible—merely having executed the code that defines the query doesn’t actually do anything. So in our example, this code: var bigFiles = from file in GetAllFilesInDirectory(@"c:\") where new FileInfo(file).Length > 10000000 select file; does nothing more than describe the query. No work is done until we start to enumerate the bigFiles result with a foreach loop. And at each iteration of that loop, it does the minimum work required to get the next item—this might involve retrieving multiple results from the underlying collection, because the where clause will keep fetching items until it either runs out or finds one that matches the condition. But even so, it does no more work than necessary. The picture may change a little as you use some of the more advanced features described later in this chapter—for example, you can tell a LINQ query to sort your data, in which case it will probably have to look at all the results before it can work out the correct order. (Although even that’s not a given—it’s possible to write a source that knows all about ordering, and if you have special knowledge about your data source, it may be possible to write a source that delivers data in order while still fetching items lazily. We’ll see providers that do this when we look at how to use LINQ with databases in a later chapter.) 274 | Chapter 8: LINQ Although deferred execution is almost always a good thing, there’s one gotcha to bear in mind. Because the query doesn’t run up front, it will run every time you evaluate it. LINQ doesn’t keep a copy of the results when you execute the query, and there are good reasons you wouldn’t want it to—it could consume a lot of memory, and would prevent you from using the technique in Example 8-9. But it does mean that relatively innocuous-looking code can turn out to be quite expensive, particularly if you’re using a LINQ provider for a database. Inadvertently evaluating the query multiple times could cause multiple trips to the database server. LINQ Operators There are around 50 standard LINQ operators. The rest of this chapter describes the most important operators, broken down by the main areas of functionality. We’ll show how to use them both from a query expression (where possible) and with an explicit method call. Sometimes it’s useful to call the LINQ query operator methods explic- itly, rather than writing a query expression. Some operators offer over- loads with advanced features that are not available in a query expression. For example, sorting strings is a locale-dependent operation—there are variations on what constitutes alphabetical ordering in different lan- guages. The query expression syntax for ordering data always uses the current thread’s default culture for ordering. If you need to use a dif- ferent culture for some reason, or you want a culture-independent order, you’ll need to call an overload of the OrderBy operator explicitly instead of using an orderby clause in a query expression. There are even some LINQ operators that don’t have an equivalent in a query expression. So understanding how LINQ uses methods is not just a case of looking at implementation details. It’s the only way to access some more advanced LINQ features. Filtering You already saw the main filtering feature of LINQ. We illustrated the where clause and the corresponding Where operator in Example 8-2 and Example 8-3, respectively. An- other filter operator worth being aware of is called OfType. It has no query expression equivalent, so you can use it only with a method call. OfType is useful when you have a collection that could contain a mixture of types, and you only want to look at the elements that have a particular type. For example, in a user interface you might want to get hold of control elements (such as buttons), ignoring purely visual elements such as images or drawings. You could write this sort of code: var controls = myPanel.Children.OfType(); LINQ Operators | 275 If myPanel.Children is a collection of objects of some kind, this code will ensure that controls is an enumeration that only returns objects that can be cast to the Control type. Although OfType has no equivalent in a query expression, that doesn’t stop you from using it in conjunction with a query expression—you can use the result of OfType as the source for a query: var controlNames = from control in myPanel.Children.OfType() where !string.IsNullOrEmpty(control.Name) select control.Name; This uses the OfType operator to filter the items down to objects of type Control, and then uses a where clause to further filter the items to just those with a nonempty Name property. Ordering Query expressions can contain an orderby clause, indicating the order in which you’d like the items to emerge from the query. In queries with no orderby clause, LINQ does not, in general, make any guarantees about the order in which items emerge. LINQ to Objects happens to return items in the order in which they emerge from the source enumeration if you don’t specify an order, but other LINQ providers will not necessarily define a default order. (In particular, database LINQ providers typically return items in an unpredictable order unless you explicitly specify an order.) So as to have some data to sort, Example 8-10 brings back the CalendarEvent class from Chapter 7. Example 8-10. Class representing a calendar event class CalendarEvent { public string Title { get; set; } public DateTimeOffset StartTime { get; set; } public TimeSpan Duration { get; set; } } When examples in this chapter refer to an events variable, assume that it was initialized with the data shown in Example 8-11. Example 8-11. Some example data List events = new List { new CalendarEvent { Title = "Swing Dancing at the South Bank", StartTime = new DateTimeOffset (2009, 7, 11, 15, 00, 00, TimeSpan.Zero), Duration = TimeSpan.FromHours(4) }, new CalendarEvent { 276 | Chapter 8: LINQ Title = "Saturday Night Swing", StartTime = new DateTimeOffset (2009, 7, 11, 19, 30, 00, TimeSpan.Zero), Duration = TimeSpan.FromHours(6.5) }, new CalendarEvent { Title = "Formula 1 German Grand Prix", StartTime = new DateTimeOffset (2009, 7, 12, 12, 10, 00, TimeSpan.Zero), Duration = TimeSpan.FromHours(3) }, new CalendarEvent { Title = "Swing Dance Picnic", StartTime = new DateTimeOffset (2009, 7, 12, 15, 00, 00, TimeSpan.Zero), Duration = TimeSpan.FromHours(4) }, new CalendarEvent { Title = "Stompin' at the 100 Club", StartTime = new DateTimeOffset (2009, 7, 13, 19, 45, 00, TimeSpan.Zero), Duration = TimeSpan.FromHours(5) } }; Example 8-12 shows a LINQ query that orders these events by start time. Example 8-12. Ordering items with LINQ var eventsByStartTime = from ev in events orderby ev.StartTime select ev; By default, the items will be sorted into ascending order. You can be explicit about this if you like: var eventsByStartTime = from ev in events orderby ev.StartTime ascending select ev; And, of course, you can sort into descending order too: var eventsByStartTime = from ev in events orderby ev.StartTime descending select ev; The expression in the orderby clause does not need to correspond directly to a property of the source object. It can be a more complex expression. For example, we could extract just the time of day to produce the slightly confusing result of events ordered by what time they start, regardless of date: var eventsByStartTime = from ev in events orderby ev.StartTime.TimeOfDay select ev; You can specify multiple criteria. Example 8-13 sorts the events: first by date (ignoring the time) and then by duration. LINQ Operators | 277 Example 8-13. Multiple sort criteria var eventsByStartDateThenDuration = from ev in events orderby ev.StartTime.Date, ev.Duration select ev; Four LINQ query operator methods correspond to the orderby clause. Most obviously, there’s OrderBy, which takes a single ordering criterion as a lambda: var eventsByStartTime = events.OrderBy(ev => ev.StartTime); That code has exactly the same effect as Example 8-12. Of course, like most LINQ operators, you can chain this together with other ones. So we could combine that with the Where operator: var longEvents = events.OrderBy(ev => ev.StartTime). Where(ev => ev.Duration > TimeSpan.FromHours(2)); This is equivalent to the following query: var longEvents = from ev in events orderby ev.StartTime where ev.Duration > TimeSpan.FromHours(2) select ev; You can customize the comparison mechanism used to sort the items by using an over- load that accepts a comparison object—it must implement IComparer† where TKey is the type returned by the ordering expression. So in these examples, it would need to be an IComparer, since that’s the type of the StartTime prop- erty we’re using to order the data. There’s not a lot of scope for discussion about what order dates come in, so this is not a useful example for plugging in an alternate com- parison. However, string comparisons do vary a lot—different languages have different ideas about what order letters come in, particularly when it comes to letters with ac- cents. The .NET Framework class library offers a StringComparer class that can provide an IComparer implementation for any language and culture supported in .NET. The following example uses this in conjunction with an overload of the OrderBy oper- ator to sort the events by their title, using a string sorting order appropriate for the French-speaking Canadian culture, and configured for case insensitivity: CultureInfo cult = new CultureInfo("fr-CA"); // 2nd argument is true for case insensitivity StringComparer comp = StringComparer.Create(cult, true); var eventsByTitle = events.OrderBy(ev => ev.Title, comp); There is no equivalent query expression—if you want to use anything other than the default comparison for a type, you must use this overload of the OrderBy operator. † This is very similar to IComparable, introduced in the preceding chapter. But while objects that implement IComparable can themselves be compared with other objects of type T, an IComparer compares two objects of type T—the objects being compared are separate from the comparer. 278 | Chapter 8: LINQ The OrderBy operator method always sorts in ascending order. To sort in descending order, there’s an OrderByDescending operator. If you want to use multiple sort criteria, as in Example 8-13, a different operator comes into play: you need to use either ThenBy or ThenByDescending. This is because the OrderBy and OrderByDescending operators discard the order of incoming elements and impose the specified order from scratch—that’s the whole point of those operators. Refining an ordering by adding further sort criteria is a different kind of operation, hence the different operators. So the method-based equivalent of Example 8-13 would look like this: var eventsByStartTime = events.OrderBy(ev => ev.StartTime). ThenBy(ev => ev.Duration); Ordering will cause LINQ to Objects to iterate through the whole source collection before returning any elements—it can only sort items once it has seen all of the items. Concatenation Sometimes you’ll end up wanting to combine two sequences of values into one. LINQ provides a very straightforward operator for this: Concat. There is no equivalent in the query expression syntax. If you wanted to combine two lists of events into one, you would use the code in Example 8-14. Example 8-14. Concatenating two sequences var allEvents = existingEvents.Concat(newEvents); Note that this does not modify the inputs. This builds a new enumeration object that returns all the elements from existingEvents, followed by all the elements from newEvents. So this can be safer than the List.AddRange method shown in Chap- ter 7, because this doesn’t modify anything. (Conversely, if you were expecting Exam- ple 8-14 to modify existingEvents, you will be disappointed.) This is a good illustration of how LINQ uses the functional style descri- bed earlier. Like mathematical functions, most LINQ operators calculate their outputs without modifying their inputs. For example, if you have two int variables called x and y, you would expect to be able to calculate x+y without that calculation changing either x or y. Con- catenation works the same way—you can produce a sequence that is the concatenation of two inputs without changing those inputs. As with most LINQ operators, concatenation uses deferred evaluation—it doesn’t start asking its source enumerations for elements in advance. Only when you start to iterate through the contents of allEvents will this start retrieving items from existingEvents. (And it won’t start asking for anything from newEvents until it has retrieved all the elements from existingEvents.) LINQ Operators | 279 Grouping LINQ provides the ability to take flat lists of data and group them. As Example 8-15 shows, we could use this to write a LINQ-based alternative to the GetEventsByDay method shown in Chapter 7. Example 8-15. Simple LINQ grouping var eventsByDay = from ev in events group ev by ev.StartTime.Date; This will arrange the objects in the events source into one group for each day. The eventsByDay variable here ends up with a slightly different type than anything we’ve seen before. It’s an IEnumerable>. So eventsByDay is an enumeration, and it returns an item for each group found by the group clause. Example 8-16 shows one way of using this. It iterates through the collec- tion of groupings, and for each grouping it displays the Key property—the value by which the items have been grouped—and then iterates through the items in the group. Example 8-16. Iterating through grouped results foreach (var day in eventsByDay) { Console.WriteLine("Events for " + day.Key); foreach (var item in day) { Console.WriteLine(item.Title); } } This produces the following output: Events for 7/11/2009 12:00:00 AM Swing Dancing at the South Bank Saturday Night Swing Events for 7/12/2009 12:00:00 AM Formula 1 German Grand Prix Swing Dance Picnic Events for 7/13/2009 12:00:00 AM Stompin' at the 100 Club This illustrates that the query in Example 8-15 has successfully grouped the events by day, but let’s look at what returned in a little more detail. Each group is represented as an IGrouping, where TKey is the type of the expression used to group the data—a DateTimeOffset in this case—and TElement is the type of the elements making up the groups—CalendarEvent in this example. IGrouping derives from IEnumerable, so you can enumerate through the contents of a group like you would any other enumeration. (In fact, the only thing IGrouping adds is the Key property, which is the grouping value.) So the query in Ex- ample 8-15 returns a sequence of sequences—one for each group (see Figure 8-1). 280 | Chapter 8: LINQ While a LINQ query expression is allowed to end with a group clause, as Exam- ple 8-15 does, it doesn’t have to finish there. If you would like to do further processing, you can add an into keyword on the end, followed by an identifier. The continuation of the query after a group ... into clause will iterate over the groups, and the identifier effectively becomes a new range variable. Example 8-17 uses this to convert each group into an array. (Calling ToArray on an IGrouping effectively discards the Key, and leaves you with just an array containing that group’s contents. So this query ends up producing an IEnumerable—a collection of arrays.) Figure 8-1. Result of groupby query LINQ Operators | 281 Example 8-17. Continuing a grouped query with into var eventsByDay = from ev in events group ev by ev.StartTime.Date into dayGroup select dayGroup.ToArray(); Like the ordering operators, grouping will cause LINQ to Objects to evaluate the whole source sequence before returning any results. Projections The select clause’s job is to define how each item should look when it comes out of the query. The official (if somewhat stuffy) term for this is projection. The simplest possible kind of projection just leaves the items as they are, as shown in Example 8-18. Example 8-18. Trivial projection var projected = from ev in events select ev; Earlier, you saw this kind of trivial select clause collapsing away to nothing. However, that doesn’t happen here, because this is what’s called a degenerate query—it contains nothing but a trivial projection. (Example 8-2 was different, because it contained a where clause in addition to the trivial select.) LINQ never reduces a query down to nothing at all, so when faced with a degenerate query, it leaves the trivial select in place, even though it appears to have nothing to do. So Example 8-18 becomes a call to the Select LINQ operator method: var projected = events.Select(ev => ev); But projections often have work to do. For example, if we want to pick out event titles, we can write this: var projected = from ev in events select ev.Title; Again, this becomes a call to the Select LINQ operator method, with a slightly more interesting projection lambda: var projected = events.Select(ev => ev.Title); We can also calculate new values in the select clause. This calculates the end time of the events: var projected = from ev in events select ev.StartTime + ev.Duration; You can use any expression you like in the select clause. In fact, there’s not even any obligation to use the range variable, although it’s likely to be a bit of a waste of time to construct a query against a data source if you ultimately don’t use any data from that source. But C# doesn’t care—any expression is allowed. The following slightly silly 282 | Chapter 8: LINQ code generates one random number for each event, in a way that is entirely unrelated to the event in question: Random r = new Random(); var projected = from ev in events select r.Next(); You can, of course, construct a new object in the select clause. There’s one interesting variation on this that often crops up in LINQ queries, which occurs when you want the query to return multiple pieces of information for each item. For example, we might want to display calendar events in a format where we show both the start and the end times. This is slightly different from how the CalendarEvent class represents things—it stores the duration rather than the end time. We could easily write a query that calcu- lates the end time, but it wouldn’t be very useful to have just that time. We’d want all the details—the title, the start time, and the end time. In other words, we’d be transforming the data slightly. We’d be taking a stream of objects where each item contains Title, StartTime, and Duration properties, and pro- ducing one where each item contains a Title, StartTime, and EndTime. Example 8-19 does exactly this. Example 8-19. Select clause with anonymous type var projected = from ev in events select new { Title = ev.Title, StartTime = ev.StartTime, EndTime = ev.StartTime + ev.Duration }; This constructs a new object for each item. But while the new keyword is there, notice that we’ve not specified the name of a type. All we have is the object initialization syntax to populate various properties—the list of values in braces after the new keyword. We haven’t even defined a type anywhere in these examples that has a Title, a StartTime, and an EndTime property. And yet this compiles. And we can go on to use the results as shown in Example 8-20. Example 8-20. Using a collection with an anonymous item type foreach (var item in projected) { Console.WriteLine("Event {0} starts at {1} and ends at {2}", item.Title, item.StartTime, item.EndTime); } These two examples are using the anonymous type feature added in C# 3.0. LINQ Operators | 283 Anonymous types If we want to define a type to represent some information in our application, we would normally use the class or struct keyword as described in Chapter 3. Typically, the type definition would live in its own source file, and in a real project we would want to devise unit tests to ensure that it works as expected. This might be enough to put you off the idea of defining a type for use in a very narrow context, such as having a convenient container for the information coming out of a query. But it’s often useful for the select clause of a query just to pick out a few properties from the source items, possibly transforming the data in some way to get it into a convenient representation. Extracting just the properties you need can become important when using LINQ with a database—database providers are typically able to transform the projection into an equivalent SQL SELECT statement. But if your LINQ query just fetches the whole row, it will end up fetching every column whether you need it or not, placing an unnecessary extra load on the database and network. There’s a trade-off here. Is the effort of creating a type worth the benefits if you’re only going to use it to hold the results of a query? If your code immediately does further processing of the data, the type will be useful to only a handful of lines of code. But if you don’t create the type, you have to deal with a compromise—you might not be able to structure the information coming out of your query in exactly the way you want. C# 3.0 shifts the balance in favor of creating a type in this scenario, by removing most of the effort required, thanks to anonymous types. This is another language feature added mainly for the benefit of LINQ, although you can use it in other scenarios if you find it useful. An anonymous type is one that the C# compiler writes for you, based on the properties in the object initializer list. So when the compiler sees this expression from Example 8-19: new { Title = ev.Title, StartTime = ev.StartTime, EndTime = ev.StartTime + ev.Duration }; it knows that it needs to supply a type, because we’ve not specified a type name after the new keyword. It will create a new class definition, and will define properties for each entry in the initializer. It will work out what types the properties should have from the types of the expressions in the initializer. For example, the ev.Title expression evalu- ates to a string, so it will add a property called Title of type string. 284 | Chapter 8: LINQ Before generating a new anonymous type, the C# compiler checks to see if it has already generated one with properties of the same name and type, specified in the same order elsewhere in your project. If it has, it just reuses that type. So if different parts of your code happen to end up creating identical anonymous types, the compiler is smart enough to share the type definition. (Normally, the order in which properties are defined has no significance, but in the case of anonymous types, C# considers two types to be equivalent only if the properties were specified in the same order.) The nice thing about this is that when we come to use the items in a collection based on an anonymous type (such as in Example 8-20) IntelliSense and compile-time check- ing work exactly as they always do—it’s just like working with a normal type, but we didn’t have to write it. From the point of view of the .NET Framework, the type generated by the C# compiler is a perfectly ordinary type like any other. It neither knows nor cares that the compiler wrote the class for us. It’s anonymous only from the point of view of our C# code— the generated type does in fact have a name, it’s just a slightly odd-looking one. It’ll be something like this: <>f__AnonymousType0`3 The C# compiler deliberately picks a name for the type that would be illegal as a C# class name (but which is still legal as far as .NET is concerned) in order to stop us from trying to use the class by its name—that would be a bad thing to do, because the compiler doesn’t guarantee to keep the name the same from one compilation to the next. The anonymity of the type name means that anonymous types are only any use within a single method. Suppose you wanted to return an anonymous type (or an IEnumera ble) from a method—what would you write as the return type if the type in question has no name? You could use Object, but the properties of the anonymous type won’t be visible. The best you could do is use dynamic, which we describe in Chapter 18. This would make it possible to access the properties, but with- out the aid of compile-time type checking or IntelliSense. So the main purpose of anonymous types is simply to provide a convenient way to get information from a query to code later in the same method that does something with that information. Anonymous types would not be very useful without the var keyword, another feature introduced in C# 3.0. As we saw earlier, when you declare a local variable with the var keyword, the compiler works out the type from the expression you use to initialize the variable. To see why we need this for anonymous types to be useful, look at Ex- ample 8-19—how would you declare the projected local variable if we weren’t using var? It’s going to be some sort of IEnumerable, but what’s T here? It’s an anonymous type, so by definition we can’t write down its name. It’s interesting to see how Visual LINQ Operators | 285 Studio reacts if we ask it to show us the type by hovering our mouse pointer over the variable—Figure 8-2 shows the resultant data tip. Visual Studio chooses to denote anonymous types with names such as 'a, 'b, and so forth. These are not legal names—they’re just placeholders, and the data tip pop up goes on to show the structure of the anonymous types they represent. Whether or not you’re using anonymous types in your projections, there’s an alterna- tive form of projection that you will sometimes find useful when dealing with multiple sources. Figure 8-2. How Visual Studio shows anonymous types Using multiple sources Earlier, Example 8-15 used a groupby clause to add some structure to a list of events— the result was a list containing one group per day, with each group itself containing a list of events. Sometimes it can be useful to go in the opposite direction—you may have structured information that you would like to flatten into a single list. You can do this in a query expression by writing multiple from clauses, as Example 8-21 shows. Example 8-21. Flattening lists using multiple from clauses var items = from day in eventsByday from item in day select item; You can think of this as having roughly the same effect as the following code: List items = new List(); foreach (IGrouping day in eventsByDay) { foreach (CalendarEvent item in day) { items.Add(item); } } That’s not exactly how it works, because the LINQ query will use deferred execution— it won’t start iterating through the source items until you start trying to iterate through the query. The foreach loops, on the other hand, are eager—they build the entire flat- tened list as soon as they run. But lazy versus eager aside, the set of items produced is the same—for each item in the first source, every item in the second source will be processed. 286 | Chapter 8: LINQ Notice that this is very different from the concatenation operator shown earlier. That also works with two sources, but it simply returns all the items in the first source, followed by all the items in the second source. But Example 8-21 will iterate through the source of the second from clause once for every item in the source of the first from clause. (So con- catenation and flattening are as different as addition and multiplica- tion.) Moreover, the second from clause’s source expression typically evaluates to a different result each time around. In Example 8-21, the second from clause uses the range variable from the first from clause as its source. This is a common technique—it’s what enables this style of query to flatten a grouped structure. But it’s not mandatory—you can use any LINQ-capable source you like; for example, any IEnumerable. Example 8-22 uses the same source array for both from clauses. Example 8-22. Alternative use of multiple from clauses int[] numbers = { 1, 2, 3, 4, 5 }; var multiplied = from x in numbers from y in numbers select x * y; foreach (int n in multiplied) { Console.WriteLine(n); } The source contains five numbers, so the resultant multiplied sequence contains 25 elements—the second from clause counts through all five numbers for each time around the first from clause. The LINQ operator method for flattening multiple sources is called SelectMany. The equivalent of Example 8-22 looks like this: var multiplied = numbers.SelectMany( x => numbers, (x, y) => x * y); The first lambda is expected to return the collection over which the nested iteration will be performed—the collection for the second from clause in the LINQ query. The second lambda is the projection from the select clause in the query. In queries with a trivial final projection, a simpler form is used, so the equivalent of Example 8-21 is: var items = days.SelectMany(day => day); Whether you’re using a multisource SelectMany or a simple single-source projection, there’s a useful variant that lets your projection know each item’s position, by passing a number into the projection. LINQ Operators | 287 Numbering items The Select and SelectMany LINQ operators both offer overloads that make it easy to number items. Example 8-23 uses this to build a list of numbered event names. Example 8-23. Adding item numbers var numberedEvents = events. Select((ev, i) => string.Format("{0}: {1}", i + 1, ev.Title)); If we iterate over this, printing out each item: foreach (string item in numberedEvents) { Console.WriteLine(item); } the results look like this: 1: Swing Dancing at the South Bank 2: Formula 1 German Grand Prix 3: Swing Dance Picnic 4: Saturday Night Swing 5: Stompin' at the 100 Club This illustrates how LINQ often makes for much more concise code than was possible before C# 3.0. Remember that in Chapter 7, we wrote a function that takes an array of strings and adds a number in a similar fashion. That required a loop with several lines of code, and it worked only if we already happened to have a collection of strings. Here we’ve turned a collection of CalendarEvents into a collection of numbered event titles with just a single method call. As you get to learn LINQ, you’ll find this happens quite a lot—situations in which you might have written a loop, or a series of loops, can often turn into fairly simple LINQ queries. Zipping The Zip operator is useful when you have two related sequences, where each element in one sequence is somehow connected with the element at the same position in the other sequence. You can unite the two sequences by zipping them back into one. Ob- viously, the name has nothing to do with the popular ZIP compression format. This operator is named after zippers of the kind used in clothing. This might be useful with a race car telemetry application of the kind we discussed in Chapter 2. You might end up with two distinct series of data produced by two different measurement sources. For example, fuel level readings and lap time readings could be two separate sequences, since such readings would likely be produced by different instruments. But if you’re getting one reading per lap in each sequence, it might be useful to combine these into a single sequence with one element per lap, as Exam- ple 8-24 shows. 288 | Chapter 8: LINQ Example 8-24. Zipping two sequences into one IEnumerable lapTimes = GetLapTimes(); IEnumerable fuelLevels = GetLapFuelLevels(); var lapInfo = lapTimes.Zip(fuelLevels, (time, fuel) => new { LapTime = time, FuelLevel = fuel }); You invoke the Zip operator on one of the input streams, passing in the second stream as the first argument. The second argument is a projection function—it’s similar to the projections used with the Select operator, except it is passed two arguments, one for each stream. So the lapInfo sequence produced by Example 8-24 will contain one item per lap, where the items are of an anonymous type, containing both the LapTime and the FuelLevel in a single item. Since the two sequences are of equal length here—the number of laps completed—it’s clear how long the output sequence will be, but what if the input lengths differ? The Zip operator stops as soon as either one of the input sequences stops, so the shorter of the two determines the length. Any spare elements in the longer stream will not be used. Getting Selective Sometimes you won’t want to work with an entire collection. For example, in an ap- plication with limited screen space, you might want to show just the next three events on the user’s calendar. While there is no way to do this directly in a query expression, LINQ defines a Take operator for this purpose. As Example 8-25 shows, you can still use the query syntax for most of the query, using the Take operator as the final stage. Example 8-25. Taking the first few results of a query var eventsByStart = from ev in events orderby ev.StartTime where ev.StartTime > DateTimeOffset.Now select ev; var next3Events = eventsByStart.Take(3); LINQ also defines a Skip operator which does the opposite of Take—it drops the first three items (or however many you ask it to drop) and then returns all the rest. If you’re interested in only the very first item, you may find the First operator more convenient. If you were to call Take(1), the method would still return a collection of items. So this code would not compile: CalendarEvent nextEvent = eventsByStart.Take(1); LINQ Operators | 289 You’d get the following compiler error: CS0266: Cannot implicitly convert type 'System.Collections.Generic.IEnumerable< CalendarEvent>' to CalendarEvent'. An explicit conversion exists (are you missing a cast?) In other words, Take always returns an IEnumerable, even if we ask for only one object. But this works: CalendarEvent nextEvent = eventsByStart.First(); First gets the first element from the enumeration and returns that. (It then abandons the enumerator—it doesn’t iterate all the way to the end of the sequence.) You may run into situations where the list might be empty. For example, suppose you want to show the user’s next appointment for today—it’s possible that there are no more appointments. If you call First in this scenario, it will throw an exception. So there’s also a FirstOrDefault operator, which returns the default value when there are no elements (e.g., null, if you’re dealing with a reference type). The Last and LastOrDefault operators are similar, except they return the very last element in the sequence, or the default value in the case of an empty sequence. A yet more specialized case is where you are expecting a sequence to contain no more than one element. For example, suppose you modify the CalendarEvent class to add an ID property intended to be used as a unique identifier for the event. (Most real calendar systems have a concept of a unique ID to provide an unambiguous way of referring to a particular calendar entry.) You might write this sort of query to find an item by ID: var matchingItem = from ev in events where ev.ID == theItemWeWant select ev; If the ID property is meant to be unique, we would hope that this query returns no more than one item. The presence of two or more items would point to a problem. If you use either the First or the FirstOrDefault operator, you’d never notice the problem—these would pick the first item and silently ignore any more. As a general rule, you don’t want to ignore signs of trouble. In this case, it would be better to use either Single or SingleOrDefault. Single would be the right choice in cases where failure to find a match would be an error, while SingleOrDefault would be appropriate if you do not neces- sarily expect to find a match. Either will throw an InvalidOperationException if the sequence contains more than one item. So given the previous query, you could use the following: CalendarEvent item = matchingItem.SingleOrDefault(); If a programming error causes multiple different calendar events to end up with the same ID, this code will detect that problem. (And if your code contains no such prob- lem, this will work in exactly the same way as FirstOrDefault.) 290 | Chapter 8: LINQ Testing the Whole Collection You may need to discover at runtime whether certain characteristics are true about any or every element in a collection. For example, if the user is adding a new event to the calendar, you might want to warn him if the event overlaps with any existing items. First, we’ll write a helper function to do the date overlap test: static bool TimesOverlap(DateTimeOffset startTime1, TimeSpan duration1, DateTimeOffset startTime2, TimeSpan duration2) { DateTimeOffset end1 = startTime1 + duration1; DateTimeOffset end2 = startTime2 + duration2; return (startTime1 < startTime2) ? (end1 > startTime2) : (startTime1 < end2); } Then we can use this to see if any events overlap with the proposed time for a new entry: DateTimeOffset newEventStart = new DateTimeOffset(2009, 7, 20, 19, 45, 00, TimeSpan.Zero); TimeSpan newEventDuration = TimeSpan.FromHours(5); bool overlaps = events.Any( ev => TimesOverlap(ev.StartTime, ev.Duration, newEventStart, newEventDuration)); The Any operator looks to see if there is at least one item for which the condition is true, and it returns true if it finds one and false if it gets to the end of the collection without having found a single item that meets the condition. So if overlaps ends up false here, we know that events didn’t contain any items whose time overlapped with the proposed new event time. There’s also an All operator that returns true only if all of the items meet the condition. We could also have used this for our overlap test—we’d just need to invert the sense of the test: bool noOverlaps = events.All( ev => !TimesOverlap(ev.StartTime, ev.Duration, newEventStart, newEventDuration)); The All operator returns true if you apply it to an empty sequence. This surprises some people, but it’s difficult to say what the right behavior is—what does it mean to ask if some fact is true about all the elements if there are no elements? This operator’s definition takes the view that it returns false if and only if at least one element does not meet the condition. And while there is some logic to that, you would probably feel misled if a company told you “All our customers think our widgets are the best they’ve ever seen” but neglected to mention that it has no customers. LINQ Operators | 291 There’s an overload of the Any operator that doesn’t take a condition. You can use this to ask the question: is there anything in this sequence? For example: bool doIHaveToGetOutOfBedToday = eventsForToday.Any(); The Any and All operators are technically known as quantifiers. More specifically, they are sometimes referred to as the existential quantifier and the universal quantifier, respectively. You may also have come across the common mathematical notation for these. The existential quantifier is written as a backward E (∃), and is conven- tionally pronounced “there exists.” This corresponds to the Any operator—it’s true if at least one item exists in the set that meets the condition. The universal quantifier is written as an upside down A (∀), and is con- ventionally pronounced “for all.” It corresponds to the All operator, and is true if all the elements in some set meet the condition. The con- vention that the universal quantifier is true for any empty set (i.e., that All returns true when you give it no elements, regardless of the condi- tion) has a splendid mathematical name: it is called a vacuous truth. Quantifiers are special cases of a more general operation called aggregation— aggregation operators perform calculations across all the elements in a set. The quan- tifiers are singled out as special cases because they have the useful property that the calculation can often terminate early: if you’re testing to see whether something is true about all the elements in the set, and you find an element for which it’s not true, you can stop right there. But for most whole-set operations that’s not true, so there are some more general-purpose aggregation operators. Aggregation Aggregation operators perform calculations that involve every single element in a col- lection, producing a single value as the result. This can be as simple as counting the number of elements—this involves all the elements in the sense that you need to know how many elements exist to get the correct count. And if you’re dealing with an IEnumerable, it is usually necessary to iterate through the whole collection because in general, enumerable sources don’t know how many items they contain in advance. So the Count operator iterates through the entire collection, and returns the number of elements it found. 292 | Chapter 8: LINQ LINQ to Objects has optimizations for some special cases. It looks for an implementation of a standard ICollection interface, which de- fines a Count property. (This is distinct from the Count operator, which, like all LINQ operators, is a method, not a property.) Collections such as arrays and List that know how many items they contain imple- ment this interface. So the Count operator may be able to avoid having to enumerate the whole collection by using the Count property. And more generally, the nature of the Count operator depends on the source—database LINQ providers can arrange for the database to cal- culate the correct value for Count, avoiding the need to churn through an entire table just to count rows. But in cases where there’s no way of knowing the count up front, such as the file enumeration in Exam- ple 8-1, Count can take a long time to complete. LINQ defines some specialized aggregation operators for numeric values. The Sum op- erator returns the sum of the values of a given expression for all items in a collection. For example, if you want to find out how many hours of meetings you have in a col- lection of events, you could do this: double totalHours = events.Sum(ev => ev.Duration.TotalHours); Average calculates the same sum, but then divides the result by the number of items, returning the mean value. Min and Max return the lowest and highest of the values cal- culated by the expression. There’s also a general-purpose aggregation operator called Aggregate. This lets you perform any operation that builds up some value by performing some calculation on each item in turn. In fact, Aggregate is all you really need—the other aggregation op- erators are simply more convenient.‡ For instance, Example 8-26 shows how to im- plement Count using Aggregate. Example 8-26. Implementing Count with Aggregate int count = events.Aggregate(0, (c, ev) => c + 1); The first argument here is a seed value—it’s the starting point for the value that will be built up as the aggregation runs. In this case, we’re building up a count, so we start at 0. You can use any value of any type here—Aggregate is a generic method that lets you use whatever type you like. The second argument is a delegate that will be invoked once for each item. It will be passed the current aggregated value (initially the seed value) and the current item. And then whatever this delegate returns becomes the new aggregated value, and will be passed in as the first argument when that delegate is called for the next item, and so ‡ That’s true for LINQ to Objects. However, database LINQ providers may implement Sum, Average, and so on using corresponding database query features. They might not be able to do this optimization if you use the general-purpose Aggregate operator. LINQ Operators | 293 on. So in this example, the aggregated value starts off at 0, and then we add 1 each time around. The final result is therefore the number of items. Example 8-26 doesn’t look at the individual items—it just counts them. If we wanted to implement Sum, we’d need to add a value from the source item to the running total instead of just adding 1: double hours = events.Aggregate(0.0, (total, ev) => total + ev.Duration.TotalHours); Calculating an average is a little more involved—we need to maintain both a running total and the count of the number of elements we’ve seen, which we can do by using an anonymous type as the aggregation value. And then we can use an overload of Aggregate that lets us provide a separate delegate to be used to determine the final value—that gives us the opportunity to divide the total by the count: double averageHours = events.Aggregate( new { TotalHours = 0.0, Count = 0 }, (agg, ev) => new { TotalHours = agg.TotalHours + ev.Duration.TotalHours, Count = agg.Count + 1 }, (agg) => agg.TotalHours / agg.Count); Obviously, it’s easier to use the specialized Count, Sum, and Average operators, but this illustrates the flexibility of Aggregate. While LINQ calls this mechanism Aggregate, it is often known by other names. In functional programming languages, it’s sometimes called fold or reduce. The latter name in particular has become slightly better known in recent years thanks to Google’s much-publicized use of a programming system called map/reduce. (LINQ’s name for map is Select, incidentally.) LINQ’s names weren’t chosen to be different for the sake of it—they are more consistent with these concepts’ names in database query languages. Most professional developers are currently likely to have rather more experience with SQL than, say, Haskell or LISP. Set Operations LINQ provides operators for some common set-based operations. If you have two col- lections, and you want to discover all the elements that are present in both collections, you can use the Intersect operator: var inBoth = set1.Intersect(set2); It also offers a Union operator, which provides all the elements from both input sets, but when it comes to the second set it will skip any elements that were already returned because they were also in the first set. So you could think of this as being like Concat, 294 | Chapter 8: LINQ except it detects and removes duplicates. In a similar vein, there’s the Distinct operator—this works on a single collection, rather than a pair of collections. Distinct ensures that it returns any given element only once, so if your input collection happens to contain duplicate entries, Distinct will skip over those. Finally, the Except operator returns only those elements from the first set that do not also appear in the second set. Joining LINQ supports joining of sources, in the sense typically associated with databases— given two sets of items, you can form a new set by combining the items from each set that have the same value for some attribute. This is a feature that tends not to get a lot of use when working with object models—relationships between objects are usually represented with references exposed via properties, so there’s not much need for joins. But joins can become much more important if you’re using LINQ with data from a relational database. (Although the Entity Framework, which we describe in a later chapter, is often able to represent relationships between tables as object references. It’ll use joins at the database level under the covers, but you may not need to use them explicitly in LINQ all that often.) Even though joins are typically most useful when working with data structured for storage in a relational database, you can still perform joins across objects—it’s possible with LINQ to Objects even if it’s not all that common. In our hypothetical calendar application, imagine that you want to add a feature where you can reconcile events in the user’s local calendar with events retrieved from his phone’s calendar, and you need to try to work out which of the imported events from the phone correspond to items already in the calendar. You might find that the only way to do this is to look for events with the same name that occur at the same time, in which case you might be able to use a join to build up a list of events from the two sources that are logically the same events: var pairs = from localEvent in events join phoneEvent in phoneEvents on new { Title = localEvent.Title, Start = localEvent.StartTime } equals new { Title = phoneEvent.Name, Start = phoneEvent.Time } select new { Local = localEvent, Phone = phoneEvent }; A LINQ join expects to be able to compare just a single object in order to determine whether two items should be joined. But we want to join items only when both the title and the time match. So this example builds an anonymously typed object to hold both values in order to be able to provide LINQ with the single object it expects. (You can use this technique for the grouping operators too, incidentally.) Note that this example also illustrates how you would deal with the relevant properties having different names. You can imagine that the imported phone events might use different property names because you might need to use some third-party import library, so this example shows LINQ Operators | 295 how the code would look if it called the relevant properties Name and Time instead of Title and StartTime. We fix this by mapping the properties from the two sources into anonymous types that have the same structure. Conversions Sometimes it’s necessary to convert the results of a LINQ query into a specific collection type. For example, you might have code that expects an array or a List. You can still use LINQ queries when creating these kinds of collections, thanks to the standard ToArray and ToList operators. Example 8-17 used ToArray to convert a grouping into an array of objects. We could extend that further to convert the query into an array of arrays, just like the original example from Chapter 7: var eventsByDay = from ev in events group ev by ev.StartTime.Date into dayGroup select dayGroup.ToArray(); CalendarEvent[][] arrayOfEventsByDay = eventsByDay.ToArray(); In this example, eventsByDay is of type IEnumerable. The final line then turns the enumeration into an array of arrays—a CalendarEvent[][]. Remember that LINQ queries typically use deferred execution—they don’t start doing any work until you start asking them for elements. But by calling ToList or ToArray, you will fully execute the query, because it builds the entire list or array in one go. As well as providing conversion operators for getting data out of LINQ and into other data types, there are some operators for getting data into LINQ’s world. Sometimes you will come across types that provide only the old .NET 1.x-style nongeneric IEnumerable interface. This is problematic for LINQ because there’s no way for it to know what kinds of objects it will find. You might happen to know that a collection will always contain CalendarEvent objects, but this would be invisible to LINQ if you are working with a library that uses old-style collections. So to work around this, LINQ defines a Cast operator—you can use this to tell LINQ what sort of items you believe are in the collection: IEnumerable oldEnum = GetCollectionFromSomewhere(); var items = from ev in oldEnum.Cast() orderby ev.StartTime select ev; As you would expect, this will throw an InvalidCastException if it discovers any ele- ments in the collection that are not of the type you said. But be aware that like most LINQ operators, Cast uses deferred execution—it casts the elements one at a time as they are requested, so any mismatch will not be discovered at the point at which you call Cast. The exception will be thrown at the point at which you reach the first non- matching item while enumerating the query. 296 | Chapter 8: LINQ Summary LINQ provides a convenient syntax for performing common operations on collections of data. The query expression syntax is reminiscent of database query languages, and can be used in conjunction with databases, as later chapters will show. But these queries are frequently used on objects in memory. The compiler transforms the query syntax into a series of method calls, meaning that the choice of LINQ implementation is de- termined by context—you can write your own custom LINQ provider, or use a built- in provider such as LINQ to Objects, LINQ to SQL, or LINQ to XML. All providers implement standard operators—methods with well-known names and signatures that implement various common query features. The features include filter- ing, sorting, grouping, and the ability to transform data through a projection. You can also perform test and aggregation operations across entire sets. Queries can be composed—most operators’ output can be used as input to other operators. LINQ uses a functional style to maximize the flexibility of composition. Summary | 297 CHAPTER 9 Collection Classes In the preceding two chapters we saw how to store information in arrays and lists, and how to sort, search, and process that information using LINQ. Important as sequential lists and rectangular arrays are, they don’t accommodate every possible requirement you could have for storing and structuring data. So in this final chapter on working with collections, we’ll look at some of the other collection classes offered by the .NET Framework. Dictionaries A dictionary is a collection that enables you to look up information associated with some kind of value. .NET calls this sort of collection a dictionary because it is remi- niscent of a traditional printed dictionary: the information is structured to make it easy to find the entry for a particular word—if you know what word you’re looking for, you can find it very quickly even among tens of thousands of definitions. The information you find when you’ve looked up the word depends on the sort of dictionary you bought—it might provide a definition of the word, but other kinds exist, such as dic- tionaries of quotations, or of etymology. Likewise, a .NET dictionary collection is structured to enable quick and easy lookup of entries. The syntax looks very similar to array access, but where you’d expect to see a number, the index can be something else, such as a string, as shown in Example 9-1. Example 9-1. Looking up an entry in a dictionary string definition = myDictionary["sea"]; Just as printed dictionaries vary in what you get when you look up a word, so can .NET dictionaries. The Dictionary type in the System.Collections.Generic namespace is a generic type, letting you choose the type for both the key—the value used for the index—and the value associated with the index. (Note that there are some restrictions regarding the key type—see the sidebar on the next page.) Example 9-1, which models 299 a traditional printed dictionary, uses strings for the index, and expects a string as the result, so myDictionary in that example would be defined as shown in Example 9-2. Example 9-2. A dictionary with string keys and string values Dictionary myDictionary = new Dictionary(); Keys, Comparison, and Hashes To be able to look up entries quickly, dictionaries impose a couple of requirements on keys. First, a dictionary entry’s key must not change in a way that affects comparisons. (This often just means that you should never change a key. However, it’s technically possible to build a type for which certain kinds of changes have no impact on compar- isons performed by the Equals methods. Such changes are invisible to the dictionary.) Second, it should provide a good hash function. To understand the first requirement—that for comparison purposes, keys must not change—consider what changing a key would mean in a printed dictionary. Suppose you look up the entry for bug in your dictionary, and then you cross out the word bug and write feature in its place. The usual way of looking up words will now fail for this entry—the entry was positioned in exactly the right place for when the key was bug. Anyone looking up feature will not think to look in the location for your amended item. And it’s the same with a dictionary collection—to enable fast lookup, dictionaries create an internal structure based on the keys items had when they were added to the dictionary. It has no way of knowing when you’ve changed a key value. If you really need to do this, you should remove the entry, and then add it back with the new key— this gives the dictionary a chance to rebuild its internal lookup data structures. This requirement is most easily met by using an immutable type, such as string, or any of the built-in numeric types. The second requirement—that key types should have a good hash function—is a bit less obvious, and has to do with how dictionary collections implement fast lookup. The base System.Object class defines a virtual method called GetHashCode, whose job is to return an int whose value loosely represents the value of the object. GetHashCode is required to be consistent with the Equals method (also defined by System.Object)— two objects or values that are equal according to Equals are required to return the same hash code. Those are the rules, and dictionaries will not work if you break them. This means that if you override Equals, you are required to override GetHashCode, and vice versa. The rules about hash codes for items that are not equal are more flexible. Ideally, non- equal items should return nonequal hash codes, but clearly that’s not always possible: a long can have any of several quintillion distinct values, but a hash code is an int, which has merely a few billion possible values. So inevitably there will be hash collisions—nonequal values that happen to have equal hash codes. For example, long returns the same hash code for the values 4,294,967,296 and 1. GetHashCode implementations should try to minimize hash collisions. The reason is that dictionaries use the hash code to work out where to put the entry—in the printed 300 | Chapter 9: Collection Classes dictionary analogy the hash code effectively tells it on which page the entry belongs. Hash collisions mean dictionary entries share a page, and the dictionary has to spend time scanning through the entries to find the right one. The fewer hash collisions you have, the faster dictionaries work. If you use built-in numeric types such as int as keys, or if you use string, you can safely ignore all of this because these types provide good hash codes. You need to care about this only if you plan to use a custom type which defines its own notion of equality (i.e., that overrides Equals). When you start using collection types that require multiple generic type arguments such as this, specifying the full type name in the variable declaration and then again in the constructor starts to look a bit verbose, so if you’re using a dictionary with a local variable, you might prefer to use the var keyword introduced in C# 3.0, as shown in Example 9-3. Example 9-3. Avoiding repetitive strain injury with var var myDictionary = new Dictionary(); Remember, the var keyword simply asks C# to work out the variable’s type by looking at the expression you’re using to initialize the variable. So Example 9-3 is exactly equiv- alent to Example 9-2. Just as arrays and other lists can be initialized with a list of values in braces, you can also provide an initializer list for a dictionary. As Example 9-4 shows, you must provide both a key and a value for each entry, so each entry is contained within nested braces to keep the key/value grouping clear. Example 9-4. Dictionary initializer list var myDictionary = new Dictionary() { { "dog", "Not a cat." }, { "sea", "Big blue wobbly thing that mermaids live in." } }; Storing individual items in a dictionary also uses an array-like syntax: myDictionary["sea"] = "Big blue wobbly thing that mermaids live in."; As you may already have guessed, dictionaries exploit the C# indexer feature we saw in Chapter 8. Common Dictionary Uses Dictionaries are extremely useful tools, because the situations they deal with—anyplace something is associated with something else—crop up all the time. They are so widely used that it’s helpful to look at some common concrete examples. Dictionaries | 301 Looking up values Computer systems often use inscrutable identifiers where people would normally use a name. For example, imagine a computer system for managing patients in a hospital. This system would need to maintain a list of appointments, and it would be useful for the hospital’s reception staff to be able to tell arriving patients where to go for their appointment. So the system would need to know about things such as buildings and departments—radiography, physiotherapy, and so on. The system will usually have some sort of unique identifier for entities such as these in order to avoid ambiguity and ensure data integrity. But users of the system will most likely want to know that an appointment is in the Dr. Marvin Munroe Memorial Build- ing, rather than, say, the building with ID 49. So the user interface will need to convert the ID to text. The information about which ID corresponds to which building typically belongs in a database, or possibly a configuration file. You wouldn’t want to bake the list into a big switch statement in the code, as that would make it hard to support multiple customers, and would also need a new software release anytime a building is built or renamed. The user interface could just look up the name in the database every time it needs to display an appointment’s information, but there are a few problems with this. First, the computer running the UI might not have access to the database—if the UI is written as a client-side application in WPF or Windows Forms, there’s a good chance that it won’t—a lot of companies put databases behind firewalls that restrict access even on the internal network. And even if it did, making a request to the server for each ID- based field takes time—in a form with several such fields, you could easily end up causing a noticeable delay. And this would be unnecessary for data that’s not expected to change from day to day. A dictionary can provide a better solution here. When the application starts up, it can load a dictionary to go from an ID to a name. Translating an ID to a displayable name then becomes as simple as this: string buildingName = buildingIdToNameMap[buildingId]; As for how you would load this dictionary in the first place, that’ll depend on where the data is stored. Whether it’s coming from a file, a database, a web service, or some- where else, you can use LINQ to initialize a dictionary—we’ll see how to do that in “Dictionaries and LINQ.” Caching Dictionaries are often used to cache information that is slow to fetch or create. Infor- mation can be placed in a dictionary the first time it is loaded, allowing an application to avoid that cost the next time the information is required. For example, suppose a doctor is with a patient, and wants to look at information regarding recent tests or medical procedures the patient has undergone. This will typically involve requesting 302 | Chapter 9: Collection Classes records from some server to describe these patient encounters, and you might find that the application can be made considerably more responsive by keeping hold of the re- cords on the client side after they have been requested so that they don’t have to be looked up again and again as the doctor scrolls through a list of records. This is a very similar idea to the lookup usage we described earlier, but there are two important differences. First, caches usually need some sort of policy that decides when to remove data from the cache—if we add every record we load into the cache and never clear out old ones, the cache will consume more memory over time, slowing the program down, which is the opposite of the intended effect. The appropriate policy for removing items from a cache will be application-specific. In a patient record viewing application, the best approach might be to clear the cache as soon as the doctor starts looking up information for a new patient, since that suggests the doctor has moved on to a new appointment and therefore won’t be looking at the previous patient’s details anytime soon. But that policy works only because of how this particular system is used—other systems may require other mechanisms. Another popular heuristic is to set an upper limit on either the number of entries or the total size of the data in a cache, and to remove items based on when they were last retrieved. The second difference between using a dictionary to cache data rather than to look up preloaded data is that in caching scenarios, the data involved is often more dynamic. A list of known countries won’t change very often, but patient records might change while in use, particularly if the patient is currently in the hospital. So when caching copies of information locally in a dictionary, you need to have some way of dealing with the fact that the information in the cache might be stale. (For example, although caching patient records locally would be useful, your application might need to deal with the possibility that new test results become available while the patient is with the doctor.) As with removal policy, detection of stale data requires application-specific logic. For example, some data might never change—accounting data usually works this way, because even when data is discovered to be incorrect, it’s not usually modified. Legal auditing requirements usually mean you have to fix problems by adding a new ac- counting record that corrects the old one. So for this sort of entry, you know that a cache of any given record will never be stale. (Newer records might exist that supersede the cached record you have, but the cache of that record will still be consistent with whatever is in the database for that entry.) Sometimes it might be possible to perform a relatively cheap test to discover whether the cached record is consistent with the information on a server. HTTP supports this— a client can send a request with an If-Modified-Since header, containing the date at which the cached information was known to be up-to-date. If the server has no newer information, it sends a very short reply to confirm this, and will not send a new copy of the data. Web browser caches use this to make web pages you’ve previously visited load faster, while ensuring that you always see the most recent version of the page. Dictionaries | 303 But you may simply have to guess. Sometimes the best staleness heuristic available to you might be something such as “If the cached record we have is more than 20 minutes old, let’s get a fresh copy from the server.” But you need to be careful with this approach. Guesswork can sometimes lead to a cache that offers no useful performance improve- ments, or which produces data that is too stale to be useful, or both. Regardless of the precise details of your cache removal and staleness policies, the ap- proach will look something like Example 9-5. (The Record type in this example is not a class library type, by the way. It’s just for illustration—it would be the class for what- ever data you want to cache.) Example 9-5. Using a dictionary for caching class RecordCache { private Dictionary cachedRecords = new Dictionary(); public Record GetRecord(int recordId) { Record result; if (cachedRecords.TryGetValue(recordId, out result)) { // Found item in cache, but is it stale? if (IsStale(result)) { result = null; } } if (result == null) { result = LoadRecord(recordId); // Add newly loaded record to cache cachedRecords[recordId] = result; } DiscardAnyOldCacheEntries(); return result; } private Record LoadRecord(int recordId) { ... Code to load the record would go here ... } private bool IsStale(Record result) { ... Code to work out whether the record is stale would go here ... } 304 | Chapter 9: Collection Classes private void DiscardAnyOldCacheEntries() { ... Cache removal policy code would go here ... } } Notice that this code does not use the indexer to look up a cache entry. Instead, it uses a method called TryGetValue. You use this when you’re not sure if the dictionary will contain the entry you’re looking for—in this case, the entry won’t be present the first time we look it up. (The dictionary would throw an exception if we use the indexer to look up a value that’s not present.) TryGetValue returns true if an entry for the given key was found, and false if not. Notice that its second argument uses the out qualifier and it uses this to return the item when it’s found, or a null reference when it’s not found. You might be wondering why TryGetValue doesn’t just use a null return value to indicate that a record wasn’t found, rather than this slightly clumsy arrangement with a bool return value and an out argument. But that wouldn’t work with value types, which cannot be null. Dictionaries can hold either reference types or value types. Dynamic properties Another common use for a dictionary is when you want something that works like a property, but where the set of available properties is not necessarily fixed. For example, WCF is designed to send and receive messages over a wide range of network technol- ogies, each of which may have its own unique characteristics. So WCF defines some normal properties and methods to deal with aspects of communication that are com- mon to most scenarios, but also provides a dictionary of dynamic properties to handle transport-specific scenarios. For example, if you are using WCF with HTTP-based communication, you might want your client code to be able to modify the User-Agent header. This header is specific to HTTP, and so WCF doesn’t provide a property for this as part of its programming model, because it wouldn’t do anything for most network protocols. Instead, you con- trol this with a dynamic property, added via the WCF Message type’s Properties dic- tionary, as Example 9-6 shows. Example 9-6. Setting a dynamic property on a WCF message Message wcfMessage = CreateMessageSomehow(); HttpRequestMessageProperty reqProps = new HttpRequestMessageProperty(); reqProps.Headers.Add(HttpRequestHeader.UserAgent, "my user agent"); wcfMessage.Properties[HttpRequestMessageProperty.Name] = reqProps; Dictionaries | 305 C# 4.0 introduces an alternative way to support dynamic properties, through the dynamic keyword (which we will describe in Chapter 18). This makes it possible to use normal C# property access syntax with properties whose availability is determined at runtime. So you might think dynamic makes dictionaries redundant. In practice, dynamic is nor- mally used only when interacting with dynamic programming systems such as scripting languages, so it’s not based on .NET’s dictionary mechanisms. Sparse arrays The final common scenario we’ll look at for dictionaries is to provide efficient storage for a sparse array. A sparse array is indexed by an integer, like a normal array, but only a tiny fraction of its elements contain anything other than the default value. For a numeric element type that would mean the array is mostly zeros, while for a reference type it would be mostly nulls. As an example of where this might be useful, consider a spreadsheet. When you create a new spreadsheet, it appears to be a large expanse of cells. But it’s not really storing information for every cell. I just ran Microsoft Excel, pressed Ctrl-G to go to a particular cell and typed in $XFD$1000000, and then entered a value for that cell. This goes to the 16,384th column (which is as wide as Excel 2007 can go), and the 1 millionth row. Yet despite spanning more than 16 billion cells, the file is only 8 KB. And that’s because it doesn’t really contain all the cells—it only stores information for the cells that contain something. The spreadsheet is sparse—it is mostly empty. And it uses a representation that makes efficient use of space when the data is sparse. If you try to create a rectangular array with 16,384 columns and 1 million rows, you’ll get an exception as such an array would go over the .NET 4 upper size limit for any single array of 2 GB. A newly created array always contains default values for all of its elements, so the information it contains is always sparse to start with—sparseness is a characteristic of the data, rather than the storage mechanism, but the fact that we simply cannot create a new, empty array this large demonstrates that a normal array doesn’t store sparse information efficiently. There is no built-in type designed specifically for storing sparse data, but we can use a dictionary to make such a thing. Example 9-7 uses a dictionary to provide storage for a single-dimensional sparse array of double elements. It uses long as the index argument type to enable the array to grow to a logical size that is larger than would be possible with an int, which tops out at around 2.1 billion. Example 9-7. A sparse array of numbers class SparseArray { private Dictionary nonEmptyValues = 306 | Chapter 9: Collection Classes new Dictionary(); public double this[long index] { get { double result; nonEmptyValues.TryGetValue(index, out result); return result; } set { nonEmptyValues[index] = value; } } } Notice that this example doesn’t bother to check the return value from TryGetValue. That’s because when it fails to find the entry, it sets the result to the default value, and in the case of a double, that means 0. And 0 is what we want to return for an entry whose value has not been set yet. The following code uses the SparseArray class: SparseArray big = new SparseArray(); big[0] = 123; big[10000000000] = 456; Console.WriteLine(big[0]); Console.WriteLine(big[2]); Console.WriteLine(big[10000000000]); This sets the value of the first element, and also the element with an index of 10 billion— this simply isn’t possible with an ordinary array. And yet it works fine here, with min- imal memory usage. The code prints out values for three indexes, including one that hasn’t been set. Here are the results: 123 0 456 Reading the value that hasn’t been set returns the default value of 0, as required. Some arrays will be sparser than others, and there will inevitably come a point of insufficient sparseness at which this dictionary-based ap- proach will end up being less efficient than simply using a large array. It’s hard to predict where the dividing line between the two techniques will fall, as it will depend on factors such as the type and quantity of data involved, and the range of index values. As with any implementa- tion choice made on the grounds of efficiency, you should compare the performance against the simpler approach to find out whether you’re getting the benefit you hoped for. Dictionaries | 307 IDictionary The examples we’ve seen so far have all used the Dictionary type defined in the System.Collections.Generic namespace. But that’s not the only dictionary. As we saw a couple of chapters ago, the IEnumerable type lets us write polymorphic code that can work with any sequential collection class. We can do the same with a dictionary— the .NET Framework class library defines an interface called IDictionary, which is reproduced in Example 9-8. Example 9-8. IDictionary namespace System.Collections.Generic { public interface IDictionary : ICollection>, IEnumerable>, IEnumerable { void Add(TKey key, TValue value); bool ContainsKey(TKey key); bool Remove(TKey key); bool TryGetValue(TKey key, out TValue value); TValue this[TKey key] { get; set; } ICollection Keys { get; } ICollection Values { get; } } } You can see the indexer—TValue this[TKey]—and the TryGetValue method that we already looked at. But as you can see, dictionaries also implement other useful standard features. The Add method adds a new entry to the dictionary. This might seem redundant because you can add new entries with the indexer, but the difference is that the indexer will happily overwrite an existing value. But if you call Add, you are declaring that you believe this to be a brand-new entry, so the method will throw an exception if the dictionary already contained a value for the specified key. There are members for helping you discover what’s already in the dictionary—you can get a list of all the keys and values from the Keys and Values properties. Both of these implement ICollection, which is a specialized version of IEnumerable that adds in useful members such as Count, Contains, and CopyTo. Notice also that IDictionary derives from IEnumerable>. This means it’s possible to enumerate through the contents of a dictionary with a foreach loop. The KeyPairValue items returned by the enumeration just package the key and associated value into a single struct. We could 308 | Chapter 9: Collection Classes add the method in Example 9-9 to the class in Example 9-7, in order to print out just those elements with a nondefault value. Example 9-9. Iterating through a dictionary’s contents public void ShowArrayContents() { foreach (var item in nonEmptyValues) { Console.WriteLine("Key: '{0}', Value: '{1}'", item.Key, item.Value); } } Remember, the presence of IEnumerable is all that LINQ to Objects needs, so we can use dictionaries with LINQ. Dictionaries and LINQ Because all IDictionary implementations are also enumerable, we can run LINQ queries against them. Given the RecordCache class in Example 9-5, we might choose to implement the cache item removal policy as shown in Example 9-10. Example 9-10. LINQ query with dictionary source private void DiscardAnyOldCacheEntries() { // Calling ToList() on source in order to query a copy // of the enumeration, to avoid exceptions due to calling // Remove in the foreach loop that follows. var staleKeys = from entry in cachedRecords.ToList() where IsStale(entry.Value) select entry.Key; foreach (int staleKey in staleKeys) { cachedRecords.Remove(staleKey); } } But it’s also possible to create new dictionaries with LINQ queries. Example 9-11 il- lustrates how to use the standard ToDictionary LINQ operator. Example 9-11. LINQ’s ToDictionary operator IDictionary buildingIdToNameMap = MyDataSource.Buildings.ToDictionary( building => building.ID, building => building.Name); This example presumes that MyDataSource is some data source class that provides a queryable collection containing a list of buildings. Since this information would typi- cally be stored in a database, you would probably use a database LINQ provider such Dictionaries | 309 as LINQ to Entities or LINQ to SQL. The nature of the source doesn’t greatly matter, though—the mechanism for extracting the resources into a dictionary object are the same in any case. The ToDictionary operator needs to be told how to extract the key from each item in the sequence. Here we’ve provided a lambda expression that retrieves the ID property—again, this property would probably be generated by a database map- ping tool such as the ones provided by the Entity Framework or LINQ to SQL. (We will be looking at data access technologies in a later chapter.) This example supplies a second lambda, which chooses the value—here we pick the Name property. This second lambda is optional—if you don’t provide it, ToDictionary will just use the entire source item from the stream as the value—so in this example, leaving out the second lambda would cause ToDictionary to return an IDictionary (where Building is whatever type of object MyDataSource.Buildings provides). The code in Example 9-11 produces the same result as this: var buildingIdToNameMap = new Dictionary(); foreach (var building in MyDataSource.Buildings) { buildingIdToNameMap.Add(building.ID, building.Name); } HashSet and SortedSet HashSet is a collection of distinct values. If you add the same value twice, it will ignore the second add, allowing any given value to be added only once. You could use this to ensure uniqueness—for example, imagine an online chat server. If you wanted to make sure that usernames are unique, you could maintain a HashSet of usernames used so far, and check that a new user’s chosen name isn’t already in use by calling the hash set’s Contains method. You might notice that List offers a Contains method, and so with a little extra code, you could implement a uniqueness check using List. However, HashSet uses the same hash-code-based fast lookup as a dictionary, so HashSet will be faster for large sets than List. HashSet was added in .NET 3.5. Prior to that, people tended to use a dictionary with nothing useful in the value as a way of getting fast hash-code-based uniqueness testing. .NET 4 adds SortedSet, which is very similar to HashSet, but adds the feature that if you iterate through the items in the set, they will come out in order. (You can provide an IComparer to define the required order, or you can use self-ordering types.) Obviously, you could achieve the same effect by applying the OrderBy LINQ operator to a HashSet, but SortedSet sorts the items as they are added, meaning that they’re already sorted by the time you want to iterate over them. 310 | Chapter 9: Collection Classes Both HashSet and SortedSet offer various handy set-based methods. You can determine whether an IEnumerable is a subset of (i.e., all its elements are also found in) a set with the IsSubsetOf, for example. The available methods are defined by the common ISet interface, reproduced in Example 9-12. Example 9-12. ISet namespace System.Collections.Generic { public interface ISet : ICollection, IEnumerable, IEnumerable { bool Add(T item); void ExceptWith(IEnumerable other); void IntersectWith(IEnumerable other); bool IsProperSubsetOf(IEnumerable other); bool IsProperSupersetOf(IEnumerable other); bool IsSubsetOf(IEnumerable other); bool IsSupersetOf(IEnumerable other); bool Overlaps(IEnumerable other); bool SetEquals(IEnumerable other); void SymmetricExceptWith(IEnumerable other); void UnionWith(IEnumerable other); } } Queues Queue is a handy collection type for processing entities on a first come, first served basis. For example, some doctors’ general practice surgeries operate an appointment- free system. Since the time taken to see each patient in these scenarios can vary wildly depending on the problem at hand, seeing patients in turn can end up being a lot more efficient than allocating fixed-length appointment slots. We could model this with a Queue (where Patient is some class defined by our application). When a patient arrives, she would be added to a queue by calling its Enqueue method: private Queue waitingPatients = new Queue(); ... public void AddPatientToQueue(Patient newlyArrivedPatient) { waitingPatients.Enqueue(newlyArrivedPatient); } When a doctor has finished seeing one patient and is ready to see the next, the Dequeue method will return the patient who has been in the queue longest, and will then remove that patient from the queue: Patient nextPatientToSee = waitingPatients.Dequeue(); Queues | 311 While this example perfectly matches how Queue works, you prob- ably wouldn’t use a Queue here in practice. You’d want to handle crashes and power failures gracefully in this application, which means that in practice, you’d probably store the list of waiting patients in a database, along with something such as a ticket number indicating their place in the queue. In-memory queues tend to show up more often in multithreaded servers for keeping track of outstanding work. But since we haven’t gotten to either the networking or the threading chapters yet, an example along those lines would be premature. Queue implements IEnumerable, so you can use LINQ queries across items in the whole queue. It also implements ICollection, so you can discover whether the queue is currently empty by inspecting its Count property. Queue operates in strict first in, first out (FIFO) order, which is to say that Dequeue will return items in exactly the same order in which they were added with Enqueue. That might be fine for general practice, but it wouldn’t work so well for the emergency room. Linked Lists If you’ve ever had to visit a hospital emergency room, you’ll know that waiting in a queue is one of the defining features of the experience unless you were either very lucky or very unlucky. If you were lucky, the queue will have been empty and you will not have had to wait. Alternatively, if you were unlucky, your condition may have been sufficiently perilous that you got to jump to the head of the queue. In medical emergencies, a triage system will be in place to work out where each arriving patient should go in the queue. A similar pattern crops up in other scenarios—frequent fliers with gold cards may be allocated standby seats at the last minute even though others have been waiting for hours; celebrities might be able to walk right into a res- taurant for which the rest of us have to book a table weeks in advance. The LinkList class is able to model these sorts of scenarios. At its simplest, you could use it like a Queue—call AddLast to add an item to the back of the queue (as Enqueue would), and RemoveFirst to take the item off the head of the queue (like Dequeue would). But you can also add an item to the front of the queue with AddFirst. Or you can add items anywhere you like in the queue with the AddBefore and AddAfter methods. Example 9-13 uses this to place new patients into the queue. Example 9-13. Triage in action private LinkedList waitingPatients = new LinkedList(); ... LinkedListNode current = waitingPatients.First; 312 | Chapter 9: Collection Classes while (current != null) { if (current.Value.AtImminentRiskOfDeath) { current = current.Next; } else { break; } } if (current == null) { waitingPatients.AddLast(newPatient); } else { waitingPatients.AddBefore(current, newPatient); } This code adds the new patient after all those patients in the queue whose lives appear to be at immediate risk, but ahead of all other patients—the patient is presumably either quite unwell or a generous hospital benefactor. (Real triage is a little more complex, of course, but you still insert items into the list in the same way, no matter how you go about choosing the insertion point.) Note the use of LinkedListNode—this is how LinkedList presents the queue’s contents. It allows us not only to see the item in the queue, but also to navigate back and forth through the queue with the Next and Previous properties. Stacks Whereas Queue operates a FIFO order, Stack operates a last in, first out (LIFO) order. Looking at this from a queuing perspective, it seems like the height of unfairness—latecomers get priority over those who arrived early. However, there are some situations in which this topsy-turvy ordering can make sense. A performance characteristic of most computers is that they tend to be able to work faster with data they’ve processed recently than with data they’ve not touched lately. CPUs have caches that provide faster access to data than a computer’s main memory can support, and these caches typically operate a policy where recently used data is more likely to stay in the cache than data that has not been touched recently. If you’re writing a server-side application, you may consider throughput to be more important than fairness—the total rate at which you process work may matter more than how long any individual work item takes to complete. In this case, a LIFO order may make the most sense—work items that were only just put into a queue are much more likely to still live in the CPU’s cache than those that were queued up ages ago, Stacks | 313 and so you’ll get better throughput during high loads if you process newly arrived items first. Items that have sat in the queue for longer will just have to wait for a lull. Like Queue, Stack offers a method to add an item, and one to remove it. It calls these Push and Pop, respectively. They are very similar to the queue’s Enqueue and Dequeue, except they both work off the same end of the list. (You could get the same effect using a LinkedList, and always calling AddFirst and RemoveFirst.) A stack could also be useful for managing navigation history. The Back button in a browser works in LIFO order—the first page it shows you is the last one you visited. (And if you want a Forward button, you could define a second stack—each time the user goes Back, Push the current page onto the Forward stack. Then if the user clicks Forward, Pop a page from the Forward stack, and Push the current page onto the Back stack.) Summary The .NET Framework class library provides various useful collection classes. We saw List in an earlier chapter, which provides a simple resizable linear list of items. Dictionaries store entries by associating them with keys, providing fast key-based lookup. HashSet and SortedSet manage sets of unique items, with optional or- dering. Queues, linked lists, and stacks each manage a queue of items, offering various strategies for how the order of addition relates to the order in which items come out of the queue. 314 | Chapter 9: Collection Classes CHAPTER 10 Strings Chapter 10 is all about strings. A bit late, you might think: we’ve had about nine chap- ters of string-based action already! Well, yes, you’d be right. That’s not terribly sur- prising, though: text is probably the single most important means an application has of communicating with its users. That is especially true as we haven’t introduced any graphical frameworks yet. I suppose we could have beeped the system speaker in Morse, although even that can be considered a text-based operation. Even with a graphical UI framework where we have pictures and buttons and graphs and sounds, they almost always have textual labels, descriptions, comments, or tool tips. Users who have difficulty reading (perhaps because they have a low-vision condition) may have that text transformed into sound by accessibility tools, but the application is still processing text strings under the covers. Even when we are dealing with integers or doubles internally within an algorithm, there comes a time when we need to represent them to humans, and preferably in a way that is meaningful to us. We usually do that (at least in part) by converting them into strings of one form or another. Strings are surprisingly complex and sophisticated entities, so we’re going to take some time to explore their properties in this chapter. First, we’ll look at what we’re really doing when we initialize a literal string. Then, we’ll see a couple of techniques which let us convert from other types to a string represen- tation and how we can control the formatting of that conversion. Next, we’ll look at various different techniques we can use to process a string. This will include composition, splitting, searching and replacing content, and what it means to compare strings of various kinds. Finally, we will look at how .NET represents strings internally, how that differs from other representations in popular use in the world, and how we can convert between those representations by using an Encoding. 315 316 | Chapter 10: Strings amples taken from different fonts: Just in case you have been on the moon since 1968, here are three ex- monplace, thanks to the ubiquity of the word processor. sole purview of designers and printers; but they’ve now become com- I think it is interesting to note that only a few years ago, fonts were the use these terms interchangeably now. of font family, typeface, and font in popular usage, and people tend to in a specific design at a certain size, but we’ve come to blur the meanings characters. Historically, it was a box containing a set of moveable type A quick reminder: a font is a particular visual design for an entire set of character by character. because Arabic scripts read right to left and not left to right; but the string is still ordered, first character is the rightmost one, and the last character is the leftmost one. This is If you look carefully, you’ll notice that the string is ordered the other way round—the ' ا', 'ل', 'ع', 'ر', 'ب', 'ي', 'ة ' :Here we have the following characters العربية :for example other textual symbol). It doesn’t even have to be an English letter. It could be Arabic, uppercase letter, lowercase letter, space, punctuation mark, number (or, in fact, any Any ordered sequence of characters is a string. Notice that each character might be an A string doesn’t have to be a whole sentence, of course, or even anything meaningful. And so on. 'W', 'e', ' ', 'c', 'o', 'u', 'l', 'd' right: We start with the first character, which is W. Then we continue on in order from left to We could consider this sentence to be a string. A string is an ordered sequence of characters: What Is a String? You’ll also notice that the “joined up” cursive form of the characters is visually quite different from their form when separated out individually. This is normal; the ultimate visual representation of the character in the string is entirely separate from the string itself. We’re just so used to the characters of our own language that we don’t tend to think of them as abstract symbols, and tend to discount any visual differences down to the choice of font or other typographical niceties when we are interpreting them. We could happily design a font where the character e looks like Q and the character f looks like A. All our text processing would continue as normal: searching and sorting would be just fine (words starting with f wouldn’t start appearing in the dictionary before words starting with e), because the data in the string is unchanged; but when we drew it on the screen, it would look more than a bit confusing.* The take-home point is that there are a bunch of layers between the .NET runtime’s representation of a string as data in memory, and its final visual appearance on a screen, in a file, or in another application (such as notepad.exe, for example). As we go through this chapter, we’ll unpick those layers as we come across them, and point out some of the common pitfalls. Let’s get on and see how the .NET Framework presents a string to us. The String and Char Types It will come as no surprise that the .NET Framework provides us with two types that correspond with strings and characters: String and Char. In fact, as we’ve seen before, these are such important types that C# even provides us with keywords that correspond to the underlying types: string and char. String needs to provide us with that “ordered sequence of characters” behavior. It does so by implementing IEnumerable, as Example 10-1 illustrates. Example 10-1. Iterating through the characters in a string string myString = "I've gone all vertical."; foreach (char theCharacter in myString) { Console.WriteLine(theCharacter); } * In fact, I don’t think that this particular typeface would catch on. The String and Char Types | 317 If you create a console application for this code, you’ll see output like this when it runs: I ' v e g o n e a l l v e r t i c a l . What exactly does that code do? First, it initializes a variable called myString which we will use to hold the reference to our string object (because String is a reference type). We then enumerate the string, yielding every Char in turn, and we output each Char to the console on its own separate line. Char is a value type, so we’re actually getting a copy of the character from the string itself. The string object is created using a literal string—a sequence of characters enclosed in double quotes: "I've gone all vertical." We’re already quite familiar with initializing a string with a literal—we probably do it without a second thought; but let’s have a look at these literals in a little more detail. Literal Strings and Chars The simplest literal string is a set of characters enclosed in double quotes, shown in the first line of Example 10-2. Example 10-2. A string literal string myString = "Literal string"; Console.WriteLine(myString); This produces the output: Literal string 318 | Chapter 10: Strings You can also initialize a string from a char[], using the appropriate constructor. One way to obtain a char array is by using char literals. A char literal is a single character, wrapped in single quotes. Example 10-3 constructs a string this way. Example 10-3. Initializing a string from char literals string myString = new string(new [] { 'H', 'e', 'l', 'l', 'o', ' ', '"', 'w', 'o', 'r', 'l', 'd', '"' }); Console.WriteLine(myString); If you compile and run this, you’ll see the following output: Hello "world" Notice that we’ve got double-quote marks in our output. That was easy to achieve with this char[], because the delimiter for an individual character is the single quote; but how could we include double quotes in the string, without resorting to a literal char array? Equally, how could we specify the single-quote character as a literal char? Escaping Special Characters The way to deal with troublesome characters in string and char literals is to escape them with the backslash character. That means that you precede the quote with a \, and it interprets the quote as part of the string, rather than the end of it. Like this:† "Literal \"string\"" If you build and run with this change, you’ll see the output, with quotes in place: Literal "string" There are several other special characters that you can escape in this way. You can find some common ones listed in Table 10-1. Table 10-1. Common escaped characters for string literals Escaped character Purpose \" Include a double quote in a string literal. \' Include a single quote in a char literal. \\ Insert a backslash. \n New line. \r Carriage return. \t Tab. There are also some rather uncommon ones, listed in Table 10-2. In general, you don’t need to worry about them, but they are quite interesting. † We’ll just show the string literal from here on, rather than repeating the boilerplate code each time. Just replace the string initializer with the example. Literal Strings and Chars | 319 Table 10-2. Less common escape characters for string literals Escaped character Purpose \0 The character represented by the char with value zero (not the character '0'). \a Alert or “Bell”. Back in the dim and distant past, terminals didn’t really have sound, so you couldn’t play a great big .wav file beautifully designed by Robert Fripp every time you wanted to alert the user to the fact that he had done something a bit wrong. Instead, you sent this character to the console, and it beeped at you, or even dinged a real bell (like the line-end on a manual typewriter). It still works today, and on some PCs there’s still a separate speaker just for making this old-school beep. Try it, but be prepared for unexpected retro-side effects like growing enormous sideburns and developing an obsession with disco. \b Backspace. Yes, you can include backspaces in your string. Write: "Hello world\b\b\b\b\bdolly" to the console, and you’ll see: Hello dolly Not all rendering engines support this character, though. You can see the same string rendered in a WPF application in Figure 10-1. Notice how the backspace characters have been ignored. Remember: output mechanisms can interpret individual characters differently, even though they’re the same character, in the same string. \f Form feed. Another special character from yesteryear. This used to push a whole page worth of paper through the printer. This is somewhat less than useful now, though. Even the console doesn’t do what you’d expect. If you write: "Hello\fworld" to the console, you’ll see something like: Hello♀world Yes, that is the symbol for “female” in the middle there. That’s because the original IBM PC defined a special character mapping so that it could use some of these characters to produce graphical symbols (like male, female, heart, club, diamond, and spade) that weren’t part of the regular character set. These mappings are sometimes called code pages, and the default code page for the console (at least for U.S. English systems) incorporates those original IBM definitions. We’ll talk more about code pages and encodings later. \v Vertical quote. This one looks like a “male” symbol (♂) in the console’s IBM-emulating code page. The first character in Table 10-2 is worth a little attention: character value 0, sometimes also referred to as the null character, although it’s not the same as a null reference— char is a value type, so it’s more like the char equivalent of the number 0. In a lot of programming systems, this character is used to mark the end of a string—C and C++ use this convention, as do many Windows APIs. However, in .NET, and therefore in C#, string objects contain the length as a separate field, and so you’re free to put null characters in your strings if you want. However, you may need to be careful—if those 320 | Chapter 10: Strings strings end up being passed to Windows APIs, it’s possible that Windows will ignore everything after the first null. There’s one more escape form that’s a little different from all the others, because you can use it to escape any character. This escape sequence begins with \u and is then followed by four hexadecimal digits, letting you specify the exact numeric value for a character. How can a textual character have a numeric value? Well, we’ll get into that in detail in the “Encoding Characters” on page 360 section, but roughly speaking, each possible character can be identified by number. For example, the uppercase letter A has the number 65, B is 66, and so on. In hexadecimal, those are 41 and 42, respectively. So we can write this string: "\u0041\u0042\u0043" which is equivalent to: "ABC" Of course, if that’s the string you want, you’d normally just write that second form. The \u escape sequence is more useful when you need a particular character that’s not on your keyboard. For example, \u00A9 is the copyright symbol: ©. Sometimes you’ll have a block of text that includes a lot of these special characters (like carriage returns, for instance) and you want to just paste it out of some other application straight into your code as a literal string without having to add lots of backslashes. While it can be done, you might question the wisdom of large quantities of text in your C# source files. You might want to store the text in a separate resource file, and load it up on demand. If you prefix the opening double-quote mark with the @ symbol, the compiler will then interpret every subsequent character (including any whitespace such as newlines, and tabs) as part of the string, until it sees a matching double-quote mark to close the string. Example 10-4 exploits this to embed new lines and indentation in a string literal. Figure 10-1. WPF ignoring control characters Literal Strings and Chars | 321 Example 10-4. Avoiding backslashes with @-quoting string multiLineString = @"Lots of lines and tabs!"; Console.WriteLine(multiLineString); This code will produce the following output: Lots of lines and tabs! Notice how it respects the whitespace between the double quotes. The @ prefix can be especially useful for literal file paths. You don’t need to escape all those backslashes. So instead of writing "C:\\some\\path" you can write just @"c:\some\path". Formatting Data for Output So, we know how to initialize literal strings, which is terribly useful; but what about our other data? How do we display an Int32 or DateTime or whatever? We’ve already met one way of converting any object to a string—the virtual ToString method, which Example 10-5 uses. Example 10-5. Converting numbers to strings with ToString int myValue = 45; string myString = myValue.ToString(); Console.WriteLine(myString); This will produce the output you might expect: 45 What if we try a decimal? Example 10-6 shows this. Example 10-6. Calling ToString on a decimal decimal myValue = 45.65M; string myString = myValue.ToString(); Console.WriteLine(myString); Again, we get the expected output: 45.65 OK, what if we have some decimals in something like an accounting ledger, and we want to format them all to line up properly, with a preceding dollar sign? 322 | Chapter 10: Strings Well, there’s an overload of ToString on each of the numeric types that takes an addi- tional parameter—a format string. Standard Numeric Format Strings In most instances, we’re not dreaming up a brand-new format for our numeric strings; if we were, people probably wouldn’t understand what we meant. Consequently, the framework provides us with a whole bunch of standard numeric format strings, for everyday use. Let’s have a look at them in action. Currency Example 10-7 shows how we format a decimal as a currency value, using an overload of the standard ToString method. Example 10-7. Currency format decimal dollarAmount = 123165.4539M; string text = dollarAmount.ToString("C"); Console.WriteLine(text); The capital C indicates that we want the decimal formatted as if it were a currency value; and here’s the output: $123,165.45 Notice how it has rounded to two decimal places (rounding down in this case), added a comma to group the digits, and inserted a dollar sign for us. Actually, I’ve lied to you a bit. On my machine the output looked like this: £123,165.45 That’s because it is configured for UK English, not U.S. English, and my default currency symbol is the one for pounds sterling. We’ll talk about formatting and globalization a little later in this chapter. That’s the simplest form of this “currency” format. We can also add a number after the C to indicate the number of decimal places we want to use, as Example 10-8 shows. Example 10-8. Specifying decimal places with currency format decimal dollarAmount = 123165.4539M; string text = dollarAmount.ToString("C3"); Console.WriteLine(text); Formatting Data for Output | 323 This will produce three decimal places in the output: $123,165.454 Notice that it is again rounding the result. If you want to truncate, or always round up, you’ll need to round the original value before you convert to a string. This formatting style is available on all of the numeric types. (We’ll see some later that apply to only particular types.) Decimal Decimal formatting is a bit confusingly named, as it actually applies to integer types, not the decimal type. It gets its name from the fact that it displays the number as a string of decimal digits (0–9), with a preceding minus sign (−) if necessary. Example 10-9 uses this format. Example 10-9. Decimal format, with explicit precision int amount = 1654539; string text = amount.ToString("D9"); We’re asking for nine digits in the output string, and it pads with leading zeros: 001654539 If you don’t supply a qualifying number of decimal digits, as Example 10-10 shows, it just uses as many as necessary. Example 10-10. Decimal format with unspecified precision int amount = -2895729; string text = amount.ToString("D"); This produces: −2895729 Hexadecimal Another one for integer types, hexadecimal formatting, shown in Example 10-11, rep- resents numbers as a string of hex digits (0–9, A–F). Example 10-11. Hexadecimal format int amount = 256; string text = amount.ToString("X"); This produces the output: 100 As with the decimal format string, you can specify a number to indicate the total number of digits to which to pad the number, as shown in Example 10-12. 324 | Chapter 10: Strings Example 10-12. Hexadecimal format with explicit precision int amount = 256; string text = amount.ToString("X4"); This produces the output: 0100 Notice that the method doesn’t prepend a 0x or similar; so there is nothing to distin- guish this as a hex string, if you happen to hit a value that does not include the digits A–F. (The convention of preceding hexadecimal values with 0x is common in C family languages, which is why C# supports it for numeric constants, but it’s not universal. VB.NET uses the prefix &H, for example. All .NET languages share the same numeric types and formatting services, so if they printed hex numbers with a C# prefix, that would be annoying for users of other languages. If you want a prefix, you have to add it yourself.) Exponential form All numeric types can be expressed in exponential form. You will probably be familiar with this notation. For example, 1.05 × 103 represents the number 1050, and 1.05 × 10−3 represents the number 0.00105. Developers use plain text editors, which don’t support formatting such as superscript, so there’s a convention for representing exponential numbers with plain, unformatted text. We can write those last two examples as 1.05E+003 and 1.05E-003, respectively. C# recognizes this convention for literal floating-point values. But we can also use it when printing out numbers. To display this form, we use the format string E, with the numeric specifier determining how many decimal places of precision we use. It will always format the result with one digit to the left of the decimal point, so you could also think of the precision specified as “one less than the number of significant figures.” Example 10-13 asks for exponential formatting with four digits of precision. Example 10-13. Exponential format double amount = 254.23875839484; string text = amount.ToString("E4"); And here’s the string it produces: 2.5424E+002 If you don’t provide a precision specifier, as in Example 10-14, you get six digits to the right of the decimal point (or fewer, if the trailing digits would be zero). Formatting Data for Output | 325 We’ll see later how these defaults can be controlled by the framework’s globalization features Example 10-14. Exponential format without precision double amount = 254.23875839484; string text = amount.ToString("E"); This produces: 2.542388E+002 Fixed point Another format string that applies to all numeric types, the fixed-point format provides the ability to display a number with a specific number of digits after the decimal point. As usual, it rounds the result, rather than truncating. Example 10-15 asks for four digits after the decimal point. Example 10-15. Fixed-point format double amount = 152.68385485; string text = amount.ToString("F4"); This produces: 152.6839 The output will be padded with trailing zeros if necessary. Example 10-16 causes this by asking for four digits where only two are required. Example 10-16. Fixed-point format causing trailing zeros double amount = 152.68; string text = amount.ToString("F4"); So, the output in this case is: 152.6800 General Sometimes you want to use fixed point, if possible, but if an occasional result demands a huge number of leading zeros, you’d prefer to fall back on the exponential form (rather than display it as zero, for instance). The “general” format string, illustrated in Exam- ple 10-17, will provide you with this behavior. It is available on all numeric types. 326 | Chapter 10: Strings Example 10-17. General format double amount = 152.68; string text = amount.ToString("G4"); Console.WriteLine(text); double amount2 = 0.00000000000015268; text = amount2.ToString("G4"); Console.WriteLine(text); This will produce the following output: 152.7 1.527E-13 Note that the precision string determines the number of significant figures in either case, not the number of decimal places (as per the fixed-point and exponential forms). As usual, rounding is used if there are more digits than the precision allows. And if you do not specify the precision (i.e., you just use "G") it chooses the number of digits based on the precision of the data you’re using—float will show fewer digits than double, for example. If you don’t specify a particular format string, the default is as though you had specified "G". Numeric The numeric format, shown in Example 10-18, is very similar to the fixed-point format, but adds a “group” separator for values with enough digits (just as the currency format does). The precision specifier can be used to determine the number of decimal places, and rounding is applied if necessary. Example 10-18. Numeric format double amount = 1520494.684848; string text = amount.ToString("N4"); Console.WriteLine(text); This will produce the following output: 1,520,494.6848 Percent Very often you need to display a number as a percentage. However, it’s common to maintain values which represent a percentage using one of the floating-point types, predivided by 100 for ease of future manipulation. Formatting Data for Output | 327 The more mathematically minded among you probably rail against people calling the value 0.58 “a percentage” when they really mean 58%; but it is, unfortunately, a some- what common convention in computer circles. Worse, it’s not consistently applied, making it hard to know whether you are dealing with predivided values, or “true” percentages. It can get especially confusing when you are frequently dealing with values less than 1 percent: double interestRatePercent = 0.2; Is that supposed to be 0.2 percent (like I get on my savings) or 20 percent APR (like my credit card)? One way to avoid ambiguity is to avoid mentioning “percent” in your variable names and always to store values as fractions, representing 100 percent as 1.0, converting into a percentage only when you come to display the number. The percent format is useful if you follow this convention: it will multiply by 100, enabling you to work with ratios internally, but to display them as percentages where necessary. It displays numbers in a fixed-point format, and adds a percentage symbol for you. The precision determines the number of decimal places to use, with the usual rounding method applied. Example 10-19 asks for four decimal places. Example 10-19. Percent format double amount = 0.684848; string text = amount.ToString("P4"); Console.WriteLine(text); This will produce: 68.4848 % Note that this format works with any numeric type—including the integer types. There’s no special treatment for an Int32 or Int16, for example. They are multiplied up by 100, in just the same way as the floating-point types. This means that you can’t format values in increments of less than 100 percent with an integer. For instance, 0 × 100 implies 0 percent, 1 × 100 implies 100 percent, and so on. Round trip The last of the standard numeric format strings we’re going to look at is the round- trip format. This is used when you are expecting the string value to be converted back into its numeric representation at some point in the future, and you want to guarantee no loss of precision. This format has no use for a precision specifier, because by definition, we always want full precision. (You can provide one if you like, because all the standard numeric for- mats follow a common pattern, including an optional precision. This format supports the common syntax rules, it just ignores the precision.) The framework will use the most compact form it can to achieve the round-trip behavior. Example 10-20 shows this format in use. 328 | Chapter 10: Strings Example 10-20. Round-trip format double amount = 0.684848; string text = amount.ToString("R"); Console.WriteLine(text); This produces the following output: 0.684848 Custom Numeric Format Strings You are not limited to the standard forms discussed in the preceding section. You can provide your own custom numeric format strings for additional control over the final output. The basic building blocks of a custom numeric format string are as follows: • The # symbol, which represents an optional digit placeholder; if the digit in this position would have been a leading or trailing 0, it will be omitted. • The 0 symbol, which represents a required digit placeholder; the string is padded with a 0 if the place is not needed. • The . (dot) symbol, which represents the location of the decimal point. • The , (comma) symbol, which performs two roles: it can enable digit grouping, and it can also scale the number down. Look at Example 10-21. Example 10-21. Custom numeric formats double value = 12.3456; Console.WriteLine(value.ToString("00.######")); value = 1.23456; Console.WriteLine(value.ToString("00.000000")); Console.WriteLine(value.ToString("##.000000")); We see the following output: 12.3456 01.234560 1.234560 You don’t actually have to put all the # symbols you require before the decimal place— a single one will suffice; but the placeholders after the decimal point, as shown in Example 10-22, are significant. Example 10-22. Placeholders after the decimal point double value = 1234.5678; text = value.ToString("#.###"); Console.WriteLine(text); Formatting Data for Output | 329 This produces: 1234.568 Notice how it is rounding the result in the usual way. The # symbol will never produce a leading or trailing zero. Take a look at Exam- ple 10-23. Example 10-23. Placeholders and leading or trailing zeros double value = 0.46; string text = value.ToString("#.###"); Console.WriteLine(text); The preceding example will produce the following output: .46 The comma serves two purposes, depending on where you put it. First, it can introduce a separator for showing digits in “groups” of three (so you can easily see the thousands, millions, billions, etc.). We get this behavior when we put a comma between a couple of digit placeholders (the placeholders being either # or 0), as Example 10-24 shows. Example 10-24. Comma for grouping digits int value = 12345678; string text = value.ToString("#,#"); Console.WriteLine(text); Our output string now looks like this: 12,345,678 On the other hand, commas placed just to the left of the decimal point act as a scale on the number. Each comma divides the result by 1,000. Example 10-25 shows two commas, dividing the output by 1,000,000. (It also includes a comma for grouping, although that will not have any effect with this particular value.) Example 10-25. Comma for scaling down output int value = 12345678; string text = value.ToString("#,#,,."); Console.WriteLine(text); This produces: 12 Format strings don’t have to have a decimal point, but you can still use commas to scale the number down even when there’s no decimal point for the commas to be to the left of—they just appear at the end of the format string instead. In effect, there’s an implied decimal point right at the end of the string if you leave it off, so in Example 10-26, the commas are still considered to be to the left of the point even though you can’t see it. 330 | Chapter 10: Strings Example 10-26. Implied decimal point int value = 12345678; string text = value.ToString("#,#,,"); Console.WriteLine(text); Again, this produces: 12 The division rounds the result, so 12745638 would produce 13 with the same formatting. You can also add your own arbitrary text to be included “as is” in the format string, as Example 10-27 shows. Example 10-27. Arbitrary text in a custom format string int value = 12345678; string text = value.ToString("###-### but ###"); Console.WriteLine(text); This time, the output is: 12-345 but 678 Notice how it includes the extra characters we included (the - and the but). Were you expecting the output to be 123-456 but 78? The framework applies the placeholder rule for the lefthand side of the decimal point, so it drops the first nonrequired placeholder, not the last one. Remember that this is a numeric conversion, not something like a telephone-number format. The behavior may be easier to understand if you replace each # with 0. In that case, we’d get 012-345 but 678. Using # just loses the leading zero. If you want to include one of the special formatting characters, you can do so by es- caping it with a backslash. Don’t forget that the C# compiler will attempt to interpret backslash as an escape character in a literal string, but in this case, we don’t want that— we want to include a backslash in the string that we pass to ToString. So unless you are using the @ symbol as a literal string prefix, you’ll need to escape the escape character as Example 10-28 shows. Example 10-28. Escaping characters in a custom format string int value = 12345678; string text = value.ToString("###-### \\# ###"); Console.WriteLine(text); Example 10-29 shows the @-quoted equivalent. Formatting Data for Output | 331 Example 10-29. @-quoting a custom format string int value = 12345678; string text = value.ToString(@"###-### \# ###"); Console.WriteLine(text); Both will produce this output: 12-345 # 678 You can also include literal strings (with or without special characters), by wrapping them in single quotes as Example 10-30 shows. Example 10-30. Literal string in a custom format string int value = 12345678; string text = value.ToString(@"###-### \# ### 'is a number'"); Console.WriteLine(text); This produces the output: 12-345 # 678 is a number Finally, you can also get the multiply-by-100 behavior for predivided percentage values using the % symbol, as shown in Example 10-31. Example 10-31. Percentage in a custom format string double value = 0.95; string text = value.ToString("#0.##%"); Console.WriteLine(text); Notice that this also includes the percentage symbol in the output: 95% There is also a per-thousand (per-mille) symbol (‰), which is Unicode character 2030. You can use this in the same way as the percentage symbol, but it multiplies up by 1,000. We’ll learn more about Unicode characters later in this chapter. Dates and Times It is not just numeric types that support formatting when they are converted to strings. The DateTime, DateTimeOffset, and TimeSpan types follow a similar pattern. DateTimeOffset is generally the preferred way to represent a particular point in time inside a program, because it builds in information about the time zone (and daylight saving if applicable), leaving no scope for ambiguity regarding the time it represents. However, DateTime is a more natural way to present times to users, partly because it has more scope for ambiguity. People very rarely explicitly say what time zone they’re thinking of—we’re used to learning that a shop opens at 9:00 a.m., or that our flight 332 | Chapter 10: Strings is due to arrive at 8:30 p.m. DateTime lives in this same slightly fuzzy world, where 9:00 a.m. is, in some sense, the same time before and after daylight saving comes into effect. So if you have a DateTimeOffset that you wish to display, unless you want to show the time zone information in the user interface, you will most likely convert it to a DateTime that’s relative to the local time zone, as Example 10-32 shows. Example 10-32. Preparing to present a DateTimeOffset to the user DateTimeOffset tmo = GetTimeFromSomewhere(); DateTime localDateTime = tmo.ToLocalTime().DateTime; There are two benefits to this. First, this gets the time into a representation likely to align with how end users normally think of times, that is, relative to whatever time zone they’re in right now. Second, DateTime makes formatting slightly easier than DateTimeOffset: DateTimeOffset supports the same ToString formats as DateTime, but DateTime offers some additional convenient methods. First, DateTime offers an overload of the ToString method which can accept a range of standard format strings. Some of the more popular ones (such as d, the short date format, and D, the long date format) are also exposed as methods. Example 10-33 il- lustrates this. Example 10-33. Showing the date in various formats DateTime time = new DateTime(2001, 12, 24, 13, 14, 15, 16); Console.WriteLine(time.ToString("d")); Console.WriteLine(time.ToShortDateString()); Console.WriteLine(time.ToString("D")); Console.WriteLine(time.ToLongDateString()); This produces: 12/24/2001 12/24/2001 24 December 2001 24 December 2001 There are also format strings and methods for the time part only, as Example 10-34 shows. Example 10-34. Getting just the time DateTime time = new DateTime(2001, 12, 24, 13, 14, 15, 16); Console.WriteLine(time.ToString("t")); Console.WriteLine(time.ToShortTimeString()); Console.WriteLine(time.ToString("T")); Console.WriteLine(time.ToLongTimeString()); Formatting Data for Output | 333 This will result in: 13:14 13:14 13:14:15 13:14:15 Or, as Example 10-35 shows, you can combine the two. Example 10-35. Getting both the time and date DateTime time = new DateTime(2001, 12, 24, 13, 14, 15, 16); Console.WriteLine(time.ToString("g")); Console.WriteLine(time.ToString("G")); Console.WriteLine(time.ToString("f")); Console.WriteLine(time.ToString("F")); Notice how the upper- and lowercase versions of all these standard formats are used to choose between the short and long time formats: 24/12/2001 13:14 24/12/2001 13:14:15 24 December 2001 13:14 24 December 2001 13:14:15 Another common format is the “round trip” shown in Example 10-36. As for the nu- meric types, this is designed for scenarios where you expect to convert both to and from strings, without loss of precision. Example 10-36. Round-trip DateTime format DateTime time = new DateTime(2001, 12, 24, 13, 14, 15, 16); Console.WriteLine(time.ToString("O")); This produces: 2001-12-24T13:14:15.0160000 (If you use a DateTimeOffset, this last format will add the time zone on the end; for example, +01:00 would indicate that the time is from a zone one hour ahead of UTC.) This round-trip format is sortable using standard string precedence rules. Another for- mat with that characteristic is the universal sortable form, shown in Example 10-37. This converts from local time to UTC before doing the format. Example 10-37. Universal sortable format DateTime time = new DateTime(2001, 12, 24, 13, 14, 15, 16); Console.WriteLine(time.ToString("u")); Because I am currently in the GMT time zone, and daylight saving is not in operation, I am at an offset of zero from UTC, so no apparent conversion takes place. But note the suffix Z which indicates a UTC time: 2001-12-24 13:14:15Z 334 | Chapter 10: Strings Dealing with dates and times is notoriously difficult, especially if you have to manage multiple time zones in a single application. There is no “silver bullet” solution. Even using DateTimeOffset internally and con- verting to local time for output is not necessarily a complete solution. You must beware of hidden problems like times that don’t exist (because we skipped forward an hour when we applied daylight saving time), or exist twice (because we skipped back an hour when we left daylight saving time). As with the numeric conversions, you also have the option of custom format strings. The key components are: d: day M: month (note that this is uppercase to distinguish it from m for minute) y: year h: hour (12-hour format) H: hour (24-hour format) m: minute s: seconds f: fractions of a second The / character will be substituted with the appropriate date separator for your locale, and : with the time separator. You can repeat the substitution character to obtain shorter/longer forms of the relevant part of the date or time. For example, you can format the day part like Example 10-38 does. Example 10-38. Formatting the day DateTime time = new DateTime(2001, 12, 24, 13, 14, 15, 16); Console.WriteLine(time.ToString("dddd")); Console.WriteLine(time.ToString("ddd")); Console.WriteLine(time.ToString("dd")); This will produce: Monday Mon 24 (As you saw in Example 10-33, a single d means something else: it shows the whole date, in short form.) Other useful formatting characters include: z: offset from UTC (with zzz providing hours and minutes) tt: the a.m./p.m. designator As with the numeric formats, you can also include string literals, escaping special char- acters in the usual way. Formatting Data for Output | 335 Going the Other Way: Converting Strings to Other Types Now that we know how to control the formatting of various types when we convert them to a string, let’s take a step aside for a moment to look at converting back. If we’ve got a string, how do we convert that to a numeric type, for instance? Probably the easiest way is to use the static methods on the Convert class, as Exam- ple 10-39 shows. Example 10-39. Converting a string to an int int converted = Convert.ToInt32("35"); This class also supports numeric conversions from a variety of different bases (specif- ically 2, 8, 10, and 16), shown in Example 10-40. Example 10-40. Converting hexadecimal strings to ints int converted = Convert.ToInt32("35", 16); int converted = Convert.ToInt32("0xFF", 16); Although we get to specify the base as a number, only binary, octal, decimal, and hex- adecimal are actually supported. If you request any other base (e.g., 7) the method will throw an ArgumentException. What happens if we pass a string that doesn’t represent an instance of the type to which we want to convert, as Example 10-41 does? Example 10-41. Attempting to convert a nonnumeric string to a number double converted = Convert.ToDouble("Well, what do you think?"); As this string cannot be converted to a double, we see a FormatException. Throwing (and catching) exceptions is a relatively expensive operation, and sometimes we want to try a particular conversion, then, if it fails, try another. We’d rather not pay for the exception if we don’t have to. Fortunately, the individual numeric types (and DateTime) give us the means to do this. Instead of using Convert, we can use the various TryParse methods they provide. Rather than returning the parsed value, it returns a bool which indicates whether the parse was successful. The parsed value is retrieved via an out parameter. Exam- ple 10-42 shows that in use. Example 10-42. Avoiding exceptions with TryParse int parsed; if (!int.TryParse("Well, how about that", out parsed)) { Console.WriteLine("That didn't parse"); } 336 | Chapter 10: Strings For each of the TryParse methods, there is an equivalent Parse, which throws a FormatException on failure and returns the parsed value on success. For many appli- cations, you can use these as an alternative to the Convert methods. Some parse methods can also offer you additional control over the process. Date Time.ParseExact, for example, allows you to provide an exact format specification for the date/time string, as Example 10-43 shows. Example 10-43. DateTime.ParseExact DateTime dt = DateTime.ParseExact("12^04^2008","dd^MM^yyyy",CultureInfo.CurrentCulture); This can be useful if you expect a nonstandard format for your string, coming from a legacy system, perhaps. Composite Formatting with String.Format The previous examples have all turned exactly one piece of information into a single string (or vice versa). Very often, though, we need to compose multiple pieces of in- formation into our final output string, with different conversions for each part. We could do that by composing strings (something we’ll look at later in this chapter), but it is often more convenient to use a helper method: String.Format. Example 10-44 shows a basic example. Example 10-44. Basic use of String.Format int val1 = 32; double val2 = 123.457; DateTime val3 = new DateTime(1999, 11, 1, 17, 22, 25); string formattedString = String.Format("Val1: {0}, Val2: {1}, Val3: {2}", val1, val2, val3); Console.WriteLine(formattedString); This method takes a format string, plus a variable number of additional parameters. Those additional parameters are substituted into the format string where indicated by a format item. At its simplest, a format item is just an index into the additional parameter array, enclosed in braces (e.g., {0}). The preceding code will therefore produce the following output: Val1: 32, Val2: 123.457, Val3: 01/11/1999 17:22:25 A specific format item can be referenced multiple times, and in any order in the format string. You can also apply the standard and custom formatting we discussed earlier to any of the individual format items. Example 10-45 shows that in action. Example 10-45. Using format strings from String.Format int first = 32; double second = 123.457; DateTime third = new DateTime(1999, 11, 1, 17, 22, 25); Formatting Data for Output | 337 string output = String.Format( "Date: {2:d}, Time: {2:t}, Val1: {0}, Val2: {1:#.##}", first, second, third); Console.WriteLine(output); Notice the colon after the index, followed by the simple or custom formatting string, which transforms the output: Date: 01/11/1999, Time: 17:22, Val1: 32, Val2: 123.46 String.Format is a very powerful technique, but you should be aware that there is some overhead in its use with value types. The additional parameters take the form of an array of objects (so that we can pass in any type for each format item). This means that the values passed in are boxed, and then unboxed. For many applications this overhead will be irrelevant, but, as always, you should measure and be aware of the hidden cost. Culture Sensitivity Up to this point, we’ve quietly ignored a significantly complicating factor in string manipulation: the fact that the rules for text vary considerably among cultures. There are also lots of different types of rules in operation, from the characters to use for particular types of separators, to the natural sorting order for characters and strings. I’ve already called out an example where the output on my UK English machine was different from that on a U.S. English computer. As another very simple example, the decimal number we write as 1.8 in U.S. or UK English would be written 1,8 in French. For the .NET Framework, these rules are encapsulated in an object of the type System.Globalization.CultureInfo. The CultureInfo class makes certain commonly used cultures accessible through static properties. CurrentCulture returns the default culture, used by all the culture-sensitive methods if you don’t supply a specific culture to a suitable overload. This value can be controlled on a per-thread basis, and defaults to the Windows default user locale. An- other per-thread value is the CurrentUICulture. By default, this is based on the current user’s personally selected preferred language, falling back on the operating system de- fault if the user hasn’t selected anything. This culture determines which resources the system uses when looking up localized resources such as strings. CurrentCulture and CurrentUICulture may sound very similar, but are often different. For example, Microsoft does not provide a version of Windows translated into British English—Windows offers British users “Favorites” and “Colors” despite a national tendency to spell those words as “Favourites” and “Colours.” But we do have the option to ask for UK conventions for dates and currency, in which case CurrentCul ture and CurrentUICulture will be British English and U.S. English, respectively. 338 | Chapter 10: Strings Finally, it’s sometimes useful to ensure that your code always behaves the same way, regardless of the user’s culture settings. For example, if you’re formatting (or parsing) text for persistent storage, you might need to read the text on a machine configured for a culture other than that on which it was created, and you will want to ensure that it is interpreted correctly. If you rely on the current culture, dates written out on a UK machine will be processed incorrectly on U.S. machines because the month and day are reversed. (In the UK, 3/12/2010 is a date in December.) The InvariantCulture property returns a culture with rules which will not vary with different installed or user- selected cultures. If you’ve been looking at the IntelliSense as we’ve been building the string format examples in this chapter, you might have noticed that none of the obviously culture-sensitive methods seem to offer an overload which takes a CultureInfo. However, on closer examination, you’ll no- tice that CultureInfo also implements the IFormatProvider interface. All of the formatting methods we’ve looked at do provide an overload which takes an instance of an object which implements IFormatProvider. Prob- lem solved! You can also create a CultureInfo object for a specific culture, by providing that cul- ture’s canonical name to the CreateSpecificCulture method on the CultureInfo object. But what are the canonical names? You may have come across some of them in the past. UK English, for instance, is en-GB, and French is fr. Example 10-46 gets a list of all the known canonical names by calling another method on CultureInfo that lists all the cultures the system knows about: GetCultures. Example 10-46. Showing available cultures var cultures = CultureInfo.GetCultures(CultureTypes.AllCultures). OrderBy(c => c.EnglishName); foreach (var culture in cultures) { Console.WriteLine("{0} : {1}", culture.EnglishName, culture.Name); } We won’t reproduce the output here, because it is a bit long. This is a short excerpt: English (United Kingdom) : en-GB English (United States) : en-US English (Zimbabwe) : en-ZW Estonian : et Estonian (Estonia) : et-EE Faroese : fo Faroese (Faroe Islands) : fo-FO Filipino : fil Filipino (Philippines) : fil-PH Finnish : fi Finnish (Finland) : fi-FI French : fr Culture Sensitivity | 339 Notice that we’re showing the English version of the name, followed by the canonical name for the culture. Example 10-47 illustrates a difference in string formatting between two different cultures. Example 10-47. Formatting numbers for different cultures CultureInfo englishUS = CultureInfo.CreateSpecificCulture("en-US"); CultureInfo french = CultureInfo.CreateSpecificCulture("fr"); double value = 1.8; Console.WriteLine(value.ToString(englishUS)); Console.WriteLine(value.ToString(french)); This will produce the output we’d expect: 1.8 1,8 Exploring Formatting Rules If you look at the CultureInfo class, you’ll see numerous properties, some of which define the culture’s rules for formatting particular kinds of information. For example, there are the DateTimeFormat and NumberFormat properties. These are instances of Date TimeFormatInfo and NumberFormatInfo, respectively, and expose a large number of properties with which you can control the formatting rules for the relevant types. These types also implement IFormatProvider, so you can use these types to provide your own custom formatting rules to the string formatting methods we looked at earlier. Example 10-48 formats a number in an unusual way. Example 10-48. Modifying the decimal separator double value = 1.8; NumberFormatInfo nfi = new NumberFormatInfo(); nfi.NumberDecimalSeparator = "^"; Console.WriteLine(value.ToString(nfi)); Here we use the NumberFormatInfo to change the decimal separator to the circumflex (hat) symbol. The resultant output is: 1^8 You can use this to control all sorts of features of the formatting engine, such as the default precision, percentage and positive/negative symbols, and separators. Now that we know how to format strings of various kinds, we’ll go back to looking at some of the features of the string itself. In particular, we’ll look at how to slice and dice an existing string in various ways. 340 | Chapter 10: Strings Accessing Characters by Index Earlier, we saw how to enumerate the characters in a string; however, we often want to be able to retrieve a character at a particular offset into the string. String defines an indexer, so we can do just that. Example 10-49 uses the indexer to retrieve the character at a particular (zero-based) index in the string. Example 10-49. Retrieving characters with a string’s indexer string myString = "Indexing"; char theThirdCharacter = myString[2]; Console.WriteLine(theThirdCharacter); If you execute that code in a console application, you’ll see: d What if we try to use the indexer to assign a value (i.e., to replace the character at that location in the string) as in Example 10-50? Example 10-50. Trying to assign a value with a string’s indexer string myString = "Indexing"; myString[2] = 'f'; // Will fail to compile Well, that doesn’t compile. We get an error: Property or indexer 'string.this[int]' cannot be assigned to -- it is read only So, the indexer is read-only. This is a part of a very important constraint on a String object. Strings Are Immutable Once a string has been created, it is immutable. You can’t slice it up into substrings, trim characters off it, add characters to it, or replace one character or substring with another. “What?” I hear you ask. “Then how are we supposed to do our string processing?” Don’t worry, you can still do all of those things, but they don’t affect the original string—copies (of the relevant pieces) are made instead. Why did the designers of the .NET Framework make strings immutable? All that copy- ing is surely going to be an overhead. Well, yes, it is, and sometimes you need to be aware of it. That being said, there are balancing performance improvements when dealing with unchanging strings. The framework can store a single instance of a string and then any variables that reference that particular sequence of characters can reference the same instance. This can actually save on allocations and reduce your working set. And in multithreaded scenarios, the fact that strings never change means it’s safe to use them Strings Are Immutable | 341 without the cross-thread coordination that is required when accessing modifiable data. As usual, “performance” considerations are largely a compromise between the com- peting needs of various possible scenarios. In our view, an overridingly persuasive argument for immutability relates to the safe use of strings as keys. Consider the code in Example 10-51. Example 10-51. Using strings as keys in a dictionary string myKey = "TheUniqueKey"; Dictionary myDictionary = new Dictionary(); myDictionary.Add(myKey, new object()); // Imagine you could do this... myKey[2] = 'o'; Remember, a string is a reference type, so the myKey variable references a string object which is initialized to "TheUniqueKey". When we add our object to the dictionary, we pass a reference to that same string object, which the dictionary will use as a key. If you cast your mind back to Chapter 9, you’ll remember that the dictionary relies on the hash code for the key object when storing dictionary entries, which can then be dis- ambiguated (if necessary) by the actual value of the key itself. Now, imagine that we could modify the original string object, using the reference we hold in that myKey variable. One characteristic of a (useful!) hash algorithm is that its output changes for any change in the original data. So all of a sudden our key’s hash code has changed. The hash for "TheUniqueKey" is different from the one for "ThoUnique Key". Sadly, the dictionary has no way of knowing that the hash for that key has changed; so, when we come to look up the value using our original reference to our key, it will no longer find a match. This can (and does!) cause all sorts of subtle bugs in applications built on runtimes that allow mutable strings. But since .NET strings are immutable, this problem cannot occur if you use strings as keys. Another, related, benefit is that you avoid the buffer-overrun issues so prevalent on other runtimes. Because you can’t modify an existing string, you can’t accidentally run over the end of your allocation and start stamping on other memory, causing crashes at best and security holes at worst. Of course, immutable strings are not the only way the .NET designers could have addressed this problem, but they do offer a very simple solution that helps the developer fall naturally into doing the right thing, without having to think about it. We think that this is a very neat piece of design. So, we can obtain (i.e., read) a character at a particular index in the string, using the square-bracket indexer syntax. What about slicing and dicing the string in other ways? 342 | Chapter 10: Strings Getting a Range of Characters You can obtain a contiguous range of characters within a string by using the Substring method. There are a couple of overloads of this method, and Exam- ple 10-52 shows them in action. Example 10-52. Using Substring string myString = "This is the silliest stuff that ere I heard."; string subString = myString.Substring(5); string anotherSubString = myString.Substring(12, 8); Console.WriteLine(subString); Console.WriteLine(anotherSubString); Notice that both of these overloads return a new string, containing the relevant portion of the original string. The first overload starts with the character at the specified index, and returns the rest of the string (regardless of how long it might be). The second starts at the specified index, and returns as many characters as are requested. A very common requirement is to get the last few characters from a string. Many plat- forms have this as a built-in function, or feature of their strings, but the .NET Frame- work leaves you to do it yourself. To do so depends on us knowing how many characters there are in the string, subtracting the offset from the end, and using that as our starting index, as Example 10-53 shows. Example 10-53. Getting characters from the righthand end of a string static string Right(string s, int length) { int startIndex = s.Length - length; return s.Substring(startIndex); } Notice how we’re using the Length property on the string to determine the total number of characters in the string, and then returning the substring from that offset (to the end). We could then use this method to take the last six characters of our string, as Exam- ple 10-54 does. Example 10-54. Using our Right method string myString = "This is the silliest stuff that ere I heard."; string subString = Right(myString, 6); Console.WriteLine(subString); If you build and run this sample, you’ll see the following output: heard. Getting a Range of Characters | 343 Extension Methods for String You will probably build up an armory of useful methods for dealing with strings. It can be helpful to aggregate them together into a set of extension methods. Here’s an example implementing the Right method that we’ve used as an example in this chapter, but modifying it to work as an extension method, and also providing an equivalent to the version of Substring that takes both a start position and a length: public static class StringExtensions { public static string Right(this string s, int length) { int startIndex = s.Length - length; return s.Substring(startIndex); } public static string Right(this string s, int offset, int length) { int startIndex = s.Length - offset; return s.Substring(startIndex, length); } } By implementing them as extension methods, we can now write code like this: string myString = "This is the silliest stuff that ere I heard."; string subString = myString.Right(6); string subString2 = myString.Right(6, 5); Console.WriteLine(subString); Console.WriteLine(subString2); This will produce output like the following: heard. heard Notice that the Length of the string is the total number of characters in the string— much as the length of an array is the total number of entities in the array, not the number of bytes allocated to it (for example). Composing Strings You can create a new string by composing one or more other strings. Example 10-55 shows one way to do this. Example 10-55. Concatenating strings string fragment1 = "To be, "; string fragment2 = "or not to be."; string composedString = fragment1 + fragment2; Console.WriteLine(composedString); 344 | Chapter 10: Strings Here, we’ve used the + operator to concatenate two strings. The C# compiler turns this into a call to the String class’s static method Concat, so Example 10-56 shows the equivalent code. Example 10-56. Calling String.Concat explicitly string composedString2 = String.Concat(fragment1, fragment2); Console.WriteLine(composedString2); Don’t forget—we’re taking the first two strings, and then creating a new string that is fragment1.Length + fragment2.Length characters long. The original strings remain unchanged. There are several overloads of Concat, all taking various numbers of strings—this ena- bles you to concatenate multiple strings in a single step without producing intermediate strings. One of the overloads, used in Example 10-57, can concatenate an entire array of strings. Example 10-57. Concatenating an array of strings static void Main(string[] args) { string[] strings = Soliloquize(); string output = String.Concat(strings); Console.WriteLine(output); Console.ReadKey(); } private static string[] Soliloquize() { return new string[] { "To be, or not to be--that is the question:", "Whether 'tis nobler in the mind to suffer", "The slings and arrows of outrageous fortune", "Or to take arms against a sea of troubles", "And by opposing end them." }; } If we build and run that example, we’ll see some output like this: To be, or not to be--that is the question:Whether 'tis nobler in the mind to suf ferThe slings and arrows of outrageous fortuneOr to take arms against a sea of t roublesAnd by opposing end them. That’s probably not quite what we meant. We’ve been provided with each line of Hamlet’s soliloquy, and we really want the single output string to have breaks after each line. Instead of using String.Concat, we can instead use String.Join to concatenate all of the strings as shown in Example 10-58. This lets us insert the string of our choice between each string. Composing Strings | 345 Example 10-58. String.Join static void Main(string[] args) { string[] strings = Soliloquize(); string output = String.Join(Environment.NewLine, strings); Console.WriteLine(output); Console.ReadKey(); } Here we’re using the Environment.NewLine constant to get the line-break string appro- priate for our platform (rather than explicitly using "\n" or "\r" or "\r\n"). For historical reasons, not all operating systems use the same sequence of characters to represent the end of a line. Windows (like DOS before it) mimics old-fashioned printers, where you had to send two control characters: a carriage return (ASCII value 13, or \r in a string or char- acter literal) would cause the print head to move back to the beginning of the line, and then a line feed (ASCII 10, or \n) would advance the paper up by one line. This meant you could send a text file directly to a printer without modification and it would print correctly, but it pro- duced the slightly clumsy situation of requiring two characters to denote the end of a line. Unix conventionally uses just a single line feed to mark the end of a line. Environment.NewLine is offered so that you don’t have to assume that you’re running on a particular platform. That being said, Console is flexible, and treats either convention as a line end. But this can matter if you’re saving files to disk. If we build and run, we’ll see the following output: To be, or not to be--that is the question: Whether 'tis nobler in the mind to suffer The slings and arrows of outrageous fortune Or to take arms against a sea of troubles And by opposing end them. Splitting It Up Again As well as joining text up, we can also split it up into smaller pieces at a particular breaking string or character. For example, we could split the final concatenated string back up at whitespace or punctuation as in Example 10-59. Example 10-59. Splitting a string string[] strings = Soliloquize(); string output = String.Join(Environment.NewLine, strings); string[] splitStrings = output.Split( new char[] { ' ', '\t', '\r', '\n', ',', '-', ':' }); bool first = true; 346 | Chapter 10: Strings foreach (string splitBit in splitStrings) { if( first ) { first = false; } else { Console.Write(", "); } Console.Write(splitBit); } If we run again, we see the following output: To, be, , or, not, to, be, , that, is, the, question, , , Whether, 'tis, nobler, in, the, mind, to, suffer, , The, slings, and, arrows, of, outrageous, fortune, , Or, to, take, arms, against, a, sea, of, troubles, , And, by, opposing, end, them. Notice how our separation characters were not included in the final output, but we do seem to have some “blanks” (which are showing up here as multiple commas in a row with nothing in between). These empty entries occur when you have multiple consec- utive separation characters, and, most often, you would rather not have to deal with them. The Split method offers an overload that takes an additional parameter of type StringSplitOptions, shown in Example 10-60, which lets us eliminate these empty entries. Example 10-60. Eliminating empty strings in String.Split string[] splitStrings = output.Split( new char[] { ' ', '\t', '\r', '\n', ',', '-', ':' }, StringSplitOptions.RemoveEmptyEntries); Our output is now the more manageable: To, be, or, not, to, be, that, is, the, question, Whether, 'tis, nobler, in, the , mind, to, suffer, The, slings, and, arrows, of, outrageous, fortune, Or, to, t ake, arms, against, a, sea, of, troubles, And, by, opposing, end, them. Upper- and Lowercase Some of the words in that output list originally appeared at the beginning of a line, and therefore have an initial uppercase letter, while others were in the body of a line, and are therefore entirely lowercase. In our output, it might be nicer if we represented them all consistently (in lower case, for example). This is easily achieved with the ToUpper and ToLower members of String. We can change our output line to the code shown in Example 10-61. Example 10-61. Forcing strings to lowercase Console.Write(splitBit.ToLower()); Composing Strings | 347 Our output is now consistently lowercase: to, be, or, not, to, be, that, is, the, question, whether, 'tis, nobler, in, the , mind, to, suffer, the, slings, and, arrows, of, outrageous, fortune, or, to, t ake, arms, against, a, sea, of, troubles, and, by, opposing, end, them. Upper- and lowercase rules vary considerably among cultures, and you should be cautious when using ToUpper and ToLower for this purpose. For culture-insensitive scenarios, there are also methods called ToUpper Invariant and ToLowerInvariant whose results are not affected by the current culture. MSDN provides a considerable amount of resources devoted to culture-sensitive string operations. A good starting point can be found here: http://msdn.microsoft.com/en-us/library/5bz7d2f8 Manipulating Text The result of the preceding section was nice and neat; but what if our array of strings had come from a user? Users have a tendency to whack the Return key a few times before they write anything at all, and add spurious spaces and tabs to the beginning and end of lines, particularly when copying and pasting between applications. They might also add commas or periods or something like that, again in the interest of tidi- ness. They might spell things incorrectly. There’s no accounting for what users might do. Let’s simulate that with a new function shown in Example 10-62. Example 10-62. Simulating messy input private static string[] SoliloquizeLikeAUser() { return new string[] { "", null, " ", String.Empty, " To be, or not to be--that is the question: ", "Whether 'tis nobelr in the mind to suffer,", "\tThe slings and arrows of outrageous fortune ,", "", "\tOr to take arms against a sea of troubles, ", "And by opposing end them.", "", "", "", "", ""}; } 348 | Chapter 10: Strings Notice their extensive use of the Return key, the tendency to put the odd comma at the end of the line, and the occasional whack of the Tab key at the beginning of lines. Sadly, if we use this function and then print the output using String.Concat like we did in Example 10-57, we end up with output like this: To be, or not to be--that is the question: Whether 'tis nobelr in the mind to suffer, The slings and arrows of outrageous fortune , Or to take arms against a sea of troubles, And by opposing end them. We can write some code to tidy this up. We can build up our output string, concate- nating the various strings, and cleaning it up as we go. This is going to involve iterating through our array of strings, inspecting them, perhaps transforming them, and then appending them to our resultant string. Example 10-63 shows how we could structure this, although it does not yet include any of the actual cleanup code. Example 10-63. Cleaning up input string[] strings = SoliloquizeLikeAUser(); string output = String.Empty; // This is equivalent to "" foreach (string line in strings) { // Do something to look at the line... // then... output = output + line + Environment.NewLine; } Console.WriteLine(output); This would work just fine; but look at what happens every time we go round the loop. We create a new string and store a reference to it in output, throwing away whatever was in output before. That’s potentially very wasteful of resources, if we do this a lot. Fortunately, the .NET Framework provides us with another type we can use for pre- cisely these circumstances: StringBuilder. Mutable Strings with StringBuilder Having said that a String is immutable, we are now going to look at a class that is very, very much like a string, and yet it can be modified. Example 10-64 shows it in action. Manipulating Text | 349 Example 10-64. Building up strings with StringBuilder string[] strings = SoliloquizeLikeAUser(); StringBuilder output = new StringBuilder(); foreach (string line in strings) { // Do something to look at the line... // then... output.AppendLine(line); } Console.WriteLine(output.ToString()); After we’ve retrieved our array of strings, we create an (empty) instance of a StringBuilder. For each string in our array, we then call the AppendLine method to append the string, along with a suitable line-end character. Notice that we don’t keep creating new instances of the StringBuilder as we go along. Instead, it automatically handles the job of allocating an appropriate amount of internal storage and appending each new string we pass it. When we construct the StringBuilder, it allocates a chunk of memory in which we can build the string—initially it allocates enough space for 16 characters. If we append something that would make the string too long to fit, it allocates a new chunk of mem- ory. Crucially, it allocates more than it needs, the idea being to have enough spare space to satisfy a few more appends without needing to allocate yet another chunk of memory. The precise details of the allocation strategy are not documented, but we’ll see it in action shortly. In an ideal world, we would avoid overallocating, and avoid repeatedly having to allo- cate more space. If we have some way of knowing in advance how long the final string will be, we can do this, because we can specify the initial capacity of the StringBuilder in its constructor. Example 10-65 illustrates the effect. Example 10-65. Capacity versus Length StringBuilder builder1 = new StringBuilder(); StringBuilder builder2 = new StringBuilder(1024); Console.WriteLine(builder1.Capacity); Console.WriteLine(builder1.Length); Console.WriteLine(builder2.Capacity); Console.WriteLine(builder1.Length); This would produce the output: 16 0 1024 0 350 | Chapter 10: Strings Notice how we’re using the Capacity to see how many characters we could have in the StringBuilder, and the Length to determine how many we do have. We can now append some content to these two strings, as Example 10-66 shows. Example 10-66. Exploring capacity StringBuilder builder1 = new StringBuilder(); StringBuilder builder2 = new StringBuilder(1024); Console.WriteLine(builder1.Capacity); Console.WriteLine(builder1.Length); Console.WriteLine(builder2.Capacity); Console.WriteLine(builder2.Length); builder1.Append('A', 24); builder2.Append('A', 24); Console.WriteLine(builder1.Capacity); Console.WriteLine(builder1.Length); Console.WriteLine(builder2.Capacity); Console.WriteLine(builder2.Length); We’re using a different overload of the Append method on StringBuilder. This one takes a Char as its first parameter, and then a repeat count. So, in each case, we append a string with 24 As. If we run this, we get the output: 16 0 1024 0 32 24 1024 24 The first four lines are the same as before, but now we see that the capacity of the first StringBuilder has increased to 32 characters, and the string it holds is 24 characters long. The second StringBuilder has retained its capacity of 1,024 characters, because that was plenty to hold the 24 characters we appended. What if we append another 12 characters to that first StringBuilder, as Exam- ple 10-67 shows? Example 10-67. Appending more text builder1.Append('B', 12); Console.WriteLine(builder1.Capacity); Console.WriteLine(builder1.Length); The additional two lines of output look like this: 64 36 Manipulating Text | 351 We’ve gone from a capacity of 16 to 32 to 64 characters. OK; can you guess what happens if we append another 30 characters (to push ourselves over the 64-character limit) as Example 10-68 does? Example 10-68. Appending yet more text builder1.Append('C', 30); Console.WriteLine(builder1.Capacity); Console.WriteLine(builder1.Length); Yup, the last two lines of output now read: 128 66 There is a geometric progression—the capacity is doubling each time we exceed its previous capacity. It does this in an attempt to minimize the amount of allocation it has to do, but in order to prevent things from getting totally out of hand, overallocation will never grow the capacity by more than 8,000 characters (in the current version of the framework, at least). Of course, if you append a string that is longer than 8,000 characters, StringBuilder will have to allocate enough space, but it won’t overallocate in that case. You may have noticed that in the preceding examples, the String Builder had to reallocate each time we called Append. How is that any better than just appending strings? Well, it isn’t, but that’s only because we deliberately contrived the examples to show what happens when you exceed the capacity. You won’t usually see such optimally bad behavior—in practice, you’ll see fewer allocations than appends. If we know we’re going to need a particular amount of space, we can manually ensure that the builder has appropriate capacity, as shown in Example 10-69. Example 10-69. Ensuring capacity builder1.EnsureCapacity(32000); Console.WriteLine(builder1.Capacity); Console.WriteLine(builder1.Length); The last two lines of output indicate that it has complied with our wishes: 32000 66 What if we then call EnsureCapacity with a smaller number? Example 10-70 tries to do this. 352 | Chapter 10: Strings Example 10-70. Attempting to reduce capacity builder1.EnsureCapacity(70); Console.WriteLine(builder1.Capacity); Console.WriteLine(builder1.Length); Here’s the output: 32000 66 Nope—it doesn’t reduce the capacity. EnsureCapacity only guarantees that the capacity is at least what you ask for, so it does nothing if there’s more than you need. OK, so StringBuilder is going to accumulate the string for us, making sure there’s enough space as we go along. What about finishing off the method we were writing so that it tidies up that user input as it goes along? The first thing we’d like to do is to correct that mistake where the user seems to have misspelled “nobler” as “nobelr”. Finding and Replacing Content Find-and-replace is a very common requirement when processing strings. Fortunately, the .NET Framework provides us with a couple of options. If we just want to find a piece of text we can use one of several overloads of the IndexOf method. This takes some text for which to look, and an index at which to start looking. By calling the method repeatedly, using the last index returned as the basis of the start index for the next search, we can find all instances of the relevant text in the input string, as Example 10-71 shows. Example 10-71. Searching for text string inputString = "If a dog and a man go into a bar, " + "is it necessarily the beginning of a joke?"; int index = −1; do { index += 1; index = inputString.IndexOf(" a ", index); Console.WriteLine(index); } while (index >= 0); This produces the output: 2 12 26 68 −1 Finding and Replacing Content | 353 Notice how the method returns −1 when it cannot find a further match. That’s finding content. What we really want to be able to do is to replace content, though. As you might expect, string also offers us a Replace function, which is shown in Example 10-72. Example 10-72. Replacing text string original = "Original text."; string replaced = original.Replace("Original", "Replaced"); Console.WriteLine(original); Console.WriteLine(replaced); This takes any match for the first parameter found in the source, and replaces it with the text in the second parameter. In this case, the output looks like this: Original text. Replaced text. As you know, strings are immutable, so Replace creates a new string containing our substitutions. Replace offers no control over how many replacements to make, and from where to start the replacement; both of these are common requirements in text processing. For- tunately, StringBuilder has a family of Replace methods which address all of these issues, performing an in-place replace with optional start index and number of re- placements to make. Remember that we had the code shown in Example 10-73. Example 10-73. Code from earlier for tidying up the text string[] strings = SoliloquizeLikeAUser(); StringBuilder output = new StringBuilder(); foreach (string line in strings) { // Do something to look at the line... // then... output.AppendLine(line); } Console.WriteLine(output.ToString()); We can now add our replacement line, by adding the code in Example 10-74 just before the final output to the console. Example 10-74. Fixing a specific typo output.Replace("nobelr", "nobler"); Console.WriteLine(output.ToString()); The relevant line now appears without the spelling error: Whether 'tis nobler in the mind to suffer OK, the next thing we’d like to do is to ignore completely blank lines. 354 | Chapter 10: Strings All Sorts of “Empty” Strings Let’s start by leaving out lines that have no content at all. There’s a special constant for the empty string; we saw it earlier: String.Empty. Let’s see what happens if we use the code in Example 10-75, which writes the line to the console only if it is not equal to String.Empty. Example 10-75. Detecting empty strings foreach (string line in strings) { if (line != String.Empty) { output.AppendLine(line); } else { System.Diagnostics.Debug.WriteLine("Found a blank line"); } } You might be wondering exactly how string comparisons are performed. Some lan- guages base string comparison on object identity so that "Abc" is not equal to a different string object that also contains "Abc". (That may seem weird, but in one sense it’s consistent: comparing reference types always means asking “do these two variables refer to the same thing?”) But in C#, when you have distinct string objects, it performs a “character-like” comparison between strings, so any two strings containing the same sequence of characters are equal. This is different from how most reference types work, but by treating strings as a special case, the result is closer to what most people would expect. (Or at least to what most people who hadn’t already become accustomed to the oddities of another language might expect.) Because not all languages use by-value string comparison, the .NET Framework supports the by-identity style too. Consequently, you get by-value comparison only if the C# compiler knows it’s dealing with strings. If you store two strings in variables of type object, the C# com- piler loses track of the fact that they are strings, so if you compare these variables with the == operator, it doesn’t know it should provide the string-specific by-value comparison, and will instead do the default by- identity comparison you get for most reference types. For the sake of working out what is going on, we’re also writing a message to the debug output each time we find a blank line. If we build and run, the output to the console looks like this: To be, or not to be--that is the question: Whether 'tis nobelr in the mind to suffer, The slings and arrows of outrageous fortune , All Sorts of “Empty” Strings | 355 Or to take arms against a sea of troubles, And by opposing end them. The debug output indicates that the code found and removed eight blank lines. (If you can’t see the Output panel in Visual Studio, you can show it with the View→Output menu item. Ensure that the “Show output from” drop down has Debug selected.) But apparently it missed some, judging by the output. So which are the eight “blank” lines—that is, the lines that are the equivalent of String.Empty? If you single-step through the debugger, you’ll see that they are the ones that look like "" and String.Empty. The ones that contain just whitespace account for some of the remaining blanks in the output. While visibly blank, these are clearly not “empty”—they contain whitespace characters. We’ll deal with that in a minute. The other line that looks “empty” but isn’t is the null string. As we said earlier, strings are reference types. There is, therefore, a considerable dif- ference between a null reference to a string, and an empty string, as far as the .NET runtime is concerned. However, a lot of applications don’t care about this distinction, so it can sometimes be useful to treat a null string in much the same way as an empty string. The String class offers a static method that lets us test for nullness-or-emptiness with a single call, which Example 10-76 uses. Example 10-76. Testing for either blank or null foreach (string line in strings) { if (!String.IsNullOrEmpty(line)) { output.AppendLine(line); } else { System.Diagnostics.Debug.WriteLine("Found a blank line"); } } Notice we have to use the ! operator, as the static method returns true if the string is null or empty. Our output is now stripped of “blank” lines except the one that contains just whitespace. If you check the debug output panel, you’ll see that nine lines have been ignored: To be, or not to be--that is the question: Whether 'tis nobelr in the mind to suffer, The slings and arrows of outrageous fortune , Or to take arms against a sea of troubles, And by opposing end them. So, what can we do about that remaining blank line at the start? We can deal with this by stripping out spurious whitespace, and then looking to see whether anything is left. 356 | Chapter 10: Strings Not only will this fix our blank-line problem, but it will also remove any whitespace that the user has left at the start and end of the line. Trimming Whitespace You often (but not always) want to trim whitespace from the beginning and/or end of a piece of text; especially user-provided text. When storing data in a SQL database, for example, it is frequently desirable to trim this whitespace. With that in mind, the framework provides us with the Trim, TrimStart, and TrimEnd methods. Example 10-77 uses Trim to remove the whitespace at the start and end of every line. Example 10-77. Trimming whitespace foreach (string line in strings) { if (line != null) { string trimmedLine = line.Trim(); if (trimmedLine.Length != 0) { output.AppendLine(trimmedLine); } else { System.Diagnostics.Debug.WriteLine( "Found a blank line (after trimming)"); } } else { System.Diagnostics.Debug.WriteLine("Found a null line"); } } Notice how we’re trimming the line once, and storing a reference to the result in a variable, then using that trimmed string in our subsequent tests. Because we’re calling a method on our string instance, we need to test it for nullness before we do that, or we’ll get a null reference exception. This means that we don’t need to call IsNullOr Empty in our later test. We know that it cannot be null. Instead, we do a quick test for emptiness. It turns out that the most efficient way to do this is not to compare against String.Empty but to check the Length of our string. If we build and run this, we see the following output: To be, or not to be--that is the question: Whether 'tis nobler in the mind to suffer, The slings and arrows of outrageous fortune , Or to take arms against a sea of troubles, And by opposing end them. Trimming Whitespace | 357 And in the output window: Found a blank line (after trimming) Found a null line Found a blank line (after trimming) Found a blank line (after trimming) Found a blank line (after trimming) Found a blank line (after trimming) Found a blank line (after trimming) Found a blank line (after trimming) Found a blank line (after trimming) You’ll notice that Trim has successfully removed all the whitespace at the beginning and end of each line, both spaces and tab characters, but left the whitespace in the middle of the line alone. Trim isn’t limited to removing whitespace characters, though. Another overload allows us to specify the array of characters we want to trim from the beginning or end of the line. We could use this to get rid of those spurious commas, too, using the code in Example 10-78. Example 10-78. Trimming specific characters string trimmedLine = line.Trim(' ', '\t', ','); This overload of Trim uses the parameter array syntax, so we can specify the characters we want to trim as a simple parameter list. In this case, we tell it to trim spaces, tabs, and commas. Our output, then, looks like this: To be, or not to be--that is the question: Whether 'tis nobler in the mind to suffer The slings and arrows of outrageous fortune Or to take arms against a sea of troubles And by opposing end them. Of course, although the output is correct for this particular input, it isn’t quite the same as the original Trim function—it isn’t removing all possible whitespace characters, just the ones we happened to remember to list. There are a surprising number of different characters that represent whitespace—as well as your basic ordinary space, .NET rec- ognizes a character for an en space (one the same width as the letter N), an em space (the same width as M), a thin space, and a hair space, to name just a few. There are more than 20 of the things! Example 10-79 shows a function that will trim all whitespace, plus any additional characters we specify. Example 10-79. Trimming any whitespace and specific additional characters private static string TrimWhitespaceAnd( string inputString, params char[] characters) 358 | Chapter 10: Strings { int start = 0; while (start < inputString.Length) { // If it is neither whitespace nor a character from our list // then we've hit the first non-trimmable character, so we can stop if (!char.IsWhiteSpace(inputString[start]) && !characters.Contains(inputString[start])) { break; } // Work forward a character start++; } // Work backwards from the end int end = inputString.Length −1; while (end >= start) { // If it is neither whitespace nor a character from our list // then we've hit the first non-trimmable character if (!char.IsWhiteSpace(inputString[end]) && !characters.Contains(inputString[end])) { break; } // Work back a character end--; } // Work out how long our string is for the // substring function int length = (end - start) + 1; if (length == inputString.Length) { // If we didn't trim anything, just return the // input string (don't create a new one return inputString; } // If the length is zero, then return the empty string if (length == 0) { return string.Empty; } return inputString.Substring(start, length); } This method works by iterating through our string, examining each character and checking to see whether it should be trimmed. If so, then we increment the start position by one character, and check the next one, until we hit a character that should not be trimmed, or the end of the string. We then do the same thing starting from the end of the string, and reversing character by character until we reach the start point. Trimming Whitespace | 359 If you wanted to write the equivalent of TrimStart or TrimEnd you would just optionally leave out the end or start checking, respectively. Finally, we create our new output string, by using the Substring method we looked at earlier. Notice how we’ve avoided creating strings unnecessarily; we don’t build up the results as we go along, and we don’t create new strings in the “no change” and “empty” cases. (We could have written a much shorter function if we weren’t worried about this: inputString.Trim().Trim(characters) would have done the whole job! However, with two calls to Trim, we end up generating two new strings instead of one. You’d need to measure your code’s performance in realistic test scenarios to find out whether the more complex code in Example 10-79 is worth the effort. We’re showing it mainly to illustrate how to dig around inside a string.) The interesting new bit of code, though, is that char.IsWhitespace method. Checking Character Types We’re generally familiar with the idea that characters might be numbers, letters, white- space, or punctuation. This is formalized in the .NET Framework, and char provides us with a bunch of static helper functions to do the categorization for us. Several are fairly self-explanatory: IsWhitespace, IsLetter, IsDigit, IsLetterOrDigit, IsPunctuation There are also a couple of useful items for testing whether a character is upper- or lowercase: IsUpper, IsLower Then there are a few less intuitively obvious items: IsNumber (you might wonder whether there was a difference between this and IsDigit?) IsSeparator, IsControl IsHighSurrogate, IsLowSurrogate Even the self-explanatory items turn out to be a little more complicated than you might think. These categories come from Unicode, and to understand that, we need to delve a little more deeply into the way that characters are encoded. Encoding Characters When we give a char variable the value 'A', what exactly is that value? 360 | Chapter 10: Strings We’ve already alluded to the fact that there is some kind of encoding going on— remember that we mentioned the IBM-derived Latin1 scheme when we were discussing escaped character literals. Computers work with binary values, typically made up of one or more bytes, and we clearly need some kind of mapping between the binary values in these bytes and the characters we want them to represent. We’ve all got to agree on what the binary values mean, or we can’t exchange information. To that end, the American Standards Asso- ciation convened a committee in the 1960s which defined (and then redefined, tweaked, and generally improved over subsequent decades) a standard called ASCII (pronounced ass‡-key): the American Standard Code for Information Interchange. This defined 128 characters, represented using 7 bits of a byte. The first 32 values from 0x00–0x19, and also the very last value, 0x7F, are called control characters, and include things like the tab character (0x09), backspace (0x09), bell (0x07), and delete (0x7F). The rest are called the printable characters, and include space (0x20), which is not a control character, but a “blank” printable character; all the upper and lowercase letters; and most of the punctuation marks in common use in English. This was a start, but it rapidly became apparent that ASCII did not have enough char- acters to deal with a lot of the common Western (“Latin”) scripts; the accented char- acters in French, or Spanish punctuation marks, for example. It also lacked common characters like the international copyright symbol ©, or the registered trademark symbol ®. Since ASCII uses only 7 bits, and most computers use 8-bit bytes, the obvious solution was to put the necessary characters into byte values not used by ASCII. Unfortunately, different mappings between byte values and characters emerged in different countries. These mappings are called code pages. If you bought a PC in, say, Norway, it would use a code page that offered all of the characters required to write in Norwegian, but if you tried to view the same file on a PC bought in Greece, the non-ASCII characters would look like gibberish because the PC would be configured with a Greek code page. IBM defined Latin-1 (much later updated and standardized as ISO-8859-1) as a single code page that provides most of what is required by most of the European languages that use Latin letters. Microsoft defined the Windows-1252 code page, which is mostly (but not entirely) compatible. Apple defined the Mac-Roman encoding, which has the same goal, but is completely different again. All of these encodings were designed to provide a single solution for Western European scripts, but they all fall short in various different ways—Dutch, for example, is missing some of its diphthongs. This is largely because 8 bits just isn’t enough to cover all possible characters in all international languages. Chinese alone has well over 100,000 characters. ‡ A sort of donkey, before anyone complains. Encoding Characters | 361 In the late 1980s and early 1990s, standardization efforts were underway to define an internationally acceptable encoding that would allow characters from all scripts to be represented in a reasonably consistent manner. This became the Unicode standard, and is the one that is in use in the .NET Framework. Unicode is a complex standard, as might be expected from something that is designed to deal with all current (and past) human languages, and have sufficient flexibility to deal with most conceivable future changes, too. It uses numbers to define more than 1 million code points in a codespace. A code point is roughly analogous to a character in other encodings, including formal definitions of special categories such as graphic char- acters, format characters, and control characters. It’s possible to represent a sequence of code points as a sequence of 16-bit values. You might be wondering how we can handle more than 1 million characters, when there are only 65,536 different values for 16-bit numbers. The answer is that we can team up pairs of characters. The first is called a high surrogate; if this is then followed by a low surrogate, it defines a character outside the normal 16-bit range. Unicode also defines complex ways of combining characters. Characters and their di- acritical marks can appear consecutively in a string, with the intention that they become combined in their ultimate visual representation; or you can use multiple characters to define special ligatures (characters that are joined together, like Æ). The .NET Framework Char, then, is a 16-bit value that represents a Unicode code point. This encoding is called UTF-16, and is the common in-memory repre- sentation for strings in most modern platforms. Throughout the Win- dows API, this format is referred to as “Unicode”. This is somewhat imprecise, as there are numerous different Unicode formats. But since none were in widespread use at the time Windows first introduced Uni- code support, Microsoft apparently felt that “UTF-16” was an unnec- essarily confusing name. But in general, when you see “Unicode” in either Windows or the .NET Framework, it means UTF-16. From that, we can see that those IsNumber, IsLetter, IsHighSurrogate, and IsLowSurrogate methods correspond to tests for particular Unicode categories. Why Encodings Matter You may ask: why do we need to know about encodings when “it just works”? That’s all very well for our in-memory representation of a string, but what happens when we save some text to disk, encrypt it, or send it across the Web as HTML? We may not want the 16-bit Unicode encoding we’ve got in memory, but something else. These encodings are really information interchange standards, as much as they are internal choices about how we represent strings. 362 | Chapter 10: Strings Most XML documents, for example, are encoded using the UTF-8 encoding. This is an encoding that lets us represent any character in the Unicode codespace, and is com- patible with ASCII for the characters in the 7-bit set. It achieves this by using variable- length characters: a single byte for the ASCII range, and two to six bytes for the rest. It takes advantage of special marker values (with the high bit set) to indicate the start of two to six byte sequences. While UTF-8 and ASCII are compatible in the sense that any file that contains ASCII text happens to be a valid UTF-8 file (and has the same meaning whether you interpret it as ASCII or UTF-8), there are two caveats. First, a lot of people are sloppy with their terminology and will describe any old 8-bit text encoding as ASCII, which is wrong. ASCII is strictly 7-bit. Latin1 text that uses characters from the top-bit-set range is not valid UTF-8. Second, it’s possible to construct a valid UTF-8 file that only uses characters from the 7-bit range, and yet is not a valid ASCII file. (For example, if you save a file from Windows Notepad as UTF-8, it will not be valid ASCII.) That’s because UTF-8 is allowed to contain certain non-ASCII features. One is the so-called BOM (Byte Order Mark), which is a sequence of bytes at the start of the file unam- biguously representing the file as UTF-8. (The bytes are 0xEF, 0xBB, 0xBF.) The BOM is optional, but Notepad always adds it if you save as UTF-8, which is likely to confuse any program that only understands how to process ASCII. We’re not going to look at any more details of these specific encodings. If you’re writing an encoder or decoder by hand, you’ll want to refer to the relevant specifications and vast bodies of work on their interpretation. Fortunately, for the rest of us mortals, the .NET Framework provides us with standard implementations of most of the encodings, so we can convert between the different representations fairly easily. Encoding and Decoding Encoding is the process of turning a text string into a sequence of bytes. Conversely, decoding is the process of turning a byte sequence into a text string. The .NET APIs for encoding and decoding represents these sequences as byte arrays. Let’s look at the code in Example 10-80 that illustrates this. First, we’ll encode some text using the UTF-8 and ASCII encodings, and write the byte values we see to the console. Example 10-80. Encoding text static void Main(string[] args) { string listenUp = "Listen up!"; Encoding Characters | 363 byte[] utf8Bytes = Encoding.UTF8.GetBytes(listenUp); byte[] asciiBytes = Encoding.ASCII.GetBytes(listenUp); Console.WriteLine("UTF-8"); Console.WriteLine("-----"); foreach (var encodedByte in utf8Bytes) { Console.Write(encodedByte); Console.Write(" "); } Console.WriteLine(); Console.WriteLine(); Console.WriteLine("ASCII"); Console.WriteLine("-----"); foreach (var encodedByte in asciiBytes) { Console.Write(encodedByte); Console.Write(" "); } Console.ReadKey(); } The framework provides us with the Encoding class. This has a set of static properties that provide us with specific instances of an Encoding object for a particular scheme. In this case, we’re using UTF8 and ASCII, which actually return instances of UTF8Encoding and ASCIIEncoding, respectively. Under normal circumstances, you do not need to know the actual type of these instances; you can just talk to the object returned through its Encoding base class. GetBytes returns us the byte array that corresponds to the actual in-memory represen- tation of a string, encoded using the relevant scheme. If we build and run this code, we see the following output: UTF-8 ----- 76 105 115 116 101 110 32 117 112 33 ASCII ----- 76 105 115 116 101 110 32 117 112 33 Notice that our encodings are identical in this case, just as promised. For basic Latin characters, UTF-8 and ASCII are compatible. (Unlike Notepad, the .NET UTF8Encod ing does not choose to add a BOM by default, so unless you use characters outside the 364 | Chapter 10: Strings ASCII range this will in fact produce files that can be understood by anything that knows how to process ASCII.) Let’s make a quick change to the string we’re trying to change, and translate it into French. Replace the first line inside the Main method with Example 10-81. Notice that we’ve got a capital E with an acute accent at the beginning. Example 10-81. Using a nonASCII character string listenUp = "Écoute-moi!"; If you don’t have a French keyboard and you’re wondering how to insert that E-acute character, there are a number of ways to do it. If you know the decimal representation of the Unicode code point, you can hold down the Alt key and type the number on the numeric keypad (and then release the Alt key). So Alt-0163 will insert the symbol for the UK currency, £, and Alt-0201 produces É. This doesn’t work for the normal number keys, though, so if you don’t have a numeric keypad—most laptops don’t—this isn’t much help. Possibly the most fun, though, is to run the charmap.exe application. The program icon for it in the Start menu is buried pretty deeply, so it’s easier to type charmap into a command prompt, the Start→Run box, or the Windows 7 Start menu search box. This is very instructive, and allows you to explore the various different character sets and (if you check the “Advanced view” box) encodings. You can see an image of it in Fig- ure 10-2. Alternatively, you could just escape the character—the string literal "\u00C9coutez moi" will produce the same result. And this has the advantage of not requiring non- ASCII values in your source file. Visual Studio is perfectly able to edit various file en- codings, including UTF-8, so you can put non-ASCII characters in strings without having to escape them, and you can even use them in identifiers. But some text-oriented tools are not so flexible, so there may be advantages in keeping your source code purely ASCII. Now, when we run again, we get the following output: UTF-8 ----- 195 137 99 111 117 116 101 45 109 111 105 33 ASCII ----- 63 99 111 117 116 101 45 109 111 105 33 We’ve quite clearly not got the same output in each case. The UTF-8 case starts with 195, 137, while the ASCII starts with 63. After this preamble, they’re again identical. So, let’s try decoding those two byte arrays back into strings, and see what happens. Insert the code in Example 10-82 before the call to Console.ReadKey. Encoding Characters | 365 Example 10-82. Decoding text string decodedUtf8 = Encoding.UTF8.GetString(utf8Bytes); string decodedAscii = Encoding.ASCII.GetString(asciiBytes); Console.WriteLine(); Console.WriteLine(); Console.WriteLine("Decoded UTF-8"); Console.WriteLine("-------------"); Console.WriteLine(decodedUtf8); Console.WriteLine(); Console.WriteLine(); Console.WriteLine("Decoded ASCII"); Console.WriteLine("-------------"); Console.WriteLine(decodedAscii); Figure 10-2. Charmap.exe in action 366 | Chapter 10: Strings We’re now using the GetString method on our Encoding objects, to decode the byte array back into a string. Here’s the output: UTF-8 ----- 195 137 99 111 117 116 101 45 109 111 105 33 ASCII ----- 63 99 111 117 116 101 45 109 111 105 33 Decoded UTF-8 ------------- Écoute-moi! Decoded ASCII ------------- ?coute-moi! The UTF-8 bytes have decoded back to our original string. This is because the UTF-8 encoding supports the E-acute character, and it does so by inserting two bytes into the array: 195 137. On the other hand, our ASCII bytes have been decoded and we see that the first char- acter has become a question mark. If you look at the encoded bytes, you’ll see that the first byte is 63, which (if you look it up in an ASCII table somewhere) corresponds to the question mark character. So this isn’t the fault of the decoder. The encoder, when faced with a character it didn’t un- derstand, inserted a question mark. So, you need to be careful that any encoding you choose is capable of supporting the characters you are using (or be prepared for the infor- mation loss if it doesn’t). OK, we’ve seen an example of the one-byte-per-character ASCII representation, and the at-least-one-byte-per-character UTF-8 representation. Let’s have a look at the un- derlying at-least-two-bytes-per-character UTF-16 encoding that the framework uses internally—Example 10-83 uses this. Example 10-83. Using UTF-16 encoding static void Main(string[] args) { string listenUpFR = "Écoute-moi!"; byte[] utf16Bytes = Encoding.Unicode.GetBytes(listenUpFR); Console.WriteLine("UTF-16"); Console.WriteLine("-----"); foreach (var encodedByte in utf16Bytes) { Encoding Characters | 367 Console.Write(encodedByte); Console.Write(" "); } Console.ReadKey(); } Notice that we’re using the Unicode encoding this time. If we compile and run, we see the following output: UTF-16 ----- 201 0 99 0 111 0 117 0 116 0 101 0 45 0 109 0 111 0 105 0 33 0 It is interesting to compare this with the ASCII output we had before: ASCII ----- 63 99 111 117 116 101 45 109 111 105 33 The first character is different, because UTF-16 can encode the E-acute correctly; thereafter, every other byte in the UTF-16 array is zero, and the next byte corresponds to the ASCII value. As we said earlier, the Unicode standard is highly compatible with ASCII, and each 16-bit value (i.e., pair of bytes) corresponds to the equivalent 7-bit value in the ASCII encoding. There’s one more note to make about this byte array, which has to do with the order of the bytes. This is easier to see if we first update the program to show the values in hex, using the formatting function we learned about earlier, as Example 10-84 shows. Example 10-84. Showing byte values of encoded text static void Main(string[] args) { string listenUpFR = "Écoute-moi!"; byte[] utf16Bytes = Encoding.Unicode.GetBytes(listenUpFR); Console.WriteLine("UTF-16"); Console.WriteLine("-----"); foreach (var encodedByte in utf16Bytes) { Console.Write(string.Format("{0:X2}", encodedByte)); Console.Write(" "); } Console.ReadKey(); } If we run again, we now see our bytes written out in hex format: UTF-16 ----- C9 00 63 00 6F 00 75 00 74 00 65 00 2D 00 6D 00 6F 00 69 00 21 00 368 | Chapter 10: Strings Encoding Characters | 369 Console.Write(string.Format("{0:X2}", encodedByte)); { foreach (var encodedByte in utf16Bytes) Console.WriteLine("-----"); Console.WriteLine("UTF-16"); byte[] utf16Bytes = Encoding.BigEndianUnicode.GetBytes(listenUpArabic); ;" إليّ أنصت ّ" = string listenUpArabic { static void Main(string[] args) Example 10-86. Big-endian Arabic And let’s try it once more, but with Arabic text, as Example 10-86 shows. 00 C9 00 63 00 6F 00 75 00 74 00 65 00 2D 00 6D 00 6F 00 69 00 21 ------ UTF-16 As you might expect, we get the following output: byte[] utf16Bytes = Encoding.BigEndianUnicode.GetBytes(listenUpFR); Example 10-85. Using big-endian UTF-16 the utf16Bytes variable with the code in Example 10-85. endian byte array, you can ask for it. Replace the line in Example 10-84 that initializes Should you need to communicate with something that expects its UTF-16 in a big- backwards. So take a deep breath and count to 01. rian position that it’s so-called “normal” numbers that are written at since about 1995 has been “backwards”. The other takes the contra- Consequently, one of us has felt like every memory dump he’s looked the ARM when he was growing up). endian (he used the 6502, and early pre-endian-switching versions of Z80 and 68000 when he was a baby developer) and the other is little Another historical note: one of your authors is big-endian (he used the used in most phones) can even be switched between flavors! followed by the LSB. Some chip architectures (like the later versions of the ARM chip like the 680x0 series used in “classic” Macs are big-endian—they expect the MSB, MSB, so the default Unicode encoding is little-endian. On the other hand, platforms family is natively a little-endian architecture. It always expects the LSB followed by the For good (but now largely historical) reasons of engineering efficiency, the Intel x86 the least-significant byte (LSB) first, followed by the most-significant byte (MSB). the 16-bit hex value 0x0063, represented in the little-endian form. That means we get to think of each pair of bytes as a character. So, our second character is 63 00. This is But remember that each UTF-16 code point is represented by a 16-bit value, so we need Console.Write(" "); } Console.ReadKey(); } And our output is: UTF-16 ----- 06 23 06 46 06 35 06 2A 00 20 06 25 06 44 06 4A 06 51 (Just to prove that you do get values bigger than 0xFF in Unicode!) Why Represent Strings As Byte Sequences? In the course of the chapters on file I/O (Chapter 11) and networking (Chapter 13), we’re going to see a number of communications and storage APIs that deal with writing arrays of bytes to some kind of target device. The byte format in which those strings go down the wires is clearly very important, and, while the framework default choices are often appropriate, knowing how (and why) you might need to choose a different encoding will ensure that you’re equipped to deal with mysterious bugs—especially when wrangling text in a language other than your own, or to/from a non-Windows platform.§ Summary In this chapter, we delved into the workings of strings, looking at the difference between the immutable String and its mutable cousin, StringBuilder. We saw how to convert other data types to and from strings, and how to control that formatting, especially when we consider cultures and languages other than our own. We saw the various ways in which we can compose strings, and the performance trade- offs of each technique. Finally, we looked at how strings are actually represented in memory, and how we may need to convert between different encodings for different applications, platforms, and configurations. § Yes, other platforms do exist. 370 | Chapter 10: Strings CHAPTER 11 Files and Streams Almost all programmers have to deal with storing, retrieving, and processing informa- tion in files at some time or another. The .NET Framework provides a number of classes and methods we can use to find, create, read, and write files and directories In this chapter we’ll look at some of the most common. Files, though, are just one example of a broader group of entities that can be opened, read from, and/or written to in a sequential fashion, and then closed. .NET defines a common contract, called a stream, that is offered by all types that can be used in this way. We’ll see how and why we might access a file through a stream, and then we’ll look at some other types of streams, including a special storage medium called isolated storage which lets us save and load information even when we are in a lower-trust environment (such as the Silverlight sandbox). Finally, we’ll look at some of the other stream implementations in .NET by way of comparison. (Streams crop up in all sorts of places, so this chapter won’t be the last we see of them—they’re important in net- working, for example.) Inspecting Directories and Files We, the authors of this book, have often heard our colleagues ask for a program to help them find duplicate files on their system. Let’s write something to do exactly that. We’ll pass the names of the directories we want to search on the command line, along with an optional switch to determine whether we want to recurse into subdirectories or not. In the first instance, we’ll do a very basic check for similarity based on filenames and sizes, as these are relatively cheap options. Example 11-1 shows our Main function. Example 11-1. Main method of duplicate file finder static void Main(string[] args) { bool recurseIntoSubdirectories = false; if (args.Length < 1) { 371 ShowUsage(); return; } int firstDirectoryIndex = 0; if (args.Length > 1) { // see if we're being asked to recurse if (args[0] == "/sub") { if (args.Length < 2) { ShowUsage(); return; } recurseIntoSubdirectories = true; firstDirectoryIndex = 1; } } // Get list of directories from command line. var directoriesToSearch = args.Skip(firstDirectoryIndex); List filesGroupedByName = InspectDirectories(recurseIntoSubdirectories, directoriesToSearch); DisplayMatches(filesGroupedByName); Console.ReadKey(); } The basic structure is pretty straightforward. First we inspect the command-line argu- ments to work out which directories we’re searching. Then we call InspectDirecto ries (shown later) to build a list of all the files in those directories. This groups the files by filename (without the full path) because we do not consider two files to be duplicates if they have different names. Finally, we pass this list to DisplayMatches, which displays any potential matches in the files we have found. DisplayMatches refines our test for duplicates further—it considers two files with the same name to be duplicates only if they have the same size. (That’s not foolproof, of course, but it’s surprisingly effective, and we will refine it further later in the chapter.) Let’s look at each of these steps in more detail. The code that parses the command-line arguments does a quick check to see that we’ve provided at least one command-line argument (in addition to the /sub switch if present) and we print out some usage instructions if not, using the method shown in Example 11-2. 372 | Chapter 11: Files and Streams Example 11-2. Showing command line usage private static void ShowUsage() { Console.WriteLine("Find duplicate files"); Console.WriteLine("===================="); Console.WriteLine( "Looks for possible duplicate files in one or more directories"); Console.WriteLine(); Console.WriteLine( "Usage: findduplicatefiles [/sub] DirectoryName [DirectoryName] ..."); Console.WriteLine("/sub - recurse into subdirectories"); Console.ReadKey(); } The next step is to build a list of files grouped by name. We define a couple of classes for this, shown in Example 11-3. We create a FileNameGroup object for each distinct filename. Each FileNameGroup contains a nested list of FileDetails, providing the full path of each file that has that name, and also the size of that file. Example 11-3. Types used to keep track of the files we’ve found class FileNameGroup { public string FileNameWithoutPath { get; set; } public List FilesWithThisName { get; set; } } class FileDetails { public string FilePath { get; set; } public long FileSize { get; set; } } For example, suppose the program searches two folders, c:\One and c:\Two, and sup- pose both of those folders contain a file called Readme.txt. Our list will contain a FileNameGroup whose FileNameWithoutPath is Readme.txt. Its nested FilesWithThis Name list will contain two FileDetails entries, one with a FilePath of c:\One \Readme.txt and the other with c:\Two\Readme.txt. (And each FileDetails will contain the size of the relevant file in FileSize. If these two files really are copies of the same file, their sizes will, of course, be the same.) We build these lists in the InspectDirectories method, which is shown in Exam- ple 11-4. This contains the meat of the program, because this is where we search the specified directories for files. Quite a lot of the code is concerned with the logic of the program, but this is also where we start to use some of the file APIs. Inspecting Directories and Files | 373 Example 11-4. InspectDirectories method private static List InspectDirectories( bool recurseIntoSubdirectories, IEnumerable directoriesToSearch) { var searchOption = recurseIntoSubdirectories ? SearchOption.AllDirectories : SearchOption.TopDirectoryOnly; // Get the path of every file in every directory we're searching. var allFilePaths = from directory in directoriesToSearch from file in Directory.GetFiles(directory, "*.*", searchOption) select file; // Group the files by local filename (i.e. the filename without the // containing path), and for each filename, build a list containing the // details for every file that has that filename. var fileNameGroups = from filePath in allFilePaths let fileNameWithoutPath = Path.GetFileName(filePath) group filePath by fileNameWithoutPath into nameGroup select new FileNameGroup { FileNameWithoutPath = nameGroup.Key, FilesWithThisName = (from filePath in nameGroup let info = new FileInfo(filePath) select new FileDetails { FilePath = filePath, FileSize = info.Length }).ToList() }; return fileNameGroups.ToList(); } To get it to compile, you’ll need to add: using System.IO; The parts of Example 11-4 that use the System.IO namespace to work with files and directories have been highlighted. We’ll start by looking at the use of the Directory class. Examining Directories Our InspectDirectories method calls the static GetFiles method on the Directory class to find the files we’re interested in. Example 11-5 shows the relevant code. 374 | Chapter 11: Files and Streams Example 11-5. Getting the files in a directory var searchOption = recurseIntoSubdirectories ? SearchOption.AllDirectories : SearchOption.TopDirectoryOnly; // Get the path of every file in every directory we're searching. var allFilePaths = from directory in directoriesToSearch from file in Directory.GetFiles(directory, "*.*", searchOption) select file; The overload of GetFiles we’re calling takes the directory we’d like to search, a filter (in the standard command-line form), and a value from the SearchOption enumeration, which determines whether to recurse down through all the subfolders. We’re using LINQ to Objects to build a list of all the files we require. As you saw in Chapter 8, a query with multiple from clauses works in a similar way to nested foreach loops. The code in Example 11-5 will end up calling GetFiles for each directory passed on the command line, and it will effectively concatenate the results of all those calls into a single list of files. The GetFiles method returns the full path for each file concerned, but when it comes to finding matches, we just want the filename. We can use the Path class to get the filename from the full path. Manipulating File Paths The Path class provides methods for manipulating strings containing file paths. Imagine we have the path c:\directory1\directory2\MyFile.txt. Table 11-1 shows you how you can slice that with various different Path methods. Table 11-1. The effect of various Path methods Method name Result GetDirectoryName c:\directory1\directory2 GetExtension .txt (note the leading “.”) GetFileName MyFile.txt GetFileNameWithoutExtension MyFile GetFullPath c:\directory1\directory2\MyFile.txt GetPathRoot c:\ What if we use a network path? Table 11-2 shows the results of the same methods when applied to this path: \\MyPC\Share1\directory2\MyFile.txt Manipulating File Paths | 375 Table 11-2. The effect of various Path methods with a network path Method name Result GetDirectoryName \\MyPC\Share1\directory2 GetExtension .txt GetFileName MyFile.txt GetFileNameWithoutExtension MyFile GetFullPath \\MyPC\Share1\directory2\MyFile.txt GetPathRoot \\MyPC\Share1 Notice how the path root includes the network hostname and the share name. What happens if we don’t use a full path, but one relative to the current directory? And what’s the current directory anyway? Path and the Current Working Directory The framework maintains a process-wide idea of the current working directory, which is the root path relative to which any file operations that do not fully qualify the path are made. The Directory class (as you might imagine) gives us the ability to manipulate it. Rather than a static property, there are two static methods to query and set the current value: GetCurrentDirectory and SetCurrentDirectory. Example 11-6 shows a call to the latter. Example 11-6. Setting the current directory Directory.SetCurrentDirectory(@"c:\"); Table 11-3 shows the results we’d get if we passed @"directory2\MyFile.txt" to the various Path methods after having run the code in Example 11-6. As you can see, most of the results reflect the fact that we’ve not provided a full path, but there’s one excep- tion: GetFullPath uses the current working directory if we provide it with a relative path. Table 11-3. The effect of various Path methods with a relative path Method name Result GetDirectoryName directory2 GetExtension .txt GetFileName MyFile.txt GetFileNameWithoutExtension MyFile GetFullPath c:\directory2\MyFile.txt GetPathRoot 376 | Chapter 11: Files and Streams Path doesn’t check that the named file exists. It only looks at the input string and, in the case of GetFullPath, the current working directory. OK, in our example, we just want the filename without the path, so we use Path.Get FileName to retrieve it. Example 11-7 shows the relevant piece of Example 11-4. Example 11-7. Getting the filename without the full path var fileNameGroups = from filePath in allFilePaths let fileNameWithoutPath = Path.GetFileName(filePath) group filePath by fileNameWithoutPath into nameGroup select ... We then use the LINQ group operator (which was described in Chapter 8) to group all of the files by name. Path contains a lot of other useful members that we’ll need a little bit later; but we can leave it for the time being, and move on to the other piece of information that we need for our matching code: the file size. The .NET Framework provides us with a class called FileInfo that contains a whole bunch of members that help us to discover things about a file. Examining File Information The various functions from the System.IO classes we’ve dealt with so far have all been static, but when it comes to retrieving information such as file size, we have to create an instance of a FileInfo object, passing its constructor the path of the file we’re in- terested in. That path can be either an absolute path like the ones we’ve seen already, or a path relative to the current working directory. FileInfo has a lot of overlapping functionality with other classes. For example, it provides a few helpers similar to Path to get details of the directory, filename, and extension. However, the only method we’re really interested in for our example is its Length prop- erty, which tells us the size of the file. Every other member on FileInfo has a functional equivalent on other classes in the framework. Even Length is duplicated on the stream classes we’ll come to later, but it is simpler for us to use FileInfo if we don’t intend to open the file itself. We use FileInfo in the final part of InspectDirectories, to put the file size into the per- file details. Example 11-8 shows the relevant excerpt from Example 11-4. Example 11-8. Getting the file size ... select new FileNameGroup { FileNameWithoutPath = nameGroup.Key, FilesWithThisName = Examining File Information | 377 (from filePath in nameGroup let info = new FileInfo(filePath) select new FileDetails { FilePath = filePath, FileSize = info.Length }).ToList() }; We’re now only one method short of a sort-of-useful program, and that’s the one that trawls through this information to find and display matches: DisplayMatches, which is shown in Example 11-9. Example 11-9. DisplayMatches private static void DisplayMatches( IEnumerable filesGroupedByName) { var groupsWithMoreThanOneFile = from nameGroup in filesGroupedByName where nameGroup.FilesWithThisName.Count > 1 select nameGroup; foreach (var fileNameGroup in groupsWithMoreThanOneFile) { // Group the matches by the file size, then select those // with more than 1 file of that size. var matchesBySize = from file in fileNameGroup.FilesWithThisName group file by file.FileSize into sizeGroup where sizeGroup.Count() > 1 select sizeGroup; foreach (var matchedBySize in matchesBySize) { string fileNameAndSize = string.Format("{0} ({1} bytes)", fileNameGroup.FileNameWithoutPath, matchedBySize.Key); WriteWithUnderlines(fileNameAndSize); // Show each of the directories containing this file foreach (var file in matchedBySize) { Console.WriteLine(Path.GetDirectoryName(file.FilePath)); } Console.WriteLine(); } } } private static void WriteWithUnderlines(string text) { Console.WriteLine(text); Console.WriteLine(new string('-', text.Length)); } We start with a LINQ query that looks for the filenames that crop up in more than one folder, because those are the only candidates for being duplicates. We iterate through 378 | Chapter 11: Files and Streams each such name with a foreach loop. Inside that loop, we run another LINQ query that groups the files of that name by size—see the first emphasized lines in Example 11-9. If InspectDirectories discovered three files called Program.cs, for example, and two of them were 278 bytes long while the other was 894 bytes long, this group clause would separate those three files into two groups. The where clause in the same query removes any groups that contain only one file. So the matchesBySize variable refers to a query that returns a group for each set of two or more files that have the same size (and because we’re inside a loop that iterates through the names, we already know they have the same name). Those are our duplicate candidates. We then write out the filename and size (and an underline separator of the same length). Finally, we write out each file location containing candidate matches using Path.GetDirectoryName. If we compile and run that lot, we’ll see the following output: Find duplicate files ==================== Looks for possible duplicate files in one or more directories Usage: findduplicatefiles [/sub] DirectoryName [DirectoryName] ... /sub - recurse into subdirectories We haven’t given it anywhere to look! How are we going to test our application? Well, we could provide it with some command-line parameters. If you open the project properties and switch to the Debug tab, you’ll see a place where you can add command- line arguments (see Figure 11-1). Figure 11-1. Setting command-line arguments However, we could do a bit better for test purposes. Example 11-10 shows a modified Main that supports a new /test command-line switch, which we can use to create test files and exercise the function. Example 11-10. Adding a /test switch static void Main(string[] args) { bool recurseIntoSubdirectories = false; if (args.Length < 1) { ShowUsage(); Examining File Information | 379 return; } int firstDirectoryIndex = 0; IEnumerable directoriesToSearch = null; bool testDirectoriesMade = false; try { // Check to see if we are running in test mode if (args.Length == 1 && args[0] == "/test") { directoriesToSearch = MakeTestDirectories(); testDirectoriesMade = true; recurseIntoSubdirectories = true; } else { if (args.Length > 1) { // see if we're being asked to recurse if (args[0] == "/sub") { if (args.Length < 2) { ShowUsage(); return; } recurseIntoSubdirectories = true; firstDirectoryIndex = 1; } } // Get list of directories from command line. directoriesToSearch = args.Skip(firstDirectoryIndex); } List filesGroupedByName = InspectDirectories(recurseIntoSubdirectories, directoriesToSearch); DisplayMatches(filesGroupedByName); Console.ReadKey(); } finally { if( testDirectoriesMade ) { CleanupTestDirectories(directoriesToSearch); } } } 380 | Chapter 11: Files and Streams In order to operate in test mode, we’ve added an alternative way to initialize the variable that holds the list of directories (directoriesToSearch). The original code, which initi- alizes it from the command-line arguments (skipping over the /sub switch if present), is still present. However, if we find the /test switch, we initialize it to point at some test directories we’re going to create (in the MakeTestDirectories method). The rest of the code can then be left as it was (to avoid running some completely different program in our test mode). Finally, we add a bit of cleanup code at the end to remove any test directories if we created them. So, how are we going to implement MakeTestDirectories? We want to create some temporary files, and write some content into them to exercise the various matching possibilities. Creating Temporary Files A quick look at Path reveals the GetTempFileName method. This creates a file of zero length in a directory dedicated to temporary files, and returns the path to that file. It is important to note that the file is actually created, whether you use it or not, and so you are responsible for cleaning it up when you are done, even if you don’t make any further use of it. Let’s create another test console application, just to try out that method. We can do that by adding the following to our main function: string fileName = Path.GetTempFileName(); // Display the filename Console.WriteLine(fileName); // And wait for some input Console.ReadKey(); But wait! If we just compile and run that, we’ll leave the file we created behind on the system. We should make sure we delete it again when we’re done. There’s nothing special about a temporary file. We create it in an unusual way, and it ends up in a particular place, but once it has been created, it’s just like any other file in the filesystem. So, we can delete it the same way we’d delete any other file. Deleting Files The System.IO namespace provides the File class, which offers various methods for doing things with files. Deleting is particularly simple: we just use the static Delete method, as Example 11-11 shows. Deleting Files | 381 Example 11-11. Deleting a file string fileName = Path.GetTempFileName(); try { // Use the file // ... // Display the filename Console.WriteLine(fileName); // And wait for some input Console.ReadKey(); } finally { // Then clean it up File.Delete(fileName); } Notice that we’ve wrapped the code in which we (could) manipulate the file further in a try block, and deleted it in a finally block. This ensures that whatever happens, we’ll always attempt to clean up after ourselves. If you compile and run this test project now, you’ll see some output like this: C:\Users\yourusername\AppData\Local\Temp\tmpCA8F.tmp The exact text will depend on your operating system version, your username, and (of course) the random filename that was created for you. If you browse to that path, you will see a zero-length file of that name. If you then press a key, allowing Console.ReadKey to return, it will drop through to the finally block, where we delete the temporary file, using the static Delete method on the File class. There are lots of scenarios where this sort of temporary file creation is just fine, but it doesn’t really suit our example application’s needs. We want to create multiple tem- porary files, in multiple different directories. GetTempFileName doesn’t really do the job for us. If we look at Path again, though, there’s another likely looking method: GetRandomFi leName. This returns a random string of characters that can be used as either a file or a directory name. It uses a cryptographically strong random number generator (which can be useful in some security-conscious scenarios), and is statistically likely to produce a unique name, thus avoiding clashes. Unlike GetTempFileName it doesn’t actually create the file (or directory); that’s up to us. If you run the code in Example 11-12: Example 11-12. Showing a random filename Console.WriteLine(Path.GetRandomFileName()); 382 | Chapter 11: Files and Streams you’ll see output similar to this: xnicz3rs.juc (Obviously, the actual characters you see will, hopefully, be different, or the statistical uniqueness isn’t all that unique!) So, we can use that method to produce our test file and directory names. But where are we going to put the files? Perhaps one of the various “well-known folders” Windows offers would suit our needs. Well-Known Folders Most operating systems have a bunch of well-known filesystem locations, and Win- dows is no exception. There are designated folders for things like the current user’s documents, pictures, or desktop; the program files directory where applications are installed; and the system folder. The .NET Framework provides a class called Environment that provides information about the world our program runs in. Its static method GetFolderPath is the one that interests us right now, because it will return the path of various well-known folders. We pass it one of the Environment.SpecialFolder enumeration values. Exam- ple 11-13 retrieves the location of one of the folders in which applications can store per-user data. Example 11-13. Getting a well-known folder location string path = Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData); Table 11-4 lists all of the well-known folders that GetFolderPath can return, and the location they give on the installed copy of Windows 7 (64-bit) belonging to one of the authors. Table 11-4. Special folders Enumeration Example location Purpose ApplicationData C:\Users\mwa\ AppData\Roaming A place for applications to store their own private infor- mation for a particular user; this may be located on a shared server, and available across multiple logins for the same user, on different machines, if the user’s domain policy is configured to do so. CommonApplicationData C:\ProgramData A place for applications to store their own private infor- mation accessible to all users. CommonProgramFiles C:\Program Files\Common Files A place where shared application components can be installed. Cookies C:\Users\mwa\ AppData\Roaming\ The location where Internet cookies are stored for this user; another potentially roaming location. Well-Known Folders | 383 Enumeration Example location Purpose Microsoft\Windows\Cookies Desktop C:\Users\mwa\ Desktop The current user’s desktop (virtual) folder. DesktopDirectory C:\Users\mwa\ Desktop The physical directory where filesystem objects on the desktop are stored (currently, but not necessarily, the same as Desktop). Favorites C:\Users\mwa\ Favorites The directory containing the current user’s favorites links. History C:\Users\mwa\ AppData\Local\ Microsoft\Windows\ History The directory containing the current user’s Internet history. InternetCache C:\Users\mwa\ AppData\Local\ Microsoft\Windows\ Temporary Internet Files The directory that contains the current user’s Internet cache. LocalApplicationData C:\Users\mwa\ AppData\Local A place for applications to store their private data associ- ated with the current user. This is guaranteed to be on the local machine (as opposed to ApplicationData which may roam with the user). MyComputer This is always an empty string because there is no real folder that corresponds to My Computer. MyDocuments C:\Users\mwa\ Documents The folder in which the current user’s documents (as op- posed to private application datafiles) are stored. MyMusic C:\Users\mwa\ Music The folder in which the current user’s music files are stored. MyPictures C:\Users\mwa\ Pictures The folder in which the current user’s picture files are stored. Personal C:\Users\mwa\ Documents The folder in which the current user’s documents are stored (synonymous with MyDocuments). ProgramFiles C:\Program Files The directory in which applications are installed. Note that there is no special folder enumeration for the 32-bit ap- plications directory on 64-bit Windows. Programs C:\Users\mwa\ AppData\Roaming\ Microsoft\Windows\ The location where application shortcuts in the Start menu’s Programs section are stored for the current user. This is another potentially roaming location. 384 | Chapter 11: Files and Streams Enumeration Example location Purpose Start Menu\Programs Recent C:\Users\mwa\ AppData\Roaming\ Microsoft\Windows\ Recent The folder where links to recently used documents are stored for the current user. This is another potentially roaming location. SendTo C:\Users\mwa\ AppData\Roaming\ Microsoft\Windows\ SendTo The location that contains the links that form the Send To menu items in the shell. This is another potentially roaming location. StartMenu C:\Users\mwa\ AppData\Roaming\ Microsoft\Windows\ Start Menu The folder that contains the Start menu items for the current user. This is another potentially roaming location. Startup C:\Users\mwa\ AppData\Roaming\ Microsoft\Windows\ Start Menu\Programs \Startup The folder that contains links to programs that will run each time the current user logs in. This is another poten- tially roaming location. System C:\Windows\ system32 The Windows system folder. Templates C:\Users\mwa\ AppData\Roaming\ Microsoft\Windows\ Templates A location in which applications can store document tem- plates for the current user. Again, this is a potentially roaming location. Notice that this doesn’t include all of the well-known folders we have these days, because the set of folders grows with each new version of Windows. Things like Videos, Games, Downloads, Searches, and Con- tacts are all missing. It also doesn’t support Windows 7 libraries in any meaningful sense. This is (sort of) by design. The method provides a lowest common denominator approach to finding useful folders on the system, in a way that works across all supported versions of the frame- work (including Windows Mobile). So, we need to choose a path in which our current user is likely to have permission to create/read/write and delete files and directories. It doesn’t have to be one that the user Well-Known Folders | 385 can see under normal circumstances. In fact, we’re going to create files with extensions that are not bound to any applications and we should not do that in a place that’s visible to the user if we want our application to be a good Windows citizen. If you create a file in a place that’s visible to the user, like Documents or Desktop, you should ensure that it always has a default application associated with it. There are two candidates for this in Table 11-4: LocalApplicationData and ApplicationData. Both of these offer places for applications to store files that the user wouldn’t normally see. (Of course, users can find these folders if they look hard enough. The goal here is to avoid putting our temporary test files in the same folders as the user’s documents.) The difference between these two folders is that if the user has a roaming profile, files in the latter folder will be copied around the network as they move from one machine to another, while files in the former folder remain on the machine on which they were created. We’re building temporary files for test purposes, so LocalApplicationData looks like the right choice. So, let’s return to our demo application, and start to implement the MakeTestDirecto ries method. The first thing we need to do is to create a few test directories. Exam- ple 11-14 contains some code to do that. Example 11-14. Creating test directories private static string[] MakeTestDirectories() { string localApplicationData = Path.Combine( Environment.GetFolderPath( Environment.SpecialFolder.LocalApplicationData), @"Programming CSharp\FindDuplicates"); // Let's make three test directories var directories = new string[3]; for (int i = 0; i < directories.Length; ++i) { string directory = Path.GetRandomFileName(); // Combine the local application data with the // new random file/directory name string fullPath = Path.Combine(localApplicationData, directory); // And create the directory Directory.CreateDirectory(fullPath); directories[i] = fullPath; Console.WriteLine(fullPath); } return directories; } 386 | Chapter 11: Files and Streams First, we use the GetFolderPath method to get the LocalApplicationData path. But we don’t want to work directly in that folder—applications are meant to create their own folders underneath this. Normally you’d create a folder named either for your company or for your organization, and then an application-specific folder inside that—we’ve used Programming CSharp as the organization name here, and FindDuplicates as the application name. We then use a for loop to create three directories with random names inside that. To create these new directories, we’ve used a couple of new methods: Path.Combine and Directory.CreateDirectory. Concatenating Path Elements Safely If you’ve written any code that manipulates paths before, you’ll have come across the leading/trailing slash dilemma. Does your path fragment have one or not? You also need to know whether the path fragment you’re going to append really is a relative path—are there circumstances under which you might need to deal with a fully quali- fied path instead? Path.Combine does away with all that anxiety. Not only will it check all those things for you and do the right thing, but it will even check that your paths contain only valid path characters. Table 11-5 contains some example paths, and the result of combining them with Path.Combine. Table 11-5. Example results of Path.Combine Path 1 Path 2 Combined C:\hello\ world C:\hello\world C:\hello world C:\hello\world C:\hello\ \world C:\hello\world hello world hello\world C:\hello world.exe c\hello\world.exe \\mybox\hello world \\mybox\hello\world world C:\hello C:\hello The last entry in that table is particularly interesting: notice that the second path is absolute, and so the combined path is “optimized” to just that second path. In our case, Example 11-14 combines the well-known folder with a subfolder name to get a folder location specific to this example. And then it combines that with our new temporary folder names, ready for creation. Concatenating Path Elements Safely | 387 Creating and Securing Directory Hierarchies Directory.CreateDirectory is very straightforward: it does exactly what its name sug- gests. In fact, it will create any directories in the whole path that do not already exist, so you can create a deep hierarchy with a single call. (You’ll notice that Exam- ple 11-14 didn’t bother to create the Programming CSharp\FindDuplicates folder— those will get created automatically the first time we run as a result of creating the temporary folders inside them.) A side effect of this is that it is safe to call it if all of the directories in the path already exist—it will just do nothing. In addition to the overload we’ve used, there’s a second which also takes a Directory Security parameter: Directory.CreateDirectory(string path, DirectorySecurity directorySecurity) The DirectorySecurity class allows you to specify filesystem access controls with a relatively simple programming model. If you’ve tried using the Win32 ACL APIs, you’ll know that it is a nightmare of GUIDs, SSIDs, and lists sensitive to item ordering. This model does away with much of the complexity. Let’s extend our create function to make sure that only our current user has read/write/ modify permissions on these directories. Example 11-15 modifies the previous example by explicitly granting the current user full control of the newly created folders. The new or changed lines are highlighted. Example 11-15. Configuring access control on new directories private static string[] MakeTestDirectories() { string localApplicationData = Path.Combine( Environment.GetFolderPath( Environment.SpecialFolder.LocalApplicationData), @"Programming CSharp\FindDuplicates"); // Get the name of the logged in user string userName = WindowsIdentity.GetCurrent().Name; // Make the access control rule FileSystemAccessRule fsarAllow = new FileSystemAccessRule( userName, FileSystemRights.FullControl, AccessControlType.Allow); DirectorySecurity ds = new DirectorySecurity(); ds.AddAccessRule(fsarAllow); // Let's make three test directories var directories = new string[3]; for (int i = 0; i < directories.Length; ++i) { string directory = Path.GetRandomFileName(); // Combine the local application data with the // new random file/directory name 388 | Chapter 11: Files and Streams string fullPath = Path.Combine(localApplicationData, directory); // And create the directory Directory.CreateDirectory(fullPath, ds); directories[i] = fullPath; Console.WriteLine(fullPath); } return directories; } You’ll need to add a couple of using directives to the top of the file before you can compile this code: using System.Security.AccessControl; using System.Security.Principal; What do these changes do? First, we make use of a type called WindowsIdentity to find the current user, and fish out its name. If you happen to want to specify the name explicitly, rather than get the current user programmatically, you can do so (e.g., MYDOMAIN\SomeUserId). Then, we create a FileSystemAccessRule, passing it the username, the FileSystem Rights we want to set, and a value from the AccessControlType enumeration which determines whether we are allowing or denying those rights. If you take a look at the FileSystemRights enumeration in MSDN, you should recognize the options from the Windows security permissions dialog in the shell. You can com- bine the individual values (as it is a Flags enumeration), or use one of the precanned sets as we have here. If you compile this application, and modify the debug settings to pass just the /test switch as the only command-line argument, when you run it you’ll see output similar to the following (but with your user ID, and some different random directory names): C:\Users\yourId\AppData\Local\Programming CSharp\FindDuplicates\yzw0iw3p.ysq C:\Users\yourId\AppData\Local\Programming CSharp\FindDuplicates\qke5k2ql.5et C:\Users\yourId\AppData\Local\Programming CSharp\FindDuplicates\5hkhspqa.osc If we take a look at the folder in Explorer, you should see your new directories (some- thing like Figure 11-2). If you right-click on one of these and choose Properties, then examine the Security tab, you should see something like Figure 11-3. Notice how the only user with permissions on this directory is the currently logged on user (in this case ian, on a domain called idg.interact). All of the usual inherited permissions have been overridden. Rather than the regular read/modify/write check- boxes, we’ve apparently got special permissions. This is because we set them explicitly in the code. Creating and Securing Directory Hierarchies | 389 Figure 11-2. Newly created folders Figure 11-3. Permissions on the new directory 390 | Chapter 11: Files and Streams We can have a look at that in more detail if we click the Advanced button, and switch to the Effective Permissions tab. Click the Select button to pick a user (see Fig- ure 11-4). First, let’s look at the effective permissions for the local administrator (this is probably MachineName\Administrator, unless you’ve changed your default adminis- trator name to try to make things slightly harder for an attacker). Figure 11-4. Selecting a user If you click OK, you’ll see the effective permissions for Administrator on that folder (Figure 11-5). You can scroll the scroll bar to prove it for yourself, but you can see that even Admin- istrator cannot actually access your folder! (This is not, of course, strictly true. Admin- istrators can take ownership of the folder and mess with the permissions themselves, but they cannot access the folder without changing the permissions first.) Try again with your own user ID. You will see results similar to Figure 11-6—we have full control. Scroll the list and you’ll see that everything is ticked. What if we wanted “not quite” full control? Say we wanted to deny the ability to write extended attributes to the file. Well, we can update our code and add a second FileSystemAccessRule. Example 11-16 shows the additional code required. Example 11-16. Denying permissions private static string[] MakeTestDirectories() { // ... FileSystemAccessRule fsarAllow = new FileSystemAccessRule( userName, FileSystemRights.FullControl, AccessControlType.Allow); Creating and Securing Directory Hierarchies | 391 ds.AddAccessRule(fsarAllow); FileSystemAccessRule fsarDeny = new FileSystemAccessRule( userName, FileSystemRights.WriteExtendedAttributes, AccessControlType.Deny); ds.AddAccessRule(fsarDeny); // ... } Notice that we’re specifying AccessControlType.Deny. Before you compile and run this, delete the folders you created with the last run, using Explorer—we’ll write some code to do that automatically in a minute, because it will get very boring very quickly! You should see very similar output to last time (just with some new directory names): C:\Users\yourId\AppData\Local\Programming CSharp\FindDuplicates\slhwbtgo.sop C:\Users\yourId\AppData\Local\Programming CSharp\FindDuplicates\bsfndkgn.ucm C:\Users\yourId\AppData\Local\Programming CSharp\FindDuplicates\tayf1uvg.y4y Figure 11-5. Effective permissions for Administrator on the new folder 392 | Chapter 11: Files and Streams Figure 11-6. Effective permissions for the current user on the new folder If you look at the permissions, you will now see both the Allow and the new Deny entries (Figure 11-7). As a double-check, take a look at the effective permissions for your current user (see Figure 11-8). In Figure 11-8 you can see that we’ve no longer got Full control, because we’ve been specifically denied Write extended attributes. Of course, we could always give that permission back to ourselves, because we’ve been allowed Change permissions, but that’s not the point! Although that isn’t the point, security permissions of all kinds are a complex affair. If your users have local or domain administrator per- missions, they can usually work around any other permissions you try to manage. You should always try to abide by the principle of least per- mission: don’t grant people more privileges than they really need to do the job. Although that will require a little more thinking up front, and can sometimes be a frustrating process while you try to configure a sys- tem, it is much preferable to a wide-open door. Creating and Securing Directory Hierarchies | 393 OK, delete those new directories using Explorer, and we’ll write some code to clean up after ourselves. We need to delete the directories we’ve just created, by implementing our CleanupTestDirectories method. Deleting a Directory You’re probably ahead of us by now. Yes, we can delete a directory using Directory.Delete, as Example 11-17 shows. Example 11-17. Deleting a directory private static void CleanupTestDirectories(IEnumerable directories) { foreach (var directory in directories) { Directory.Delete(directory); } } We’re just iterating through the set of new directories we stashed away earlier, deleting them. Figure 11-7. Permissions now that we’ve denied write extended attributes 394 | Chapter 11: Files and Streams OK, we’ve got our test directories. We’d now like to create some test files to use. Just before we return from MakeTestDirectories, let’s add a call to a new method to create our files, as Example 11-18 shows. Example 11-18. Creating files in the test directories ... CreateTestFiles(directories); return directories; Example 11-19 shows that method. Example 11-19. The CreateTestFiles method private static void CreateTestFiles(IEnumerable directories) { string fileForAllDirectories = "SameNameAndContent.txt"; string fileSameInAllButDifferentSizes = "SameNameDifferentSize.txt"; int directoryIndex = 0; // Let's create a distinct file that appears in each directory foreach (string directory in directories) Figure 11-8. Effective permissions with write extended attributes denied Deleting a Directory | 395 { directoryIndex++; // Create the distinct file for this directory string filename = Path.GetRandomFileName(); string fullPath = Path.Combine(directory, filename); CreateFile(fullPath, "Example content 1"); // And now the one that is in all directories, with the same content fullPath = Path.Combine(directory, fileForAllDirectories); CreateFile(fullPath, "Found in all directories"); // And now the one that has the same name in // all directories, but with different sizes fullPath = Path.Combine(directory, fileSameInAllButDifferentSizes); StringBuilder builder = new StringBuilder(); builder.AppendLine("Now with"); builder.AppendLine(new string('x', directoryIndex)); CreateFile(fullPath, builder.ToString()); } } As you can see, we’re running through the directories, and creating three files in each. The first has a different, randomly generated filename in each directory, and remember, our application only considers files with the same names as being possible duplicates, so we expect the first file we add to each directory to be considered unique. The second file has the same filename and content (so they will all be the same size) in every folder. The third file has the same name every time, but its content varies in length. Well, we can’t put off the moment any longer; we’re going to have to create a file, and write some content into it. There are lots and lots and lots (and lots) of different ways of doing that with the .NET Framework, so how do we go about picking one? Writing Text Files Our first consideration should always be to “keep it simple,” and use the most con- venient method for the job. So, what is the job? We need to create a file, and write some text into it. File.WriteAllText looks like a good place to start. Writing a Whole Text File at Once The File class offers three methods that can write an entire file out in a single step: WriteAllBytes, WriteAllLines, and WriteAllText. The first of these works with binary, but our application has text. As you saw in Chapter 10, we could use an Encoding to convert our text into bytes, but the other two methods here will do that for us. (They all use UTF-8.) 396 | Chapter 11: Files and Streams WriteAllLines takes a collection of strings, one for each line, but our code in Exam- ple 11-19 prepares content in the form of a single string. So as Example 11-20 shows, we use WriteAllText to write the file out with a single line of code. (In fact, we probably didn’t need to bother putting this code into a separate method. However, this will make it easier for us to illustrate some of the alternatives later.) Example 11-20. Writing a string into a new file private static void CreateFile(string fullPath, string contents) { File.WriteAllText(fullPath, contents); } The path can be either relative or absolute, and the file will be created if it doesn’t already exist, and overwritten if it does. This was pretty straightforward, but there’s one problem with this technique: it requires us to have the entire file contents ready at the point where we want to start writing text. This application already does that, but this won’t always be so. What if your program performs long and complex processing that produces very large volumes of text? Writ- ing the entire file at once like this would involve having the whole thing in memory first. But there’s a slightly more complex alternative that makes it possible to generate gigabytes of text without consuming much memory. Writing Text with a StreamWriter The File class offers a CreateText method, which takes the path to the file to create (either relative or absolute, as usual), and creates it for you if it doesn’t already exist. If the file is already present, this method overwrites it. Unlike the WriteAllText method, it doesn’t write any data initially—the newly created file will be empty at first. The method returns an instance of the StreamWriter class, which allows you to write to the file. Example 11-21 shows the code we need to use that. Example 11-21. Creating a StreamWriter private static void CreateFile(string fullPath, string p) { using (StreamWriter writer = File.CreateText(fullPath)) { // Use the stream writer here } } We’re no longer writing the whole file in one big lump, so we need to let the StreamWriter know when we’re done. To make life easier for us, StreamWriter imple- ments IDisposable, and closes the underlying file if Dispose is called. This means that we can wrap it in a using block, as Example 11-21 shows, and we can be assured that it will be closed even if an exception is thrown. Writing Text Files | 397 So, what is a StreamWriter? The first thing to note is that even though this chapter has “Stream” in the title, this isn’t actually a Stream; it’s a wrapper around a Stream. It derives from a class called TextWriter, which, as you might guess, is a base for types which write text into things, and a StreamWriter is a TextWriter that writes text into a Stream. TextWriter defines lots of overloads of Write and WriteLine methods, very sim- ilar to those we’ve been using on Console in all of our examples so far. If it is so similar in signature, why doesn’t Console derive from Text Writer? TextWriter is intended to be used with some underlying resource that needs proper lifetime management, so it implements IDisposable. Our code would be much less readable if we had to wrap every call on Console with a using block, or remember to call Dispose—especially as it isn’t really necessary. So, why make TextWriter implement IDisposa ble? We do that so that our text-writing code can be implemented in terms of this base class, without needing to know exactly what sort of TextWriter we’re talking to, and still handle the cleanup properly. The File class’s CreateText method calls a constructor on StreamWriter which opens the newly created file, and makes it ready for us to write; something like this: return new StreamWriter(fullPath, false); There’s nothing to stop you from doing this yourself by hand, and there are many situations where you might want to do so; but the helper methods on File tend to make your code smaller, and more readable, so you should consider using those first. We’ll look at using Stream Writer (and its partner, StreamReader) in this way later in the chapter, when we’re dealing with different sorts of underlying streams. Hang on, though. We’ve snuck a second parameter into that constructor. What does that Boolean mean? When you create a StreamWriter, you can choose to overwrite any existing file content (the default), or append to what is already there. The second Boo- lean parameter to the constructor controls that behavior. As it happen, passing false here means we want to overwrite. This is a great example of why it’s better to define nicely named enu- merations, rather than controlling this sort of thing with a bool. If the value had not been false, but some mythical value such as OpenBehav ior.Overwrite, we probably wouldn’t have needed to explain what it did. C# 4.0 added the ability to use argument names when calling methods, so we could have written new StreamWriter(fullPath, append: false), which improves matters slightly, but doesn’t help you when you come across code that hasn’t bothered to do that. 398 | Chapter 11: Files and Streams So, now we can easily complete the implementation of our CreateFile method, as shown in Example 11-22. Example 11-22. Writing a string with StreamWriter private static void CreateFile(string fullPath, string p) { using (StreamWriter writer = File.CreateText(fullPath)) { writer.Write(p); } } We just write the string we’ve been provided to the file. In this particular application, Example 11-22 isn’t an improvement on Example 11-20—we’re just writing a single string, so WriteAllText was a better fit. But StreamWriter is an important technique for less trivial scenarios. StreamReader/Writer and Text Encodings We learned in Chapter 10 that there are a number of different encodings that can be used for text characters (like ASCII, UTF-8, and Unicode). Those encodings determine exactly what sequence of bytes represents any particular character. StreamWriter (and StreamReader) need to take account of those encodings when they write or read data from a stream. By default, writers use a UTF-8 encoding, while readers attempt to determine the en- coding from the content of the file, but you can override that and provide your own Encoding to the constructor. Likewise, the File.WriteAllText method used in Exam- ple 11-20 defaults to UTF-8, but it too offers an overload that accepts an Encoding. OK, let’s build and run this code again (press F5 to make sure it runs in the debugger). And everything seems to be going very well. We see the output we’d hoped for: C:\Users\mwa\AppData\Local\up022gsm.241 C:\Users\mwa\AppData\Local\gdovysqk.cqn C:\Users\mwa\AppData\Local\xyhazu3n.4pw SameNameAndContent.txt ---------------------- C:\Users\mwa\AppData\Local\up022gsm.241 C:\Users\mwa\AppData\Local\gdovysqk.cqn C:\Users\mwa\AppData\Local\xyhazu3n.4pw That is to say, one file is found duplicated in three directories. All the others have failed to match, exactly as we’d expect. Unfortunately, almost before we’d had a chance to read that, the debugger halted ex- ecution to report an unhandled exception. It crashes in the code we added in Exam- ple 11-17 to delete the directories, because the directories are not empty. Writing Text Files | 399 For now, we’re going to have to clean up those directories by hand again, and make another change to our code. Clearly, the problem is that the Directory.Delete method doesn’t delete the files and directories inside the directory itself. This is easily fixed, because there is another overload of that method which does allow us to delete the files recursively—you just pass a Boolean as the second parameter (true for recursive deletes, and false for the default behavior). Don’t add this parameter unless you’re absolutely sure that the code is working correctly, looking only at the test directory, and not executing this code in nontest mode. We don’t want a host of emails appearing telling us that we deleted your entire, non-backed-up source and docu- ment tree because you followed this next instruction, having deviated slightly from the earlier instructions. If you want to avoid having to clean up the directories by hand, though, and you’re really, really sure everything is fine, you could add this, at your own risk: Directory.Delete(directory, true); So far, we have quietly ignored the many, many things that can go wrong when you’re using files and streams. Now seems like a good time to dive into that murky topic. When Files Go Bad: Dealing with Exceptions Exceptions related to file and stream operations fall into three broad categories: • The usual suspects you might get from any method: incorrect parameters, null references, and so on • I/O-related problems • Security-related problems The first category can, of course, be dealt with as normal—if they occur (as we discussed in Chapter 6) there is usually some bug or unexpected usage that you need to deal with. The other two are slightly more interesting cases. We should expect problems with file I/O. Files and directories are (mostly) system-wide shared resources. This means that anyone can be doing something with them while you are trying to use them. As fast as you’re creating them, some other process might be deleting them. Or writing to them; or locking them so that you can’t touch them; or altering the permissions on them so that you can’t see them anymore. You might be working with files on a network share, in which case different computers may be messing with the files, or you might lose connectivity partway through working with a file. This “global” nature of files also means that you have to deal with concurrency prob- lems. Consider this piece of code, for example, that makes use of the (almost totally 400 | Chapter 11: Files and Streams redundant) method File.Exists, shown in Example 11-23, which determines whether a file exists. Example 11-23. The questionable File.Exists method if (File.Exists("SomeFile.txt")) { // Play with the file } Is it safe to play with the file in there, on the assumption that it exists? No. In another process, even from another machine if the directory is shared, someone could nip in and delete the file or lock it, or do something even more nefarious (like substitute it for something else). Or the user might have closed the lid of his laptop just after the method returns, and may well be in a different continent by the time he brings it out of sleep mode, at which point you won’t necessarily have access to the same network shares that seemed to be visible just one line of code ago. So you have to code extremely defensively, and expect exceptions in your I/O code, even if you checked that everything looked OK before you started your work. Unlike most exceptions, though, abandoning the operation is not always the best choice. You often see transient problems, like a USB drive being temporarily unavail- able, for example, or a network glitch temporarily hiding a share from us, or aborting a file copy operation. (Transient network problems are particularly common after a laptop resumes from suspend—it can take a few seconds to get back on the network, or maybe even minutes if the user is in a hotel and has to sign up for an Internet con- nection before connecting back to the office VPN. Abandoning the user’s data is not a user-friendly response to this situation.) When an I/O problem occurs, the framework throws one of several exceptions derived from IOException (or, as we’ve already seen, IOException itself) listed here: IOException This is thrown when some general problem with I/O has occurred. This is the base for all of the more specific exception types, but it is sometimes thrown in its own right, with the Message text describing the actual problem. This makes it somewhat less useful for programmatic interpretation; you usually have to allow the user to intervene in some way when you catch one of these. DirectoryNotFoundException This is thrown when an attempt is made to access a directory that does not exist. This commonly occurs because of an error in constructing a path (particularly when relative paths are in play), or because some other process has moved or de- leted a directory during an operation. When Files Go Bad: Dealing with Exceptions | 401 DriveNotFoundException This is thrown when the root drive in a path is no longer available. This could be because a drive letter has been mapped to a network location which is no longer available, or a removable device has been removed. Or because you typed the wrong drive letter! FileLoadException This is a bit of an anomaly in the family of IOExceptions, and we’re including it in this list only because it can cause some confusion. It is thrown by the runtime when an assembly cannot be loaded; as such, it has more to do with assemblies than files and streams. FileNotFoundException This is thrown when an attempt is made to access a file that does not exist. As with DirectoryNotFoundException, this is often because there has been some error in constructing a path (absolute or relative), or because something was moved or deleted while the program was running. PathTooLongException This is an awkward little exception, and causes a good deal of confusion for de- velopers (which is one reason correct behavior in the face of long paths is a part of Microsoft’s Designed For Windows test suite). It is thrown when a path provided is too long. But what is “too long”? The maximum length for a path in Windows used to be 260 characters (which isn’t very long at all). Recent versions allow paths up to about (but not necessarily exactly) 32,767 characters, but making use of that from .NET is awkward. There’s a detailed discussion of Windows File and Path lengths if you fall foul of the problem in the MSDN documentation at http://msdn .microsoft.com/library/aa365247, and a discussion of the .NET-specific issues at http://go.microsoft.com/fwlink/?LinkID=163666. If you are doing anything with I/O operations, you will need to think about most, if not all, of these exceptions, deciding where to catch them and what to do when they occur. Let’s look back at our example again, and see what we want to do with any exceptions that might occur. As a first pass, we could just wrap our main loop in a try/catch block, as Example 11-24 does. Since our application’s only job is to report its findings, we’ll just display a message if we encounter a problem. Example 11-24. A first attempt at handling I/O exceptions try { List filesGroupedByName = InspectDirectories(recurseIntoSubdirectories, directoriesToSearch); DisplayMatches(foundFiles); Console.ReadKey(); } 402 | Chapter 11: Files and Streams catch (PathTooLongException ptlx) { Console.WriteLine("The specified path was too long"); Console.WriteLine(ptlx.Message); } catch (DirectoryNotFoundException dnfx) { Console.WriteLine("The specified directory was not found"); Console.WriteLine(dnfx.Message); } catch (IOException iox) { Console.WriteLine(iox.Message); } catch (UnauthorizedAccessException uax) { Console.WriteLine("You do not have permission to access this directory."); Console.WriteLine(uax.Message); } catch (ArgumentException ax) { Console.WriteLine("The path provided was not valid."); Console.WriteLine(ax.Message); } finally { if (testDirectoriesMade) { CleanupTestDirectories(directoriesToSearch); } } We’ve decided to provide specialized handling for the PathTooLongException and DirectoryNotFoundException exceptions, as well as generic handling for IOException (which, of course, we have to catch after the exceptions derived from it). In addition to those IOException-derived types, we’ve also caught UnauthorizedAcces sException. This is a security exception, rather than an I/O exception, and so it derives from a different base (SystemException). It is thrown if the user does not have permission to access the directory concerned. Let’s see that in operation, by creating an additional test directory and denying our- selves access to it. Example 11-25 shows a function to create a directory where we deny ourselves the ListDirectory permission. Example 11-25. Denying permission private static string CreateDeniedDirectory(string parentPath) { string deniedDirectory = Path.GetRandomFileName(); string fullDeniedPath = Path.Combine(parentPath, deniedDirectory); string userName = WindowsIdentity.GetCurrent().Name; DirectorySecurity ds = new DirectorySecurity(); FileSystemAccessRule fsarDeny = When Files Go Bad: Dealing with Exceptions | 403 new FileSystemAccessRule( userName, FileSystemRights.ListDirectory, AccessControlType.Deny); ds.AddAccessRule(fsarDeny); Directory.CreateDirectory(fullDeniedPath, ds); return fullDeniedPath; } We can call it from our MakeTestDirectories method, as Example 11-26 shows (along with suitable modifications to the code to accommodate the extra directory). Example 11-26. Modifying MakeTestDirectories for permissions test private static string[] MakeTestDirectories() { // ... // Let's make three test directories // and leave space for a fourth to test access denied behavior var directories = new string[4]; for (int i = 0; i < directories.Length - 1; ++i) { ... as before ... } CreateTestFiles(directories.Take(3)); directories[3] = CreateDeniedDirectory(localApplicationData); return directories; } But hold on a moment, before you build and run this. If we’ve denied ourselves per- mission to look at that directory, how are we going to delete it again in our cleanup code? Fortunately, because we own the directory that we created, we can modify the permissions again when we clean up. Finding and Modifying Permissions Example 11-27 shows a method which can give us back full control over any directory (providing we have the permission to change the permissions). This code makes some assumptions about the existing permissions, but that’s OK here because we created the directory in the first place. Example 11-27. Granting access to a directory private static void AllowAccess(string directory) { DirectorySecurity ds = Directory.GetAccessControl(directory); string userName = WindowsIdentity.GetCurrent().Name; 404 | Chapter 11: Files and Streams // Remove the deny rule FileSystemAccessRule fsarDeny = new FileSystemAccessRule( userName, FileSystemRights.ListDirectory, AccessControlType.Deny); ds.RemoveAccessRuleSpecific(fsarDeny); // And add an allow rule FileSystemAccessRule fsarAllow = new FileSystemAccessRule( userName, FileSystemRights.FullControl, AccessControlType.Allow); ds.AddAccessRule(fsarAllow); Directory.SetAccessControl(directory, ds); } Notice how we’re using the GetAccessControl method on Directory to get hold of the directory security information. We then construct a filesystem access rule which matches the deny rule we created earlier, and call RemoveAccessRuleSpecific on the DirectorySecurity information we retrieved. This matches the rule up exactly, and then removes it if it exists (or does nothing if it doesn’t). Finally, we add an allow rule to the set to give us full control over the directory, and then call the Directory.SetAccessControl method to set those permissions on the di- rectory itself. Let’s call that method from our cleanup code, compile, and run. (Don’t forget, we’re deleting files and directories, and changing permissions, so take care!) Here’s some sample output: C:\Users\mwa\AppData\Local\ufmnho4z.h5p C:\Users\mwa\AppData\Local\5chw4maf.xyu C:\Users\mwa\AppData\Local\s1ydovhu.0wk You do not have permission to access this directory. Access to the path 'C:\Users\mwa\AppData\Local\byjijkza.3cj\' is denied. These methods make it relatively easy to manage permissions when you create and manipulate files, but they don’t make it easy to decide what those permissions should be! It is always tempting just to make everything available to anyone—you can get your code compiled and “working” much quicker that way; but only for “not very secure” values of “working,” and that’s something that has to be of concern for every developer. Your application could be the one that miscreants decide to exploit to turn your users’ PCs to the dark side. When Files Go Bad: Dealing with Exceptions | 405 I warmly recommend that you crank UAC up to the maximum (and put up with the occasional security dialog), run Visual Studio as a nonadministrator (as far as is possi- ble), and think at every stage about the least possible privileges you can grant to your users that will still let them get their work done. Making your app more secure benefits everyone: not just your own users, but everyone who doesn’t receive a spam email or a hack attempt because the bad guys couldn’t exploit your application. We’ve now handled the exception nicely—but is stopping really the best thing we could have done? Would it not be better to log the fact that we were unable to access particular directories, and carry on? Similarly, if we get a DirectoryNotFoundException or FileNot FoundException, wouldn’t we want to just carry on in this case? The fact that someone has deleted the directory from underneath us shouldn’t matter to us. If we look again at our sample, it might be better to catch the DirectoryNotFoundExcep tion and FileNotFoundException inside the InspectDirectories method to provide a more fine-grained response to errors. Also, if we look at the documentation for FileInfo, we’ll see that it may actually throw a base IOException under some circum- stances, so we should catch that here, too. And in all cases, we need to catch the security exceptions. We’re relying on LINQ to iterate through the files and folders, which means it’s not entirely obvious where to put the exception handling. Example 11-28 shows the code from InspectDirectories that iterates through the folders, to get a list of files. We can’t put exception handling code into the middle of that query. Example 11-28. Iterating through the directories var allFilePaths = from directory in directoriesToSearch from file in Directory.GetFiles(directory, "*.*", searchOption) select file; However, we don’t have to. The simplest way to solve this is to put the code that gets the directories into a separate method, so we can add exception handling, as Exam- ple 11-29 shows. Example 11-29. Putting exception handling in a helper method private static IEnumerable GetDirectoryFiles( string directory, SearchOption searchOption) { try { return Directory.GetFiles(directory, "*.*", searchOption); } catch (DirectoryNotFoundException dnfx) { Console.WriteLine("Warning: The specified directory was not found"); Console.WriteLine(dnfx.Message); } catch (UnauthorizedAccessException uax) 406 | Chapter 11: Files and Streams { Console.WriteLine( "Warning: You do not have permission to access this directory."); Console.WriteLine(uax.Message); } return Enumerable.Empty(); } This method defers to Directory.GetFiles, but in the event of one of the expected errors, it displays a warning, and then just returns an empty collection. There’s a problem here when we ask GetFiles to search recursively: if it encounters a problem with even just one directory, the whole opera- tion throws, and you’ll end up not looking in any directories. So while Example 11-29 makes a difference only when the user passes multiple directories on the command line, it’s not all that useful when using the /sub option. If you wanted to make your error handling more fine- grained still, you could write your own recursive directory search. The GetAllFilesInDirectory example in Chapter 7 shows how to do that. If we modify the LINQ query to use this, as shown in Example 11-30, the overall pro- gress will be undisturbed by the error handling. Example 11-30. Iterating in the face of errors var allFilePaths = from directory in directoriesToSearch from file in GetDirectoryFiles(directory, searchOption) select file; And we can use a similar technique for the LINQ query that populates the fileNameGroups—it uses FileInfo, and we need to handle exceptions for that. Exam- ple 11-31 iterates through a list of paths, and returns details for each file that it was able to access successfully, displaying errors otherwise. Example 11-31. Handling exceptions from FileInfo private static IEnumerable GetDetails(IEnumerable paths) { foreach (string filePath in paths) { FileDetails details = null; try { FileInfo info = new FileInfo(filePath); details = new FileDetails { FilePath = filePath, FileSize = info.Length }; When Files Go Bad: Dealing with Exceptions | 407 } catch (FileNotFoundException fnfx) { Console.WriteLine("Warning: The specified file was not found"); Console.WriteLine(fnfx.Message); } catch (IOException iox) { Console.Write("Warning: "); Console.WriteLine(iox.Message); } catch (UnauthorizedAccessException uax) { Console.WriteLine( "Warning: You do not have permission to access this file."); Console.WriteLine(uax.Message); } if (details != null) { yield return details; } } } We can use this from the final LINQ query in InspectDirectories. Example 11-32 shows the modified query. Example 11-32. Getting details while tolerating errors var fileNameGroups = from filePath in allFilePaths let fileNameWithoutPath = Path.GetFileName(filePath) group filePath by fileNameWithoutPath into nameGroup select new FileNameGroup { FileNameWithoutPath = nameGroup.Key, FilesWithThisName = GetDetails(nameGroup).ToList() }; Again, this enables the query to process all accessible items, while reporting errors for any problematic files without having to stop completely. If we compile and run again, we see the following output: C:\Users\mwa\AppData\Local\dcyx0fv1.hv3 C:\Users\mwa\AppData\Local\0nf2wqwr.y3s C:\Users\mwa\AppData\Local\kfilxte4.exy Warning: You do not have permission to access this directory. Access to the path 'C:\Users\mwa\AppData\Local\r2gl4q1a.ycp\' is denied. SameNameAndContent.txt ---------------------- C:\Users\mwa\AppData\Local\dcyx0fv1.hv3 C:\Users\mwa\AppData\Local\0nf2wqwr.y3s C:\Users\mwa\AppData\Local\kfilxte4.exy 408 | Chapter 11: Files and Streams We’ve dealt cleanly with the directory to which we did not have access, and have con- tinued with the job to a successful conclusion. Now that we’ve found a few candidate files that may (or may not) be the same, can we actually check to see that they are, in fact, identical, rather than just coincidentally having the same name and length? Reading Files into Memory To compare the candidate files, we could load them into memory. The File class offers three likely looking static methods: ReadAllBytes, which treats the file as binary, and loads it into a byte array; File.ReadAllText, which treats it as text, and reads it all into a string; and File.ReadLines, which again treats it as text, but loads each line into its own string, and returns an array of all the lines. We could even call File.OpenRead to obtain a StreamReader (equivalent to the StreamWriter, but for reading data—we’ll see this again later in the chapter). Because we’re looking at all file types, not just text, we need to use one of the binary- based methods. File.ReadAllBytes returns a byte[] containing the entire contents of the file. We could then compare the files byte for byte, to see if they are the same. Here’s some code to do that. First, let’s update our DisplayMatches function to do the load and compare, as shown by the highlighted lines in Example 11-33. Example 11-33. Updating DisplayMatches for content comparison private static void DisplayMatches( IEnumerable filesGroupedByName) { var groupsWithMoreThanOneFile = from nameGroup in filesGroupedByName where nameGroup.FilesWithThisName.Count > 1 select nameGroup; foreach (var fileNameGroup in groupsWithMoreThanOneFile) { // Group the matches by the file size, then select those // with more than 1 file of that size. var matchesBySize = from match in fileNameGroup.FilesWithThisName group match by match.FileSize into sizeGroup where sizeGroup.Count() > 1 select sizeGroup; foreach (var matchedBySize in matchesBySize) { List content = LoadFiles(matchedBySize); CompareFiles(content); } } } Reading Files into Memory | 409 Notice that we want our LoadFiles function to return a List of FileContents objects. Example 11-34 shows the FileContents class. Example 11-34. File content information class internal class FileContents { public string FilePath { get; set; } public byte[] Content { get; set; } } It just lets us associate the filename with the contents so that we can use it later to display the results. Example 11-35 shows the implementation of LoadFiles, which uses ReadAllBytes to load in the file content. Example 11-35. Loading binary file content private static List LoadFiles(IEnumerable fileList) { var content = new List(); foreach (FileDetails item in fileList) { byte[] contents = File.ReadAllBytes(item.FilePath); content.Add(new FileContents { FilePath = item.FilePath, Content = contents }); } return content; } We now need an implementation for CompareFiles, which is shown in Example 11-36. Example 11-36. CompareFiles method private static void CompareFiles(List files) { Dictionary> potentiallyMatched = BuildPotentialMatches(files); // Now, we're going to look at every byte in each CompareBytes(files, potentiallyMatched); DisplayResults(files, potentiallyMatched); } This isn’t exactly the most elegant way of comparing several files. We’re building a big dictionary of all of the potential matching combinations, and then weeding out the ones that don’t actually match. For large numbers of potential matches of the same size this could get quite inefficient, but we’ll not worry about that right now! Exam- ple 11-37 shows the function that builds those potential matches. 410 | Chapter 11: Files and Streams Example 11-37. Building possible match combinations private static Dictionary> BuildPotentialMatches(List files) { // Builds a dictionary where the entries look like: // { 0, { 1, 2, 3, 4, ... N } } // { 1, { 2, 3, 4, ... N } // ... // { N - 1, { N } } // where N is one less than the number of files. var allCombinations = Enumerable.Range(0, files.Count - 1).ToDictionary( x => files[x], x => files.Skip(x + 1).ToList()); return allCombinations; } This set of potential matches will be whittled down to the files that really are the same by CompareBytes, which we’ll get to momentarily. The DisplayResults method, shown in Example 11-38, runs through the matches and displays their names and locations. Example 11-38. Displaying matches private static void DisplayResults( List files, Dictionary> currentlyMatched) { if (currentlyMatched.Count == 0) { return; } var alreadyMatched = new List(); Console.WriteLine("Matches"); foreach (var matched in currentlyMatched) { // Don't do it if we've already matched it previously if (alreadyMatched.Contains(matched.Key)) { continue; } else { alreadyMatched.Add(matched.Key); } Console.WriteLine("-------"); Console.WriteLine(matched.Key.FilePath); foreach (var file in matched.Value) { Console.WriteLine(file.FilePath); alreadyMatched.Add(file); } } Console.WriteLine("-------"); } Reading Files into Memory | 411 This leaves the method shown in Example 11-39 that does the bulk of the work, com- paring the potentially matching files, byte for byte. Example 11-39. Byte-for-byte comparison of all potential matches private static void CompareBytes( List files, Dictionary> potentiallyMatched) { // Remember, this only ever gets called with files of equal length. int fileLength = files[0].Content.Length; var sourceFilesWithNoMatches = new List(); for (int fileByteOffset = 0; fileByteOffset < fileLength; ++fileByteOffset) { foreach (var sourceFileEntry in potentiallyMatched) { byte[] sourceContent = sourceFileEntry.Key.Content; for (int otherIndex = 0; otherIndex < sourceFileEntry.Value.Count; ++otherIndex) { // Check the byte at i in each of the two files, if they don't // match, then we remove them from the collection byte[] otherContent = sourceFileEntry.Value[otherIndex].Content; if (sourceContent[fileByteOffset] != otherContent[fileByteOffset]) { sourceFileEntry.Value.RemoveAt(otherIndex); otherIndex -= 1; if (sourceFileEntry.Value.Count == 0) { sourceFilesWithNoMatches.Add(sourceFileEntry.Key); } } } } foreach (FileContents fileWithNoMatches in sourceFilesWithNoMatches) { potentiallyMatched.Remove(fileWithNoMatches); } // Don't bother with the rest of the file if // there are no further potential matches if (potentiallyMatched.Count == 0) { break; } sourceFilesWithNoMatches.Clear(); } } We’re going to need to add a test file that differs only in the content. In CreateTest Files add another filename that doesn’t change as we go round the loop: string fileSameSizeInAllButDifferentContent = "SameNameAndSizeDifferentContent.txt"; 412 | Chapter 11: Files and Streams Then, inside the loop (at the bottom), we’ll create a test file that will be the same length, but varying by only a single byte: // And now one that is the same length, but with different content fullPath = Path.Combine(directory, fileSameSizeInAllButDifferentContent); builder = new StringBuilder(); builder.Append("Now with "); builder.Append(directoryIndex); builder.AppendLine(" extra"); CreateFile(fullPath, builder.ToString()); If you build and run, you should see some output like this, showing the one identical file we have in each file location: C:\Users\mwa\AppData\Local\e33yz4hg.mjp C:\Users\mwa\AppData\Local\ung2xdgo.k1c C:\Users\mwa\AppData\Local\jcpagntt.ynd Warning: You do not have permission to access this directory. Access to the path 'C:\Users\mwa\AppData\Local\cmoof2kj.ekd\' is denied. Matches ------- C:\Users\mwa\AppData\Local\e33yz4hg.mjp\SameNameAndContent.txt C:\Users\mwa\AppData\Local\ung2xdgo.k1c\SameNameAndContent.txt C:\Users\mwa\AppData\Local\jcpagntt.ynd\SameNameAndContent.txt ------- Needless to say, this isn’t exactly very efficient; and it is unlikely to work so well when you get to those DVD rips and massive media repositories. Even your 64-bit machine probably doesn’t have quite that much memory available to it.* There’s a way to make this more memory-efficient. Instead of loading the file completely into memory, we can take a streaming approach. Streams You can think of a stream like one of those old-fashioned news ticker tapes. To write data onto the tape, the bytes (or characters) in the file are typed out, one at a time, on the continuous stream of tape. We can then wind the tape back to the beginning, and start reading it back, character by character, until either we stop or we run off the end of the tape. Or we could give the tape to someone else, and she could do the same. Or we could read, say, 1,000 characters off the tape, and copy them onto another tape which we give to someone to work on, then read the next 1,000, and so on, until we run out of characters. * In fact, it is slightly more constrained than that. The .NET Framework limits arrays to 2 GB, and will throw an exception if you try to load a larger file into memory all at once. Streams | 413 Once upon a time, we used to store programs and data in exactly this way, on a stream of paper tape with holes punched in it; the basic tech- nology for this was invented in the 19th century. Later, we got magnetic tape, although that was less than useful in machine shops full of electric motors generating magnetic fields, so paper systems (both tape and punched cards) lasted well into the 1980s (when disk systems and other storage technologies became more robust, and much faster). The concept of a machine that reads data items one at a time, and can step forward or backward through that stream, goes back to the very foundations of modern computing. It is one of those highly resilient metaphors that only really falls down in the face of highly parallelized algorithms: a single input stream is often the choke point for scalability in that case. To illustrate this, let’s write a method that’s equivalent to File.ReadAllBytes using a stream (see Example 11-40). Example 11-40. Reading from a stream private static byte[] ReadAllBytes(string filename) { using (FileStream stream = File.OpenRead(filename)) { long streamLength = stream.Length; if (streamLength > 0x7fffffffL) { throw new InvalidOperationException( "Unable to allocate more than 0x7fffffffL bytes" + "of memory to read the file"); } // Safe to cast to an int, because // we checked for overflow above int bytesToRead = (int) stream.Length; // This could be a big buffer! byte[] bufferToReturn = new byte[bytesToRead]; // We're going to start at the beginning int offsetIntoBuffer = 0; while (bytesToRead > 0) { int bytesRead = stream.Read(bufferToReturn, offsetIntoBuffer, bytesToRead); if (bytesRead == 0) { throw new InvalidOperationException( "We reached the end of file before we expected..." + "Has someone changed the file while we weren't looking?"); } // Read may return fewer bytes than we asked for, so be // ready to go round again. bytesToRead -= bytesRead; offsetIntoBuffer += bytesRead; 414 | Chapter 11: Files and Streams } return bufferToReturn; } } The call to File.OpenRead creates us an instance of a FileStream. This class derives from the base Stream class, which defines most of the methods and properties we’re going to use. First, we inspect the stream’s Length property to determine how many bytes we need to allocate in our result. This is a long, so it can support truly enormous files, even if we can allocate only 2 GB of memory. If you try using the stream.Length argument as the array size without checking it for size first, it will compile, so you might wonder why we’re doing this check. In fact, C# converts the argument to an int first, and if it’s too big, you’ll get an OverflowException at runtime. By checking the size explicitly, we can provide our own error message. Then (once we’ve set up a few variables) we call stream.Read and ask it for all of the data in the stream. It is entitled to give us any number of bytes it likes, up to the number we ask for. It returns the actual number of bytes read, or 0 if we’ve hit the end of the stream and there’s no more data. A common programming error is to assume that the stream will give you as many bytes as you asked for. Under simple test conditions it usually will if there’s enough data. However, streams can and sometimes do return you less in order to give you some data as soon as possible, even when you might think it should be able to give you everything. If you need to read a certain amount before proceeding, you need to write code to keep calling Read until you get what you require, as Exam- ple 11-40 does. Notice that it returns us an int. So even if .NET did let us allocate arrays larger than 2 GB (which it doesn’t) a stream can only tell us that it has read 2 GB worth of data at a time, and in fact, the third argument to Read, where we tell it how much we want, is also an int, so 2 GB is the most we can ask for. So while FileStream is able to work with larger files thanks to the 64-bit Length property, it will split the data into more modest chunks of 2 GB or less when we read. But then one of the main reasons for using streams in the first place is to avoid having to deal with all the content in one go, so in practice we tend to work with much smaller chunks in any case. Streams | 415 So we always call the Read method in a loop. The stream maintains the current read position for us, but we need to work out where to write it in the destination array (offsetIntoBuffer). We also need to work out how many more bytes we have to read (bytesToRead). We can now update the call to ReadAllBytes in our LoadFile method so that it uses our new implementation: byte[] contents = ReadAllBytes(item.Filename); If this was all you were going to do, you wouldn’t actually implement ReadAllBytes yourself; you’d use the one in the framework! This is just by way of an example. We’re going to make more interesting use of streams shortly. Build and run again, and you should see output with exactly the same form as before: C:\Users\mwa\AppData\Local\1ssoimgj.wqg C:\Users\mwa\AppData\Local\cjiymq5b.bfo C:\Users\mwa\AppData\Local\diss5tgl.zae Warning: You do not have permission to access this directory. Access to the path 'C:\Users\mwa\AppData\Local\u1w0rj0o.2xe\' is denied. Matches ------- C:\Users\mwa\AppData\Local\1ssoimgj.wqg\SameNameAndContent.txt C:\Users\mwa\AppData\Local\cjiymq5b.bfo\SameNameAndContent.txt C:\Users\mwa\AppData\Local\diss5tgl.zae\SameNameAndContent.txt ------- That’s all very well, but we haven’t actually improved anything. We wanted to avoid loading all of those files into memory. Instead of loading the files, let’s update our FileContents class to hold a stream instead of a byte array, as Example 11-41 shows. Example 11-41. FileContents using FileStream internal class FileContents { public string FilePath { get; set; } public FileStream Content { get; set; } } We’ll have to update the code that creates the FileContents too, in our LoadFiles method from Example 11-35. Example 11-42 shows the change required. Example 11-42. Modifying LoadFiles content.Add(new FileContents { FilePath = item.FilePath, Content = File.OpenRead(item.FilePath) }); 416 | Chapter 11: Files and Streams (You can now delete our ReadAllBytes implementation, if you want.) Because we’re opening all of those files, we need to make sure that we always close them all. We can’t implement the using pattern, because we’re handing off the refer- ences outside the scope of the function that creates them, so we’ll have to find some- where else to call Close. DisplayMatches (Example 11-33) ultimately causes the streams to be created by calling LoadFiles, so DisplayMatches should close them too. We can add a try/finally block in that method’s innermost foreach loop, as Example 11-43 shows. Example 11-43. Closing streams in DisplayMatches foreach (var matchedBySize in matchesBySize) { List content = LoadFiles(matchedBySize); try { CompareFiles(content); } finally { foreach (var item in content) { item.Content.Close(); } } } The last thing to update, then, is the CompareBytes method. The previous version, shown in Example 11-39, relied on loading all the files into memory upfront. The modified version in Example 11-44 uses streams. Example 11-44. Stream-based CompareBytes private static void CompareBytes( List files, Dictionary> potentiallyMatched) { // Remember, this only ever gets called with files of equal length. long bytesToRead = files[0].Content.Length; // We work through all the files at once, so allocate a buffer for each. Dictionary fileBuffers = files.ToDictionary(x => x, x => new byte[1024]); var sourceFilesWithNoMatches = new List(); while (bytesToRead > 0) { // Read up to 1k from all the files. int bytesRead = 0; foreach (var bufferEntry in fileBuffers) { FileContents file = bufferEntry.Key; byte[] buffer = bufferEntry.Value; Streams | 417 int bytesReadFromThisFile = 0; while (bytesReadFromThisFile < buffer.Length) { int bytesThisRead = file.Content.Read( buffer, bytesReadFromThisFile, buffer.Length - bytesReadFromThisFile); if (bytesThisRead == 0) { break; } bytesReadFromThisFile += bytesThisRead; } if (bytesReadFromThisFile < buffer.Length && bytesReadFromThisFile < bytesToRead) { throw new InvalidOperationException( "Unexpected end of file - did a file change?"); } bytesRead = bytesReadFromThisFile; // Will be same for all files } bytesToRead -= bytesRead; foreach (var sourceFileEntry in potentiallyMatched) { byte[] sourceFileContent = fileBuffers[sourceFileEntry.Key]; for (int otherIndex = 0; otherIndex < sourceFileEntry.Value.Count; ++otherIndex) { byte[] otherFileContent = fileBuffers[sourceFileEntry.Value[otherIndex]]; for (int i = 0; i < bytesRead; ++i) { if (sourceFileContent[i] != otherFileContent[i]) { sourceFileEntry.Value.RemoveAt(otherIndex); otherIndex -= 1; if (sourceFileEntry.Value.Count == 0) { sourceFilesWithNoMatches.Add(sourceFileEntry.Key); } break; } } } } foreach (FileContents fileWithNoMatches in sourceFilesWithNoMatches) { potentiallyMatched.Remove(fileWithNoMatches); } // Don't bother with the rest of the file if there are // not further potential matches if (potentiallyMatched.Count == 0) { break; } sourceFilesWithNoMatches.Clear(); 418 | Chapter 11: Files and Streams } } Rather than reading entire files at once, we allocate small buffers, and read in 1 KB at a time. As with the previous version, this new one works through all the files of a particular name and size simultaneously, so we allocate a buffer for each file. We then loop round, reading in a buffer’s worth from each file, and perform compar- isons against just that buffer (weeding out any nonmatches). We keep going round until we either determine that none of the files match or reach the end of the files. Notice how each stream remembers its position for us, with each Read starting where the previous one left off. And since we ensure that we read exactly the same quantity from all the files for each chunk (either 1 KB, or however much is left when we get to the end of the file), all the streams advance in unison. This code has a somewhat more complex structure than before. The all-in-memory version in Example 11-39 had three loops—the outer one advanced one byte at a time, and then the inner two worked through the various potential match combinations. But because the outer loop in Example 11-44 advances one chunk at a time, we end up needing an extra inner loop to compare all the bytes in a chunk. We could have sim- plified this by only ever reading a single byte at a time from the streams, but in fact, this chunking has delivered a significant performance improvement. Testing against a folder full of source code, media resources, and compilation output containing 4,500 files (totaling about 500 MB), the all-in-memory version took about 17 seconds to find all the duplicates, but the stream version took just 3.5 seconds! Profiling the code re- vealed that this performance improvement was entirely a result of the fact that we were comparing the bytes in chunks. So for this particular application, the additional com- plexity was well worth it. (Of course, you should always measure your own code against representative problems—techniques that work well in one scenario don’t necessarily perform well everywhere.) Moving Around in a Stream What if we wanted to step forward or backward in the file? We can do that with the Seek method. Let’s imagine we want to print out the first 100 bytes of each file that we reject, for debug purposes. We can add some code to our CompareBytes method to do that, as Example 11-45 shows. Example 11-45. Seeking within a stream if (sourceFileContent[i] != otherFileContent[i]) { sourceFileEntry.Value.RemoveAt(otherIndex); otherIndex -= 1; if (sourceFileEntry.Value.Count == 0) { sourceFilesWithNoMatches.Add(sourceFileEntry.Key); Streams | 419 } #if DEBUG // Remember where we got to long currentPosition = sourceFileEntry.Key.Content.Position; // Seek to 0 bytes from the beginning sourceFileEntry.Key.Content.Seek(0, SeekOrigin.Begin); // Read 100 bytes from for (int index = 0; index < 100; ++index) { var val = sourceFileEntry.Key.Content.ReadByte(); if (val < 0) { break; } if (index != 0) { Console.Write(", "); } Console.Write(val); } Console.WriteLine(); // Put it back where we found it sourceFileEntry.Key.Content.Seek(currentPosition, SeekOrigin.Begin); #endif break; } We start by getting hold of the current position within the stream using the Position property. We do this so that the code doesn’t lose its place in the stream. (Even though we’ve detected a mismatch here, remember we’re comparing lots of files here—perhaps this same file matches one of the other candidates. So we’re not necessarily finished with it yet.) The first parameter of the Seek method tells us how far we are going to seek from our origin—we’re passing 0 here because we want to go to the beginning of the file. The second tells us what that origin is going to be. SeekOrigin.Begin means the beginning of the file, SeekOrigin.End means the end of the file (and so the offset counts backward—you don’t need to say −100, just 100). There’s also SeekOrigin.Current which allows you to move relative to the current po- sition. You could use this to read 10 bytes ahead, for example (maybe to work out what you were looking at in context), and then seek back to where you were by calling Seek(-10, SeekOrigin.Current). Not all streams support seeking. For example, some streams represent network connections, which you might use to download gigabytes of data. The .NET Framework doesn’t remember every single byte just in case you ask it to seek later on, so if you attempt to rewind such a stream, Seek will throw a NotSupportedException. You can find out whether seeking is supported from a stream’s CanSeek property. 420 | Chapter 11: Files and Streams Writing Data with Streams We don’t just have to use streaming APIs for reading. We can write to the stream, too. One very common programming task is to copy data from one stream to another. We use this kind of thing all the time—copying data, or concatenating the content of several files into another, for example. (If you want to copy an entire file, you’d use File.Copy, but streams give you the flexibility to concatenate or modify data, or to work with nonfile sources.) Example 11-46 shows how to read data from one stream and write it into another. This is just for illustrative purposes—.NET 4 added a new CopyTo method to Stream which does this for you. In practice you’d need Example 11-46 only if you were targeting an older version of the .NET Framework, but it’s a good way to see how to write to a stream. Example 11-46. Copying from one stream to another private static void WriteTo(Stream source, Stream target, int bufferLength) { bufferLength = Math.Max(100, bufferLength); var buffer = new byte[bufferLength]; int bytesRead; do { bytesRead = source.Read(buffer, 0, buffer.Length); if (bytesRead != 0) { target.Write(buffer, 0, bytesRead); } } while (bytesRead > 0); } We create a buffer which is at least 100 bytes long. We then Read from the source and Write to the target, using the buffer as the intermediary. Notice that the Write method takes the same parameters as the read: the buffer, an offset into that buffer, and the number of bytes to write (which in this case is the number of bytes read from the source buffer, hence the slightly confusing variable name). As with Read, it steadily advances the current position in the stream as it writes, just like that ticker tape. Unlike Read, Write will always process as many bytes as we ask it to, so with Write, there’s no need to keep looping round until it has written all the data. Obviously, we need to keep looping until we’ve read everything from the source stream. Notice that we keep going until Read returns 0. This is how streams indicate that we’ve reached the end. (Some streams don’t know in advance how large they are, so you can rely on the Length property for only certain kinds of streams such as FileStream. Testing for a return value of 0 is the most general way to know that we’ve reached the end.) Streams | 421 Reading, Writing, and Locking Files So, we’ve seen how to read and write data to and from streams, and how we can move the current position in the stream by seeking to some offset from a known position. Up until now, we’ve been using the File.OpenRead and File.OpenWrite methods to create our file streams. There is another method, File.Open, which gives us access to some extra features. The simplest overload takes two parameters: a string which is the path for the file, and a value from the FileMode enumeration. What’s the FileMode? Well, it lets us specify exactly what we want done to the file when we open it. Table 11-6 shows the values available. Table 11-6. FileMode enumeration FileMode Purpose CreateNew Creates a brand new file. Throws an exception if it already existed. Create Creates a new file, deleting any existing file and overwriting it if necessary. Open Opens an existing file, seeking to the beginning by default. Throws an exception if the file does not exist. OpenOrCreate Opens an existing file, or creates a new file if it doesn’t exist. Truncate Opens an existing file, and deletes all its contents. The file is automatically opened for writing only. Append Opens an existing file and seeks to the end of the file. The file is automatically opened for writing only. You can seek in the file, but only within any information you’ve appended—you can’t touch the existing content. If you use this two-argument overload, the file will be opened in read/write mode. If that’s not what you want, another overload takes a third argument, allowing you to control the access mode with a value from the FileAccess enumeration. Table 11-7 shows the supported values. Table 11-7. FileAccess enumeration FileAccess Purpose Read Open read-only. Write Open write-only. ReadWrite Open read/write. All of the file-opening methods we’ve used so far have locked the file for our exclusive use until we close or Dispose the object—if any other program tries to open the file while we have it open, it’ll get an error. However, it is possible to play nicely with other users by opening the file in a shared mode. We do this by using the overload which specifies a value from the FileShare enumeration, which is shown in Table 11-8. This is a flags enumeration, so you can combine the values if you wish. 422 | Chapter 11: Files and Streams Table 11-8. FileShare enumeration FileShare Purpose None No one else can open the file while we’ve got it open. Read Other people can open the file for reading, but not writing. Write Other people can open the file for writing, but not reading (so read/write will fail, for example). ReadWrite Other people can open the file for reading or writing (or both). This is equivalent to Read | Write. Delete Other people can delete the file that you’ve created, even while we’ve still got it open. Use with care! You have to be careful when opening files in a shared mode, particularly one that permits modifications. You are open to all sorts of potential exceptions that you could normally ignore (e.g., people deleting or truncating it from underneath you). If you need even more control over the file when you open it, you can create a FileStream instance directly. FileStream Constructors There are two types of FileStream constructors—those for interop scenarios, and the “normal” ones. The “normal” ones take a string for the file path, while the interop ones require either an IntPtr or a SafeFileHandle. These wrap a Win32 file handle that you have retrieved from somewhere. (If you’re not already using such a thing in your code, you don’t need to use these versions.) We’re not going to cover the interop scenarios here. If you look at the list of constructors, the first thing you’ll notice is that quite a few of them duplicate the various permutations of FileShare, FileAccess, and FileMode over- loads we had on File.Open. You’ll also notice equivalents with one extra int parameter. This allows you to provide a hint for the system about the size of the internal buffer you’d like the stream to use. Let’s look at buffering in more detail. Stream Buffers Many streams provide buffering. This means that when you read and write, they actually use an intermediate in-memory buffer. When writing, they may store your data in an internal buffer, before periodically flushing the data to the actual output device. Simi- larly, when you read, they might read ahead a whole buffer full of data, and then return to you only the particular bit you need. In both cases, buffering aims to reduce the number of I/O operations—it means you can read or write data in relatively small increments without incurring the full cost of an operating system API call every time. FileStream Constructors | 423 There are many layers of buffering for a typical storage device. There might be some memory buffering on the actual device itself (many hard disks do this, for example), the filesystem might be buffered (NTFS always does read buffering, and on a client operating system it’s typically write-buffered, although this can be turned off, and is off by default for the server configurations of Windows). The .NET Framework pro- vides stream buffering, and you can implement your own buffers (as we did in our example earlier). These buffers are generally put in place for performance reasons. Although the default buffer sizes are chosen for a reasonable trade-off between performance and robustness, for an I/O-intensive application, you may need to hand-tune this using the appropriate constructors on FileStream. As usual, you can do more harm than good if you don’t measure the impact on performance carefully on a suitable range of your target sys- tems. Most applications will not need to touch this value. Even if you don’t need to tune performance, you still need to be aware of buffering for robustness reasons. If either the process or the OS crashes before the buffers are written out to the physical disk, you run the risk of data loss (hence the reason write buffering is typically disabled on the server). If you’re writing frequently to a Stream or StreamWriter, the .NET Framework will flush the write buffers periodically. It also ensures that everything is properly flushed when the stream is closed. However, if you just stop writing data but you leave the stream open, there’s a good chance data will hang around in memory for a long time without getting written out, at which point data loss starts to become more likely. In general, you should close files as early as possible, but sometimes you’ll want to keep a file open for a long time, yet still ensure that particular pieces of data get written out. If you need to control that yourself, you can call Flush. This is particularly useful if you have multiple threads of execution accessing the same stream. You can synchronize writes and ensure that they are flushed to disk before the next worker gets in and messes things up! Later in this chapter, we’ll see an example where explicit flushing is extremely important. Setting Permissions During Construction Another parameter we can set in the constructor is the FileSystemRights. We used this type earlier in the chapter to set filesystem permissions. FileStream lets us set these directly when we create a file using the appropriate constructor. Similarly, we can also specify an instance of a FileSecurity object to further control the permissions on the underlying file. 424 | Chapter 11: Files and Streams Setting Advanced Options Finally, we can optionally pass another enumeration to the FileStream constructor, FileOptions, which contains some advanced filesystem options. They are enumerated in Table 11-9. This is a flags-style enumeration, so you can combine these values. Table 11-9. FileOptions enumeration FileOptions Purpose None No options at all. WriteThrough Ignores any filesystem-level buffers, and writes directly to the output device. This affects only the O/S, and not any of the other layers of buffering, so it’s still your responsibility to call Flush. RandomAccess Indicates that we’re going to be seeking about in the file in an unsystematic way. This acts as a hint to the OS for its caching strategy. We might be writing a video-editing tool, for example, where we expect the user to be leaping about through the file. SequentialScan Indicates that we’re going to be sequentially reading from the file. This acts as a hint to the OS for its caching strategy. We might be writing a video player, for example, where we expect the user to play through the stream from beginning to end. Encrypted Indicates that we want the file to be encrypted so that it can be decrypted and read only by the user who created it. DeleteOnClose Deletes the file when it is closed. This is very handy for temporary files. If you use this option, you never hit the problem where the file still seems to be locked for a short while even after you’ve closed it (because its buffers are still flushing asynchronously). Asynchronous Allows the file to be accessed asynchronously. The last option, Asynchronous, deserves a section all to itself. Asynchronous File Operations Long-running file operations are a common bottleneck. How many times have you clicked the Save button, and seen the UI lock up while the disk operation takes place (especially if you’re saving a large file to a network location)? Developers commonly resort to a background thread to push these long operations off the main thread so that they can display some kind of progress or “please wait” UI (or let the user carry on working). We’ll look at that approach in Chapter 16; but you don’t necessarily have to go that far. You can use the asynchronous mode built into the stream instead. To see how it works, look at Example 11-47. Example 11-47. Asynchronous file I/O static void Main(string[] args) { string path = "mytestfile.txt"; // Create a test file using (var file = File.Create(path, 4096, FileOptions.Asynchronous)) Asynchronous File Operations | 425 { // Some bytes to write byte[] myBytes = new byte[] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 }; IAsyncResult asyncResult = file.BeginWrite( myBytes, 0, myBytes.Length, // A callback function, written as an anonymous delegate delegate(IAsyncResult result) { // You *must* call EndWrite() exactly once file.EndWrite(result); // Then do what you like Console.WriteLine( "Called back on thread {0} when the operation completed", System.Threading.Thread.CurrentThread.ManagedThreadId); }, null); // You could do something else while you waited... Console.WriteLine( "Waiting on thread {0}...", System.Threading.Thread.CurrentThread.ManagedThreadId); // Waiting on the main thread asyncResult.AsyncWaitHandle.WaitOne(); Console.WriteLine( "Completed {0} on thread {1}...", asyncResult.CompletedSynchronously ? "synchronously" : "asynchronously", System.Threading.Thread.CurrentThread.ManagedThreadId); Console.ReadKey(); return; } } If you put this code in a new console application, and then compile and run, you’ll get output similar to this (the actual thread IDs will vary from run to run): Waiting on thread 10... Completed asynchronously on thread 10... Called back on thread 6 when the operation completed So, what is happening? When we create our file, we use an overload on File.Create that takes the FileOptions we discussed earlier. (Yes, back then we showed that by constructing the FileStream directly, but the File class supports this too.) This lets us open the file with asynchronous behavior enabled. Then, instead of calling Write, we call BeginWrite. This takes two additional parameters. The first is a delegate to a callback function of type AsyncCallback, which the framework will call when it has finished the operation to let us know that it has completed. The second is an object that we can pass in, that will get passed back to us in the callback. 426 | Chapter 11: Files and Streams This user state object is common to a lot of asynchronous operations, and is used to get information from the calling site to callbacks from the worker thread. It has become less useful in C# with the availability of lambdas and anonymous methods which have access to variables in their enclosing state. We’ve used an anonymous method to provide the callback delegate. The first thing we do in that method is to call file.EndWrite, passing it the IAsyncResult we’ve been provided in the callback. You must call EndWrite exactly once for every time you call BeginWrite, because it cleans up the resources used to carry out the operation asyn- chronously. It doesn’t matter whether you call it from the callback, or on the main application thread (or anywhere else, for that matter). If the operation has not com- pleted, it will block the calling thread until it does complete, then do its cleanup. Should you call it twice with the same IAsyncResult for any reason the framework will throw an exception. In a typical Windows Forms or WPF application, we’d probably put up some progress dialog of some kind, and just process messages until we got our callback. In a server- side application we’re more likely to want to kick off several pieces of work like this, and then wait for them to finish. To do this, the IAsyncResult provides us with an AsyncWaitHandle, which is an object we can use to block our thread until the work is complete. So, when we run, our main thread happens to have the ID 10. It blocks until the oper- ation is complete, and then prints out the message about being done. Notice that this was, as you’d expect, on the same thread with ID 10. But after that, we get a message printed out from our callback, which was called by the framework on another thread entirely. It is important to note that your system may have behaved differently. It is possible that the callback might occur before execution continued on the main thread. You have to be extremely careful that your code doesn’t depend on these operations happening in a particular order. We’ll discuss these issues in a lot more detail in Chapter 16. We recommend you read that before you use any of these asynchronous techniques in production code. Remember that we set the FileOptions.Asynchronous flag when we opened the file to get this asynchronous behavior? What happens if we don’t do that? Let’s tweak the code so that it opens with FileOptions.None instead, and see. Example 11-48 shows the statements from Example 11-47 that need to be modified Asynchronous File Operations | 427 Example 11-48. Not asking for asynchronous behavior ... // Create a test file using (var file = File.Create(path, 4096, FileOptions.None)) { ... If you build and run that, you’ll see some output similar to this: Waiting on thread 9... Completed asynchronously on thread 9... Called back on thread 10 when the operation completed What’s going on? That all still seemed to be asynchronous! Well yes, it was, but under the covers, the problem was solved in two different ways. The first one used the underlying support Windows provides for asynchronous I/O in the filesystem to handle the asynchronous file operation. In the second case, the .NET Framework had to do some work for us to grab a thread from the thread pool, and execute the read operation on that to deliver the asynchronous behavior. That’s true right now, but bear in mind that these are implementation details and could change in future versions of the framework. The prin- ciple will remain the same, though. So far, everything we’ve talked about has been related to files, but we can create streams over other things, too. If you’re a Silverlight developer, you’ve probably been skimming over all of this a bit—after all, if you’re running in the web browser you can’t actually read and write files in the filesystem. There is, however, another option that you can use (along with all the other .NET developers out there): isolated storage. Isolated Storage In the duplicate file detection application we built earlier in this chapter, we had to go to some lengths to find a location, and pick filenames for the datafiles we wished to create in test mode, in order to guarantee that we don’t collide with other applications. We also had to pick locations that we knew we would (probably) have permission to write to, and that we could then load again. Isolated storage takes this one stage further and gives us a means of saving and loading data in a location unique to a particular piece of executing code. The physical location itself is abstracted away behind the API; we don’t need to know where the runtime is actually storing the data, just that the data is stored safely, and that we can retrieve it again. (Even if we want to know where the files are, the isolated storage API won’t tell us.) This helps to make the isolated storage framework a bit more operating-system- agnostic, and removes the need for full trust (unlike regular file I/O). Hence it can be 428 | Chapter 11: Files and Streams used by Silverlight developers (who can target other operating systems such as Mac OS X) as well as those of us building server or desktop client applications for Windows. This compartmentalization of the information by characteristics of the executing code gives us a slightly different security model from regular files. We can constrain access to particular assemblies, websites, and/or users, for instance, through an API that is much simpler (although much less sophisticated) than the regular file security. Although isolated storage provides you with a simple security model to use from managed code, it does not secure your data effectively against unmanaged code running in a relatively high trust context and trawling the local filesystem for information. So, you should not trust sensitive data (credit card numbers, say) to isolated storage. That being said, if someone you cannot trust has successfully run unmanaged code in a trusted context on your box, isolated storage is probably the least of your worries. Stores Our starting point when using isolated storage is a store and you can think of any given store as being somewhat like one of the well-known directories we dealt with in the regular filesystem. The framework creates a folder for you when you first ask for a store with a particular set of isolation criteria, and then gives back the same folder each time you ask for the store with the same criteria. Instead of using the regular filesystem APIs, we then use special methods on the store to create, move, and delete files and directories within that store. First, we need to get hold of a store. We do that by calling one of several static members on the IsolatedStorageFile class. Example 11-49 starts by getting the user store for a particular assembly. We’ll discuss what that means shortly, but for now it just means we’ve got some sort of a store we can use. It then goes on to create a folder and a file that we can use to cache some information, and retrieve it again on subsequent runs of the application. Example 11-49. Creating folders and files in a store static void Main(string[] args) { IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForAssembly(); // Create a directory - safe to call multiple times store.CreateDirectory("Settings"); // Open or create the file using (IsolatedStorageFileStream stream = store.OpenFile( "Settings\\standardsettings.txt", System.IO.FileMode.OpenOrCreate, System.IO.FileAccess.ReadWrite)) { UseStream(stream); } Isolated Storage | 429 Console.ReadKey(); } We create a directory in the store, called Settings. You don’t have to do this; you could put your file in the root directory for the store, if you wanted. Then, we use the OpenFile method on the store to open a file. We use the standard file path syntax to specify the file, relative to the root for this store, along with the FileMode and FileAc cess values that we’re already familiar with. They all mean the same thing in isolated storage as they do with normal files. That method returns us an IsolatedStorageFile Stream. This class derives from FileStream, so it works in pretty much the same way. So, what shall we do with it now that we’ve got it? For the purposes of this example, let’s just write some text into it if it is empty. On a subsequent run, we’ll print the text we wrote to the console. Reading and Writing Text We’ve already seen StreamWriter, the handy wrapper class we can use for writing text to a stream. Previously, we got hold of one from File.CreateText, but remember we mentioned that there’s a constructor we can use to wrap any Stream (not just a FileStream) if we want to write text to it? Well, we can use that now, for our Isolated StorageFileStream. Similarly, we can use the equivalent StreamReader to read text from the stream if it already exists. Example 11-50 implements the UseStream method that Example 11-49 called after opening the stream, and it uses both StreamReader and StreamWriter. Example 11-50. Using StreamReader and StreamWriter with isolated storage static void UseStream(Stream stream) { if (stream.Length > 0) { using (StreamReader reader = new StreamReader(stream)) { Console.WriteLine(reader.ReadToEnd()); } } else { using (StreamWriter writer = new StreamWriter(stream)) { writer.WriteLine( "Initialized settings at {0}", DateTime.Now.TimeOfDay); Console.WriteLine("Settings have been initialized"); } } } In the case where we’re writing, we construct our StreamWriter (in a using block, be- cause we need to Dispose it when we’re done), and then use the WriteLine method to 430 | Chapter 11: Files and Streams write our content. Remember that WriteLine adds an extra new line on the end of the text, whereas Write just writes the text provided. In the case where we are reading, on the other hand, we construct a StreamReader (also in a using block), and then read the entire content using ReadToEnd. This reads the entire content of the file into a single string. So, if you build and run this once, you’ll see some output that looks a lot like this: Settings have been initialized That means we’ve run through the write path. Run a second (or subsequent) time, and you’ll see something more like this: Initialized settings at 10:34:47.7014833 That means we’ve run through the read path. When you run this, you’ll notice that we end up outputting an extra blank line at the end, because we’ve read a whole line from the file—we called writer.WriteLine when generating the file—and then used Console.WriteLine, which adds another end of line after that. You have to be a little careful when manipulating text like this, to ensure that you don’t end up with huge amounts of unwanted whitespace because ev- eryone in some processing chain is generously adding new lines or other whitespace at the end! This is a rather neat result. We can use all our standard techniques for reading and writing to an IsolatedStorageFileStream once we’ve acquired a suitable file: the other I/O types such as StreamReader don’t need to know what kind of stream we’re using. Defining “Isolated” So, what makes isolated storage “isolated”? The .NET Framework partitions informa- tion written into isolated storage based on some characteristics of the executing code. Several types of isolated store are available to you: • Isolation by user and assembly (optionally supporting roaming) • Isolation by user, domain, and assembly (optionally supporting roaming) • Isolation by user and application (optionally supporting roaming) • Isolation by user and site (only on Silverlight) • Isolation by machine and assembly • Isolation by machine, domain, and assembly • Isolation by machine and application Silverlight supports only two of these: by user and site, and by user and application. Isolated Storage | 431 Isolation by user and assembly In Example 11-50, we acquired a store isolated by user and assembly, using the static method IsolatedStorageFile.GetUserStoreForAssembly. This store is unique to a par- ticular user, and the assembly in which the calling code is executing. You can try this out for yourself. If you log in to your box as a user other than the one under which you’ve already run our example app, and run it again, you’ll see some output like this: Settings have been initialized That means our settings file doesn’t exist (for this user), so we must have been given a new store. As you might expect, the user is identified by the authenticated principal for the current thread. Typically, this is the logged-on user that ran the process; but this could have been changed by impersonation (in a web application, for example, you might be run- ning in the context of the web user, rather than that of the ASP.NET process that hosts the site). Identifying the assembly is slightly more complex. If you have signed the assembly, it uses the information in that signature (be it a strong name signature, or a software publisher signature, with the software publishing signature winning if it has both). If, on the other hand, the assembly is not signed, it will use the URL for the assembly. If it came from the Internet, it will be of the form: http://some/path/to/myassembly.dll If it came from the local filesystem, it will be of the form: file:///C:/some/path/to/myassembly.dll Figure 11-9 illustrates how multiple stores get involved when you have several users and several different assemblies. User 1 asks MyApp.exe to perform some task, which asks for user/assembly isolated storage. It gets Store 1. Imagine that User 1 then asks MyApp.exe to perform some other task that requires the application to call on MyAs- sembly.dll to carry out the work. If that in turn asks for user/assembly isolated storage, it will get a different store (labeled Store 2 in the diagram). We get a different store, because they are different assemblies. When a different user, User 2, asks MyApp.exe to perform the first task, which then asks for user/assembly isolated storage, it gets a different store again—Store 3 in the diagram—because they are different users. OK, what happens if we make two copies of MyApp.exe in two different locations, and run them both under the same user account? The answer is that it depends.... If the applications are not signed the assembly identification rules mean that they don’t match, and so we get two different isolated stores. If they are signed the assembly identification rules mean that they do match, so we get the same isolated store. 432 | Chapter 11: Files and Streams Our app isn’t signed, so if we try this experiment, we’ll see the standard “first run” output for our second copy. Be very careful when using isolated storage with signed assemblies. The information used from the signature includes the Name, Strong Name Key, and Major Version part of the version info. So, if you rev your application from 1.x to 2.x, all of a sudden you’re getting a different isolated storage scope, and all your existing data will “vanish.” One way to deal with this is to use a distinct DLL to access the store, and keep its version numbers constant. Isolation by user, domain, and assembly Isolating by domain means that we look for some information about the application domain in which we are running. Typically, this is the full URL of the assembly if it was downloaded from the Web, or the local path of the file. Notice that this is the same rule as for the assembly identity if we didn’t sign it! The purpose of this isolation model is to allow a single signed assembly to get different stores if it is run from different locations. You can see a diagram that illustrates this in Figure 11-10. Figure 11-9. User and assembly isolation Isolated Storage | 433 To get a store with this isolation level, we can call the IsolatedStorageFile class’s GetUserStoreForDomain method. Isolation by user and application A third level of isolation is by user and application. What defines an “application”? Well, you have to sign the whole lot with a publisher’s (Authenticode) signature. A regular strong-name signature won’t do (as that will identify only an individual assembly). If you want to try this out quickly for yourself, you can run the Click- Once Publication Wizard on the Publish tab of your example project settings. This will generate a suitable test certificate and sign the app. To get a store with user and application isolation, we call the IsolatedStorageFile class’s GetUserStoreForApplication method. Figure 11-10. Assembly and domain isolation compared 434 | Chapter 11: Files and Streams If you haven’t signed your application properly, this method will throw an exception. So, it doesn’t matter which assembly you call from; as long as it is a part of the same application, it will get the same store. You can see this illustrated in Figure 11-11. Figure 11-11. Application isolation This can be particularly useful for settings that might be shared between several different application components. Machine isolation What if your application or component has some data you want to make available to all users on the system? Maybe you want to cache common product information or imagery to avoid a download every time you start the app. For these scenarios you need machine isolation. Isolated Storage | 435 As you saw earlier, there is an isolation type for the machine which corresponds to each isolation type for the user. The same resolution rules apply in each case. The methods you need are: GetMachineStoreForApplication GetMachineStoreForDomain GetMachineStoreForAssembly Managing User Storage with Quotas Isolated storage has the ability to set quotas on particular storage scopes. This allows you to limit the amount of data that can be saved in any particular store. This is par- ticularly important for applications that run with partial trust—you wouldn’t want Silverlight applications automatically loaded as part of a web page to be able to store vast amounts of data on your hard disk without your permission. You can find out a store’s current quota by looking at the Quota property on a particular IsolatedStorageFile. This is a long, which indicates the maximum number of bytes that may be stored. This is not a “bytes remaining” count—you can use the Available FreeSpace property for that. Your available space will go down slightly when you create empty di- rectories and files. This reflects the fact that such items consume space on disk even though they are nominally empty. The quota can be increased using the IncreaseQuotaTo method, which takes a long which is the new number of bytes to which to limit the store. This must be larger than the previous number of bytes, or an ArgumentException is thrown. This call may or may not succeed—the user will be prompted, and may refuse your request for more space. You cannot reduce the quota for a store once you’ve set it, so take care! Managing Isolated Storage As a user, you might want to look at the data stored in isolated storage by applications running on your machine. It can be complicated to manage and debug isolated storage, but there are a few tools and techniques to help you. First, there’s the storeadm.exe tool. This allows you to inspect isolated storage for the current user (by default), or the current machine (by specifying the /machine option) or current roaming user (by specifying /roaming). 436 | Chapter 11: Files and Streams So, if you try running this command: storeadm /MACHINE /LIST you will see output similar to this (listing the various stores for this machine, along with the evidence that identifies them): Microsoft (R) .NET Framework Store Admin 4.0.30319.1 Copyright (c) Microsoft Corporation. All rights reserved. Record #1 [Assembly] Size : 0 Record #2 [Domain] [Assembly] Size : 0 Notice that there are two stores in that example. One is identified by some assembly evidence (the strong name key, name, and major version info). The other is identified by both domain and assembly evidence. Because the sample application is in a single assembly, the assembly evidence for both stores happens to be identical! You can also add the /REMOVE parameter which will delete all of the isolated storage in use at the specified scope. Be very careful if you do this, as you may well delete storage used by another application entirely. Isolated Storage | 437 That’s all very well, but you can’t see the place where those files are stored. That’s because the actual storage is intended to be abstracted away behind the API. Sometimes, however, it is useful to be able to go and pry into the actual storage itself. Remember, this is an implementation detail, and it could change be- tween versions. It has been consistent since the first version of the .NET Framework, but in the future, Microsoft could decide to store it all in one big file hidden away somewhere, or using some mystical API that we don’t have access to. We can take advantage of the fact that the debugger can show us the private innards of the IsolatedStorageFile class. If we set a breakpoint on the store.CreateFile line in our sample application, we can inspect the IsolatedStorageFile object that was returned by GetUserStoreForApplication in the previous line. You will see that there is a private field called m_RootDir. This is the actual root directory (in the real filesystem) for the store. You can see an example of that as it is on my machine in Figure 11-12. Figure 11-12. IsolatedStorageFile internals If you copy that path and browse to it using Windows Explorer, you’ll see something like the folder in Figure 11-13. There’s the Settings directory that we created! As you might expect, if you were to look inside, you’d see the standardsettings.txt file our program created. 438 | Chapter 11: Files and Streams Figure 11-13. An isolated storage folder As you can see, this is a very useful debugging technique, allowing you to inspect and modify the contents of files in isolated storage, and identify exactly which store you have for a particular scope. It does rely on implementation details, but since you’d only ever do this while debugging, the code you ultimately ship won’t depend on any non- public features of isolated storage. OK. So far, we’ve seen two different types of stream; a regular file, and an isolated storage file. We use our familiar stream tools and techniques (like StreamReader and StreamWriter), regardless of the underlying type. So, what other kinds of stream exist? Well, there are lots; several subsystems in the .NET framework provide stream-based APIs. We’ll see some networking ones in Chap- ter 13, for example. Another example is from the .NET Framework’s security features: CryptoStream (which is used for encrypting and decrypting a stream of data). There’s also a MemoryStream in System.IO which uses memory to store the data in the stream. Streams That Aren’t Files In this final section, we’ll look at a stream that is not a file. We’ll use a stream from .NET’s cryptographic services to encrypt a string. This encrypted string can be decrypted later as long as we know the key. The test program in Example 11-51 illus- trates this. Example 11-51. Using an encryption stream static void Main(string[] args) { byte[] key; byte[] iv; // Get the appropriate key and initialization vector for the algorithm SelectKeyAndIV(out key, out iv); Streams That Aren’t Files | 439 string superSecret = "This is super secret"; Console.WriteLine(superSecret); string encryptedText = EncryptString(superSecret, key, iv); Console.WriteLine(encryptedText); string decryptedText = DecryptString(encryptedText, key, iv); Console.WriteLine(decryptedText); Console.ReadKey(); } It is going to write a message to the console, encrypt it, write the encrypted text to the console, decrypt it, and write the result of that back to the console. All being well, the first line should be the same as the last, and the middle line should look like gibberish! Of course, it’s not very useful to encrypt and immediately decrypt again. This example illustrates all the parts in one program—in a real appli- cation, decryption would happen in a different place than encryption. The first thing we do is get a suitable key and initialization vector for our cryptographic algorithm. These are the two parts of the secret key that are shared between whoever is encrypting and decrypting our sensitive data. A detailed discussion of cryptography is somewhat beyond the scope of this book, but here are a few key points to get us going. Unenciphered data is known as the plain text, and the encrypted version is known as cipher text. We use those terms even if we’re dealing with nontextual data. The key and the initialization vector (IV) are used by a cryptographic algorithm to encrypt the unenciphered data. A cryptographic algorithm that uses the same key and IV for both encryption and decryption is called a symmetric algorithm (for obvious reasons). Asymmetric algorithms also exist, but we won’t be using them in this example. Needless to say, if an unauthorized individual gets hold of the key and IV, he can happily decrypt any of your cipher text, and you no longer have a communications channel free from prying eyes. It is therefore extremely important that you take care when sharing these secrets with the people who need them, to ensure that no one else can intercept them. (This turns out to be the hardest part—key management and especially human factors turn out to be security weak points far more often than the technological details. This is a book about programming, so we won’t even attempt to solve that problem. We recommend the book Secrets and Lies: Digital Security in a Networked World by Bruce Schneier [John Wiley & Sons] for more information.) 440 | Chapter 11: Files and Streams We’re calling a method called SelectKeyAndIV to get hold of the key and IV. In real life, you’d likely be sharing this information between different processes, usually even on different machines; but for the sake of this demonstration, we’re just creating them on the fly, as you can see in Example 11-52. Example 11-52. Creating a key and IV private static void SelectKeyAndIV(out byte[] key, out byte[] iv) { var algorithm = TripleDES.Create(); algorithm.GenerateIV(); algorithm.GenerateKey(); key = algorithm.Key; iv = algorithm.IV; } TripleDES is an example of a symmetric algorithm, so it derives from a class called SymmetricAlgorithm. All such classes provide a couple of methods called GenerateIV and GenerateKey that create cryptographically strong random byte arrays to use as an initialization vector and a key. See the sidebar below for an explanation of why we need to use a particular kind of random number generator when cryptography is involved. How Random Are Random Numbers? What does “cryptographically strong” mean when we’re talking about random num- bers? Well, it turns out that most random number generators are not all that random. The easiest way to illustrate this is with a little program that seeds the standard .NET Framework random number generator with an arbitrary integer (3), and then displays some random numbers to the console: static void Main(string[] args) { Random random = new Random(3); for (int i = 0; i < 5; ++i) { Console.WriteLine(random.Next()); } Console.ReadKey(); } If you compile and run, you should see this output: 630327709 1498044246 1857544709 426253993 1203643911 No, I’m not Nostradamus. It is just that the “random” algorithm is actually entirely predictable, given a particular seed. Normally that seed comes from Environment.Tick Count, which means that you normally see different behavior each time. Thus, we have the illusion of “randomness.” But this isn’t good enough for encryption purposes; Streams That Aren’t Files | 441 encryption schemes have been broken in the past because attackers were able to guess a computer’s tick count. Then there’s the question of how uniformly distributed those “random” numbers are, or whether the algorithm has a tendency to generate clusters of random numbers. Get- ting a smooth, unpredictable stream of random numbers from an algorithm is a very hard problem, and the smoother you want it the more expensive it gets (in general). Lack of randomness (i.e., predictability) in your random number generator can signif- icantly reduce the strength of a cryptographic algorithm based on its results. The upshot of this is that you shouldn’t use System.Random if you are particularly sen- sitive to the randomness of your random numbers. This isn’t just limited to security applications—you might want to think about your approach if you were building an online casino application, for example. OK, with that done, we can now implement our EncryptString method. This takes the plain text string, the key, and the initialization vector, and returns us an encrypted string. Example 11-53 shows an implementation. Example 11-53. Encrypting a string private static string EncryptString(string plainText, byte[] key, byte[] iv) { // Create a crypto service provider for the TripleDES algorithm var serviceProvider = new TripleDESCryptoServiceProvider(); using (MemoryStream memoryStream = new MemoryStream()) using (var cryptoStream = new CryptoStream( memoryStream, serviceProvider.CreateEncryptor(key, iv), CryptoStreamMode.Write)) using (StreamWriter writer = new StreamWriter(cryptoStream)) { // Write some text to the crypto stream, encrypting it on the way writer.Write(plainText); // Make sure that the writer has flushed to the crypto stream writer.Flush(); // We also need to tell the crypto stream to flush the final block out to // the underlying stream, or we'll // be missing some content... cryptoStream.FlushFinalBlock(); // Now, we want to get back whatever the crypto stream wrote to our memory // stream. return GetCipherText(memoryStream); } } We’re going to write our plain text to a CryptoStream, using the standard Stream Writer adapter. This works just as well over a CryptoStream as any other, but instead of coming out as plain text, it will be enciphered for us. How does that work? 442 | Chapter 11: Files and Streams An Adapting Stream: CryptoStream CryptoStream is quite different from the other streams we’ve met so far. It doesn’t have any underlying storage of its own. Instead, it wraps around another Stream, and then uses an ICryptoTransform either to transform the data written to it from plain text into cipher text before writing it to that output stream (if we put it into CryptoStream Mode.Write), or to transform what it has read from the underlying stream and turning it back into plain text before passing it on to the reader (if we put it into CryptoStream Mode.Read). So, how do we get hold of a suitable ICryptoTransform? We’re making use of a factory class called TripleDESCryptoServiceProvider. This has a method called CreateEncryp tor which will create an instance of an ICryptoTransform that uses the TripleDES algo- rithm to encrypt our plain text, with the specified key and IV. A number of different algorithms are available in the framework, with various strengths and weaknesses. In general, they also have a number of different configuration options, the defaults for which can vary be- tween versions of the .NET Framework and even versions of the oper- ating system on which the framework is deployed. To be successful, you’re going to have to ensure that you match not just the key and the IV, but also the choice of algorithm and all its options. In general, you should carefully set everything up by hand, and avoid relying on the defaults (unlike this example, which, remember, is here to illustrate streams). We provide all of those parameters to its constructor, and then we can use it (almost) like any other stream. In fact, there is a proviso about CryptoStream. Because of the way that most crypto- graphic algorithms work on blocks of plain text, it has to buffer up what is being written (or read) until it has a full block, before encrypting it and writing it to the underlying stream. This means that, when you finish writing to it, you might not have filled up the final block, and it might not have been flushed out to the destination stream. There are two ways of ensuring that this happens: • Dispose the CryptoStream. • Call FlushFinalBlock on the CryptoStream. In many cases, the first solution is the simplest. However, when you call Dispose on the CryptoStream it will also Close the underlying stream, which is not always what you want to do. In this case, we’re going to use the underlying stream some more, so we don’t want to close it just yet. Instead, we call Flush on the StreamWriter to ensure that it has flushed all of its data to the CryptoStream, and then FlushFinalBlock on the Streams That Aren’t Files | 443 CryptoStream itself, to ensure that the encrypted data is all written to the underlying stream. We can use any sort of stream for that underlying stream. We could use a file stream on disk, or one of the isolated storage file streams we saw earlier in this chapter, for example. We could even use one of the network streams we’re going to see in Chap- ter 13. However, for this example we’d like to do everything in memory, and the frame- work has just the class for us: the MemoryStream. In Memory Alone: The MemoryStream MemoryStream is very simple in concept. It is just a stream that uses memory as its backing store. We can do all of the usual things like reading, writing, and seeking. It’s very useful when you’re working with APIs that require you to provide a Stream, and you don’t already have one handy. If we use the default constructor (as in our example), we can read and write to the stream, and it will automatically grow in size as it needs to accommodate the data being written. Other constructors allow us to provide a start size suitable for our purposes (if we know in advance what that might be). We can even provide a block of memory in the form of a byte[] array to use as the underlying storage for the stream. In that case, we are no longer able to resize the stream, and we will get a NotSupportedException if we try to write too much data. You would normally supply your own byte[] array when you already have one and need to pass it to something that wants to read from a stream. We can find out the current size of the underlying block of memory (whether we allo- cated it explicitly, or whether it is being automatically resized) by looking at the stream’s Capacity property. Note that this is not the same as the maximum number of bytes we’ve ever written to the stream. The automatic resizing tends to overallocate to avoid the overhead of constant reallocation when writing. In general, you can determine how many bytes you’ve actually written to by looking at the Position in the stream at the beginning and end of your write operations, or the Length property of the MemoryStream. Having used the CryptoStream to write the cipher text into the stream, we need to turn that into a string we can show on the console. Representing Binary As Text with Base64 Encoding Unfortunately, the cipher text is not actually text at all—it is just a stream of bytes. We can’t use the UTF8Encoding.UTF8.GetString technique we saw in Chapter 10 to turn the bytes into text, because these bytes don’t represent UTF-8 encoded characters. Instead, we need some other sort of text-friendly representation if we’re going to be able to print the encrypted text to the console. We could write each byte out as hex digits. That would be a perfectly reasonable string representation. 444 | Chapter 11: Files and Streams However, that’s not very compact (each byte is taking five characters in the string!): 0x01 0x0F 0x03 0xFA 0xB3 A much more compact textual representation is Base64 encoding. This is a very popular textual encoding of arbitrary data. It’s often used to embed binary in XML, which is a fundamentally text-oriented format. And even better, the framework provides us with a convenient static helper method to convert from a byte[] to a Base64 encoded string: Convert.ToBase64String. If you’re wondering why there’s no Encoding class for Base64 to corre- spond to the Unicode, ASCII, and UTF-8 encodings we saw in Chap- ter 10, it’s because Base64 is a completely different kind of thing. Those other encodings are mechanisms that define binary representations of textual information. Base64 does the opposite—it defines a textual rep- resentation for binary information. Example 11-54 shows how we make use of that in our GetCipherText method. Example 11-54. Converting to Base64 private static string GetCipherText(MemoryStream memoryStream) { byte[] buffer = memoryStream.ToArray(); return System.Convert.ToBase64String(buffer, 0, buffer.Length); } We use a method on MemoryStream called ToArray to get a byte[] array containing all the data written to the stream. Don’t be caught out by the ToBuffer method, which also returns a byte[] array. ToBuffer returns the whole buffer including any “extra” bytes that have been allocated but not yet used. Finally, we call Convert.ToBase64String to get a string representation of the underlying data, passing it the byte[], along with a start offset into that buffer of zero (so that we start with the first byte), and the length. That takes care of encryption. How about decryption? That’s actually a little bit easier. Example 11-55 shows how. Example 11-55. Decryption private static string DecryptString(string cipherText, byte[] key, byte[] iv) { // Create a crypto service provider for the TripleDES algorithm var serviceProvider = new TripleDESCryptoServiceProvider(); // Decode the cipher-text bytes back from the base-64 encoded string Streams That Aren’t Files | 445 byte[] cipherTextBytes = Convert.FromBase64String(cipherText); // Create a memory stream over those bytes using (MemoryStream memoryStream = new MemoryStream(cipherTextBytes)) // And create a cryptographic stream over the memory stream, // using the specified algorithm // (with the provided key and initialization vector) using (var cryptoStream = new CryptoStream( memoryStream, serviceProvider.CreateDecryptor(key, iv), CryptoStreamMode.Read)) // Finally, create a stream reader over the stream, and recover the // original text using (StreamReader reader = new StreamReader(cryptoStream)) { return reader.ReadToEnd(); } } First, we use Convert.FromBase64String to convert our Base64 encoded string back to an array of bytes. We then construct a MemoryStream over that byte[] by passing it to the appropriate constructor. As before, we wrap the MemoryStream with a CryptoStream, this time passing it the ICryptoTransform created by a call to CreateDecryptor on our TripleDESCryptoService Provider, and putting it into CryptoStreamMode.Read. Finally, we construct our old friend the StreamReader over the CryptoStream, and read the content back as a string. So, what’s actually happening here? CryptoStream uses the ICryptoTransform to take care of turning the cipher text in the MemoryStream back into plain text. If you remember, that plain text is actually the set of UTF-8 encoded bytes we originally wrote to the stream with the StreamWriter back in the encryption phase. So, the StreamReader takes those and converts them back into a string for us. You can see that illustrated in Figure 11-14. This is a very powerful example of how we can plug together various components in a kind of pipeline to achieve quite complex processing, from simple, easily understood building blocks that conform to a common pattern, but which have no dependencies on each other’s implementation details. The Stream abstraction is the key to this flexibility. 446 | Chapter 11: Files and Streams Figure 11-14. Encryption and decryption pipeline using streams Summary In this chapter we looked at the classes in the System.IO namespace that relate to files and streams. We saw how we can use static methods on the File, Directory, and Path classes to manage and manipulate files and folders in the filesystem, including creating, deleting, appending, and truncating data, as well as managing their access permissions. We saw how to use StreamReader and StreamWriter to deal with reading and writing text from files, and how we can also read and write binary data using the underlying Stream objects themselves, including the ability to Seek backward and forward in the file. Summary | 447 We then looked at a special type of file stream called isolated storage. This gives us the ability to manage the scope of file access to particular users, machines, applications, or even assemblies. We gain control over quotas (the maximum amount of space any particular store is allowed to use), and get to use local file storage in normally restricted security contexts like that of a Silverlight application, for example. Finally, we looked at some streams that aren’t files, including MemoryStream, which uses memory as its underlying storage mechanism, and CryptoStream, which has no storage of its own, delegating that responsibility to another stream. We showed how these patterns can be used to plug streams together into a processing pipeline. 448 | Chapter 11: Files and Streams CHAPTER 12 XML XML (the eXtensible Markup Language) provides an industry-standard method for encoding structured information. It defines syntactic and structural rules that enable software applications to process XML files even when they don’t understand all of the data. XML specifications are defined and maintained by the World Wide Web Consortium (W3C). The latest version is XML 1.1 (Second Edition). However, XML 1.0 (currently in its fifth edition) is the most popular version, and is supported by all XML parsers. W3C states that: You are encouraged to create or generate XML 1.0 documents if you do not need the new features in XML 1.1; XML Parsers are expected to understand both XML 1.0 and XML 1.1 (see http://www.w3.org/xml/core/#publications/). This chapter will introduce XML 1.0 only, and in fact, will focus on just the most commonly used XML features. We’ll introduce you to the XDocument and XElement classes first, and you’ll learn how to create and manipulate XML documents. Of course, once you have a large document, you’ll want to be able to find substrings, and we’ll show you two different ways to do that, using LINQ. The .NET Framework also allows you to serialize your objects as XML, and deserialize them at their destina- tion. We’ll cover those methods at the end of the chapter. XML Basics (A Quick Review) XML is a markup language, not unlike HTML, except that it is extensible—that is, applications that use XML can (and do) create new kinds of elements and attributes. 449 Elements In XML, a document is a hierarchy of elements. An element is typically defined by a pair of tags, called the start and end tags. In the following example, FirstName is an element: Orlando A start tag contains the element name surrounded by a pair of angle brackets: An end tag is similar, except that the element name is preceded by a forward slash: An element may contain content between its start and end tags. In this example, the element contains text, but content can also contain child elements. For example, this Customer element has three child elements: Orlando Gee orlando0@hotmail.com The top-level element in an XML document is called its root element. Every document has exactly one root element. An element does not have to contain content, but every element (except for the root element) has exactly one parent element. Elements with the same parent element are called sibling elements. In this example, Customers (plural) is the root. The children of the root element, Customers, are the three Customer (singular) elements: Orlando Gee orlando0@hotmail.com Keith Harris keith0@hotmail.com Donna Carreras donna0@hotmail.com Janet Gates janet1@hotmail.com 450 | Chapter 12: XML Lucy Harrington lucy0@hotmail.com Each Customer has one parent (Customers) and three children (FirstName, LastName, and EmailAddress). Each of these, in turn, has one parent (Customer) and zero children. When an element has no content—no child elements and no text—you can optionally use a more compact representation, where you write just a single tag, with a slash just before the closing angle bracket. For example, this: means exactly the same as this: This empty element tag syntax is the only syntax in which an element is represented by just a single tag. Unless you are using this form, it is illegal to omit the closing tag. XHTML XHTML is an enhanced standard of HTML that follows the stricter rules of XML val- idity. The two most important XML rules that make XHTML different from plain HTML follow: • No elements may overlap, though they may nest. So this is legal, because the elements are nested: ... You may not write: ... because in the latter case, element2 overlaps element1 rather than being neatly nes- ted within it. (Ordinary HTML allows this.) • Every element must be closed, which means that for each opened element, you must have a closing tag (or the element tag must be self-closing). So while plain old HTML permits:
XML Basics (A Quick Review) | 451 in XHTML we must either write this:

or use the empty element tag form:
X Stands for eXtensible The key point of XML is to provide an extensible markup language. Here’s an incredibly short pop-history lesson: HTML was derived from the Standard Generalized Markup Language (SGML). HTML has many wonderful attributes (if you’ll pardon the pun), but if you want to add a new element to HTML, you have two choices: apply to the W3C and wait, or strike out on your own and be “nonstandard.” There was a strong need for the ability for two organizations to get together and specify tags that they could use for data exchange. Hey! Presto! XML was born as a more general-purpose markup language that allows users to define their own tags. This is the critical distinction of XML. Creating XML Documents Because XML documents are structured text documents, you can create them using a text editor and process them using string manipulation functions. To paraphrase David Platt, you can also have an appendectomy through your mouth, but it takes longer and hurts more. To make the job easier, .NET implements classes and utilities that provide XML func- tionality. There are several to choose from. There are the streaming XML APIs (which support XmlReader and XmlWriter), which never attempt to hold the whole document in memory—you work one element at a time, and while that enables you to handle very large documents without using much memory, it can be tricky to code for. So there are simpler APIs that let you build an object model that represents an XML document. Even here, you have a choice. One set of XML APIs is based on the XML Document Object Model (DOM), a standard API implemented in many programming systems, not just .NET. However, the DOM is surprisingly cumbersome to work with, so .NET 3.5 introduced a set of APIs that are easier to use from .NET. These are designed to work well with LINQ, and so they’re often referred to as LINQ to XML. These are now the preferred XML API if you don’t need streaming. (Silverlight doesn’t even offer the XML DOM APIs, so LINQ to XML is your only nonstreaming option there.) Despite the name, it’s not strictly necessary to use LINQ when using the LINQ to XML classes—Example 12-1 uses this API to write a list of customers to an XML document. 452 | Chapter 12: XML Example 12-1. Creating an XML document using System; using System.Collections.Generic; using System.Xml.Linq; namespace Programming_CSharp { // Simple customer class public class Customer { public string FirstName { get; set; } public string LastName { get; set; } public string EmailAddress { get; set; } } // Main program public class Tester { static void Main() { List customers = CreateCustomerList(); var customerXml = new XDocument(); var rootElem = new XElement("Customers"); customerXml.Add(rootElem); foreach (Customer customer in customers) { // Create new element representing the customer object. var customerElem = new XElement("Customer"); // Add element representing the FirstName property // to the customer element. var firstNameElem = new XElement("FirstName", customer.FirstName); customerElem.Add(firstNameElem); // Add element representing the LastName property // to the customer element. var lastNameElem = new XElement("LastName", customer.LastName); customerElem.Add(lastNameElem); // Add element representing the EmailAddress property // to the customer element. var emailAddress = new XElement("EmailAddress", customer.EmailAddress); customerElem.Add(emailAddress); // Finally add the customer element to the XML document rootElem.Add(customerElem); } Console.WriteLine(customerXml.ToString()); Console.Read(); } Creating XML Documents | 453 // Create a customer list with sample data private static List CreateCustomerList() { List customers = new List { new Customer { FirstName = "Orlando", LastName = "Gee", EmailAddress = "orlando0@hotmail.com"}, new Customer { FirstName = "Keith", LastName = "Harris", EmailAddress = "keith0@hotmail.com" }, new Customer { FirstName = "Donna", LastName = "Carreras", EmailAddress = "donna0@hotmail.com" }, new Customer { FirstName = "Janet", LastName = "Gates", EmailAddress = "janet1@hotmail.com" }, new Customer { FirstName = "Lucy", LastName = "Harrington", EmailAddress = "lucy0@hotmail.com" } }; return customers; } } } The program will produce this output: Orlando Gee orlando0@hotmail.com Keith Harris keith0@hotmail.com Donna Carreras donna0@hotmail.com Janet Gates janet1@hotmail.com Lucy Harrington lucy0@hotmail.com 454 | Chapter 12: XML As it happens, this example would have needed less code if we had used LINQ, but for this first example, we wanted to keep things simple. We’ll show the LINQ version shortly. In .NET, the System.Xml.Linq namespace contains the LINQ to XML classes we can use to create and process XML documents. The Customer class and the CreateCustomerList function in the main Tester class con- tain straightforward code to give us some data to work with, so we will not go over them. The main attraction in this example is the XML creation in the Main function. First, we create a new XML document object: var customerXml = new XDocument(); Next, we create the root element and add it to the document: var rootElem = new XElement("Customers"); customerXml.Add(rootElem); After these two operations, the customerXml object represents an XML document con- taining an empty element, which might look either like this: or like this: LINQ to XML tends to use the empty element tag form where possible, so if you were to call ToString() on customerXml at this point, it would produce that second version. Of course, you may already have an XML document, and you may want to turn that into an XDocument object. Example 12-2 shows how to load a string into a new XDocument. Example 12-2. Loading XML from a string XDocument doc = XDocument.Parse(""); There’s also a Load method, which has several overloads. You can pass in a URL, in which case it will fetch the XML from there and then parse it. You can also pass in a Stream or a TextReader, the abstract types from the System.IO namespace that represent a stream of bytes (such as a file), or a source of text (such as a file of some known character encoding). XML Elements With the root element in hand, you can add each customer as a child node: foreach (Customer customer in customers) { // Create new element representing the customer object. var customerElem = new XElement("Customer"); Creating XML Documents | 455 In this example, we make each property of the customer object a child element of the customer element: // Add element representing the FirstName property to the Customer element. var firstNameElem = new XElement("FirstName", customer.FirstName); cstomerElem.Add(firstNameElem); This adds the FirstName child element. We’re passing the customer’s first name as the second constructor argument, which will make that the content of the element. The result will look like this: Orlando The other two properties, LastName and EmailAddress, are added to the customer ele- ment in exactly the same way. Here’s an example of the complete customer element: Orlando Gee orlando0@hotmail.com Finally, the newly created customer element is added to the XML document as a child of the root element: // Finally add the customer element to the XML document rootElem.Add(customerElem); } Once all customer elements are created, this example prints the XML document: Console.WriteLine(customerXml.ToString()); When you call ToString() on any of the LINQ to XML objects (whether they represent the whole document, as in this case, or just some fragment of a document such as an XElement), it produces the XML text, and it formats it with indentation, making it easy to read. There are ways to produce more compact representations—if you’re sending the XML across a network to another computer, size may be more important than readability. To see a terser representation, we could do this: Console.WriteLine(customerXml.ToString(SaveOptions.DisableFormatting)); That will print the XML as one long line with no spaces. XML Attributes An XML element may have a set of attributes, which store additional information about the element. An attribute is a key/value pair contained in the start tag of an XML element: If you’re using an empty element tag, the attributes appear in the one and only tag: 456 | Chapter 12: XML The next example demonstrates how you can mix the use of child elements and at- tributes. It creates customer elements with the customer’s name stored in attributes and the email address stored as a child element: orlando0@hotmail.com The only difference between this and Example 12-1 is that we create XAttribute objects for the FirstName and LastName properties instead of XElement objects: // Add an attribute representing the FirstName property // to the customer element. var firstNameAttr = new XAttribute("FirstName", customer.FirstName); customerElem.Add(firstNameAttr); // Add an attribute representing the LastName property // to the customer element. var lastNameAttr = new XAttribute("LastName", customer.LastName); customerElem.Add(lastNameAttr); As with elements, we just add the attribute to the parent element. Example 12-3 shows the complete sample code and output. Example 12-3. Creating an XML document containing elements and attributes using System; using System.Collections.Generic; using System.Xml.Linq; namespace Programming_CSharp { // Simple customer class public class Customer { // Same as in Example 12-1 } // Main program public class Tester { static void Main() { List customers = CreateCustomerList(); var customerXml = new XDocument(); var rootElem = new XElement("Customers"); customerXml.Add(rootElem); foreach (Customer customer in customers) { // Create new element representing the customer object. var customerElem = new XElement("Customer"); // Add an attribute representing the FirstName property // to the customer element. var firstNameAttr = new XAttribute("FirstName", Creating XML Documents | 457 customer.FirstName); customerElem.Add(firstNameAttr); // Add an attribute representing the LastName property // to the customer element. var lastNameAttr = new XAttribute("LastName", customer.LastName); customerElem.Add(lastNameAttr); // Add element representing the EmailAddress property // to the customer element. var emailAddress = new XElement("EmailAddress", customer.EmailAddress); customerElem.Add(emailAddress); // Finally add the customer element to the XML document rootElem.Add(customerElem); } Console.WriteLine(customerXml.ToString()); Console.Read(); } // Create a customer list with sample data private static List CreateCustomerList() { List customers = new List { new Customer { FirstName = "Orlando", LastName = "Gee", EmailAddress = "orlando0@hotmail.com"}, new Customer { FirstName = "Keith", LastName = "Harris", EmailAddress = "keith0@hotmail.com" }, new Customer { FirstName = "Donna", LastName = "Carreras", EmailAddress = "donna0@hotmail.com" }, new Customer { FirstName = "Janet", LastName = "Gates", EmailAddress = "janet1@hotmail.com" }, new Customer { FirstName = "Lucy", LastName = "Harrington", EmailAddress = "lucy0@hotmail.com" } }; return customers; } } } Output: orlando0@hotmail.com 458 | Chapter 12: XML keith0@hotmail.com donna0@hotmail.com janet1@hotmail.com lucy0@hotmail.com While it’s often convenient to be able to create and add elements and attributes one step at a time, these classes offer constructors that allow us to do more work in a single step. If we know exactly what we want to put in an element, this can lead to neater looking code. For example, we can replace the foreach loop with the code in Example 12-4. Example 12-4. Constructing an XElement all at once foreach (Customer customer in customers) { // Create new element representing the customer object. var customerElem = new XElement("Customer", new XAttribute("FirstName", customer.FirstName), new XAttribute("LastName", customer.LastName), new XElement("EmailAddress", customer.EmailAddress) ); // Finally add the customer element to the XML document rootElem.Add(customerElem); } The only difference is that we’re passing all the XAttribute and XElement objects to the containing XElement constructor, rather than passing them to Add one at a time. As well as being more compact, it’s pretty easy to see how this code relates to the structure of the XML element being produced. We can also use this technique in conjunction with LINQ. Putting the LINQ in LINQ to XML We’ve seen several examples that construct an XElement, passing the name as the first argument, and the content as the second. We’ve passed strings, child elements, and attributes, but we can also provide an implementation of IEnumerable. So if we add a using System.Linq; directive to the top of our file, we could use a LINQ query as the second constructor argument as Example 12-5 shows. Creating XML Documents | 459 Example 12-5. Generating XML elements with LINQ var customerXml = new XDocument(new XElement("Customers", from customer in customers select new XElement("Customer", new XAttribute("FirstName", customer.FirstName), new XAttribute("LastName", customer.LastName), new XElement("EmailAddress", customer.EmailAddress) ))); This generates the whole of the XML document in a single statement. So the work that took 25 lines of code in Example 12-1 comes down to just seven. Example 12-6 shows the whole example, with its much simplified Main method. Example 12-6. Building XML with LINQ using System; using System.Collections.Generic; using System.Linq; using System.Xml.Linq; namespace Programming_CSharp { // Simple customer class public class Customer { // Same as in Example 12-1 } // Main program public class Tester { static void Main() { List customers = CreateCustomerList(); var customerXml = new XDocument(new XElement("Customers", from customer in customers select new XElement("Customer", new XAttribute("FirstName", customer.FirstName), new XAttribute("LastName", customer.LastName), new XElement("EmailAddress", customer.EmailAddress) ))); Console.WriteLine(customerXml.ToString()); Console.Read(); } // Create a customer list with sample data private static List CreateCustomerList() { List customers = new List { new Customer { FirstName = "Orlando", LastName = "Gee", 460 | Chapter 12: XML EmailAddress = "orlando0@hotmail.com"}, new Customer { FirstName = "Keith", LastName = "Harris", EmailAddress = "keith0@hotmail.com" }, new Customer { FirstName = "Donna", LastName = "Carreras", EmailAddress = "donna0@hotmail.com" }, new Customer { FirstName = "Janet", LastName = "Gates", EmailAddress = "janet1@hotmail.com" }, new Customer { FirstName = "Lucy", LastName = "Harrington", EmailAddress = "lucy0@hotmail.com" } }; return customers; } } } We’re not really doing anything special here—this LINQ query is just relying on plain old LINQ to Objects—the same techniques we already saw in Chapter 8. But this is only half the story. LINQ to XML is not just about creating XML. It also supports reading XML. Being able to create XML documents to store data to be processed or exchanged is great, but it would not be of much use if you could not find information in them easily. LINQ to XML lets you use the standard LINQ operators to search for information in XML documents. Searching in XML with LINQ We’ll need an example document to search through. Here’s the document from Ex- ample 12-3, reproduced here for convenience: orlando0@hotmail.com keith0@hotmail.com donna0@hotmail.com janet1@hotmail.com lucy0@hotmail.com Searching in XML with LINQ | 461 Example 12-7 lists the code for the example. Example 12-7. Searching an XML document using LINQ using System; using System.Collections.Generic; using System.Linq; using System.Xml.Linq; namespace Programming_CSharp { public class Customer { public string FirstName { get; set; } public string LastName { get; set; } public string EmailAddress { get; set; } } public class Tester { private static XDocument CreateCustomerListXml() { List customers = CreateCustomerList(); var customerXml = new XDocument(new XElement("Customers", from customer in customers select new XElement("Customer", new XAttribute("FirstName", customer.FirstName), new XAttribute("LastName", customer.LastName), new XElement("EmailAddress", customer.EmailAddress) ))); return customerXml; } private static List CreateCustomerList() { List customers = new List { new Customer {FirstName = "Douglas", LastName = "Adams", EmailAddress = "dAdams@foo.com"}, new Customer {FirstName = "Richard", LastName = "Dawkins", EmailAddress = "rDawkins@foo.com"}, new Customer {FirstName = "Kenji", LastName = "Yoshino", EmailAddress = "kYoshino@foo.com"}, new Customer {FirstName = "Ian", LastName = "McEwan", EmailAddress = "iMcEwan@foo.com"}, new Customer {FirstName = "Neal", LastName = "Stephenson", EmailAddress = "nStephenson@foo.com"}, new Customer {FirstName = "Randy", 462 | Chapter 12: XML LastName = "Shilts", EmailAddress = "rShilts@foo.com"}, new Customer {FirstName = "Michelangelo", LastName = "Signorile ", EmailAddress = "mSignorile@foo.com"}, new Customer {FirstName = "Larry", LastName = "Kramer", EmailAddress = "lKramer@foo.com"}, new Customer {FirstName = "Jennifer", LastName = "Baumgardner", EmailAddress = "jBaumgardner@foo.com"} }; return customers; } static void Main() { XDocument customerXml = CreateCustomerListXml(); Console.WriteLine("Search for single element..."); var query = from customer in customerXml.Element("Customers").Elements("Customer") where customer.Attribute("FirstName").Value == "Douglas" select customer; XElement oneCustomer = query.SingleOrDefault(); if (oneCustomer != null) { Console.WriteLine(oneCustomer); } else { Console.WriteLine("Not found"); } Console.WriteLine("\nSearch using descendant axis... "); query = from customer in customerXml.Descendants("Customer") where customer.Attribute("FirstName").Value == "Douglas" select customer; oneCustomer = query.SingleOrDefault(); if (oneCustomer != null) { Console.WriteLine(oneCustomer); } else { Console.WriteLine("Not found"); } Console.WriteLine("\nSearch using element values... "); query = from emailAddress in customerXml.Descendants("EmailAddress") where emailAddress.Value == "dAdams@foo.com" Searching in XML with LINQ | 463 select emailAddress; XElement oneEmail = query.SingleOrDefault(); if (oneEmail != null) { Console.WriteLine(oneEmail); } else { Console.WriteLine("Not found"); } Console.WriteLine("\nSearch using child element values... "); query = from customer in customerXml.Descendants("Customer") where customer.Element("EmailAddress").Value == "dAdams@foo.com" select customer; oneCustomer = query.SingleOrDefault(); if (oneCustomer != null) { Console.WriteLine(oneCustomer); } else { Console.WriteLine("Not found"); } } // end main } // end class } // end namespace Output: Search for single element... dAdams@foo.com Search using descendant axis... dAdams@foo.com Search using element values... dAdams@foo.com Search using child element values... dAdams@foo.com This example refactors Example 12-3 by extracting the creation of the sample customer list XML document into the CreateCustomerListXml() method. You can now simply call this function in the Main() function to create the XML document. 464 | Chapter 12: XML Searching for a Single Node The first search in Example 12-7 is to find a customer whose first name is “Douglas”: var query = from customer in customerXml.Element("Customers").Elements("Customer") where customer.Attribute("FirstName").Value == "Douglas" select customer; XElement oneCustomer = query.SingleOrDefault(); if (oneCustomer != null) { Console.WriteLine(oneCustomer); } else { Console.WriteLine("Not found"); } In general, you will have some ideas about the structure of XML documents you are going to process; otherwise, it will be difficult to find the information you want. Here we know the node we are looking for sits just one level below the root element. So the source of the LINQ query—the part after the in keyword—fetches the root Customers element using the singular Element method, and then asks for all of its chil- dren called Customers by using the plural Elements method: from customer in customerXml.Element("Customers").Elements("Customer") We specify the search conditions with a where clause, as we would do in any LINQ query. In this case, we want to search on the value of the FirstName attribute: where customer.Attribute("FirstName").Value == "Douglas" The select clause is trivial—we just want the query to return all matching elements. Finally, we execute the query using the standard LINQ SingleOrDefault operator, which, as you may recall, returns the one result of the query, unless it failed to match anything, in which case it will return null. (And if there are multiple matches, it throws an exception.) We therefore test the result against null before attempting to use it: if (oneCustomer != null) { Console.WriteLine(oneCustomer); } else { Console.WriteLine("Not found"); } In this example, the method is successful, and the resultant element is displayed. Searching in XML with LINQ | 465 Search Axes In practice, you don’t always know exactly where the information you require will be in the XML document when you write the code. For these cases, LINQ to XML provides the ability to search in different ways—if you are familiar with the XPath query lan- guage* for XML, this is equivalent to the XPath concept of a search axis. This specifies the relationship between the element you’re starting from and the search target nodes. The Element and Elements methods we used earlier only ever search one level—they look in the children of whatever object you call them on. But we can instead use the Descendants method to look not just in the children, but also in their children’s children, and so on. So the source for the next query in Example 12-7 looks for all elements called Customer anywhere in the document. This is more compact, but also less precise. query = from customer in customerXml.Descendants("Customer") Other methods available for querying along different axes include Parent, Ancestors, ElementsAfterSelf, ElementsBeforeSelf, and Attributes. The first two look up the tree and are similar to Elements and Descendants, in that Parent looks up just one level, while Ancestors will search up through the document all the way to the root. ElementsBefor eSelf and ElementsAfterSelf search for elements that have the same parent as the cur- rent item, and which appear either before or after it in the document. Attributes searches in an element’s attributes rather than its child elements. (If you are familiar with XPath, you will know that these correspond to the parent, ancestor, following- sibling, preceding-sibling, and attribute axes.) Where Clauses The first query in Example 12-7 included a where clause that looked for a particular attribute value on an element. You can, of course, use other criteria. The third query looks at the content of the element itself—it uses the Value property to extract the content as text: where emailAddress.Value == "dAdams@foo.com" You can get more ambitious, though—the where clause can dig further into the structure of the XML. The fourth query’s where clause lets through only those elements whose child EmailAddress element has a particular value: where customer.Element("EmailAddress").Value == "dAdams@foo.com" * XPath is supported by both LINQ to XML and the DOM APIs. (Unless you’re using Silverlight, in which case the DOM API is missing entirely, and the XPath support is absent from LINQ to XML.) So if you prefer that, you can use it instead, or you can use a mixture of LINQ and XPath. 466 | Chapter 12: XML XML Serialization So far, our code has constructed the objects representing the Customer XML elements by hand. As XML is becoming popular, especially with the increasingly widespread use of web services, it can be useful to automate this process. If you expect to work with XML elements that always have a particular structure, it can be convenient to serialize objects to or from XML. Working with conventional objects can be a lot easier than using lots of explicit XML code. The .NET Framework provides a built-in serialization mechanism to reduce the coding efforts by application developers. The System.Xml.Serialization namespace defines the classes and utilities that implement methods required for serializing and deserial- izing objects. Example 12-8 illustrates this. Example 12-8. Simple XML serialization and deserialization using System; using System.IO; using System.Xml.Serialization; namespace Programming_CSharp { // Simple customer class public class Customer { public string FirstName { get; set; } public string LastName { get; set; } public string EmailAddress { get; set; } // Overrides the Object.ToString() to provide a // string representation of the object properties. public override string ToString() { return string.Format("{0} {1}\nEmail: {2}", FirstName, LastName, EmailAddress); } } // Main program public class Tester { static void Main() { Customer c1 = new Customer { FirstName = "Orlando", LastName = "Gee", EmailAddress = "orlando0@hotmail.com" }; XmlSerializer serializer = new XmlSerializer(typeof(Customer)); StringWriter writer = new StringWriter(); XML Serialization | 467 serializer.Serialize(writer, c1); string xml = writer.ToString(); Console.WriteLine("Customer in XML:\n{0}\n", xml); Customer c2 = serializer.Deserialize(new StringReader(xml)) as Customer; Console.WriteLine("Customer in Object:\n{0}", c2.ToString()); Console.ReadKey(); } } } Output: Customer in XML: Orlando Gee orlando0@hotmail.com Customer in Object: Orlando Gee Email: orlando0@hotmail.com To serialize an object using .NET XML serialization, you need to create an XmlSerializer object: XmlSerializer serializer = new XmlSerializer(typeof(Customer)); You must pass in the type of the object to be serialized to the XmlSerializer constructor. If you don’t know the object type at design time, you can discover it by calling its GetType() method: XmlSerializer serializer = new XmlSerializer(c1.GetType()); You also need to decide where the serialized XML document should be stored. In this example, you simply send it to a StringWriter: StringWriter writer = new StringWriter(); serializer.Serialize(writer, c1); string xml = writer.ToString(); Console.WriteLine("Customer in XML:\n{0}\n", xml); The resultant XML string is then displayed on the console: Orlando Gee orlando0@hotmail.com 468 | Chapter 12: XML The first line is an XML declaration. This is to let the consumers (human users and software applications) of this document know that this is an XML file, the official ver- sion to which this file conforms, and the encoding format used. This is optional in XML, but this code always produces one. The root element here is the Customer element, with each property represented as a child element. The xmlns:xsi and xmlns:xsd attributes relate to the XML Schema spec- ification. They are optional, and don’t do anything useful in this example, so we will not explain them further. If you are interested, please read the XML specification or other documentation, such as the MSDN Library, for more details. Aside from those optional parts, this XML representation of the Customer object is equivalent to the one created in Example 12-1. However, instead of writing numerous lines of code to deal with the XML specifics, you need only three lines using .NET XML serialization classes. Furthermore, it is just as easy to reconstruct an object from its XML form: Customer c2 = serializer.Deserialize(new StringReader(xml)) as Customer; Console.WriteLine("Customer in Object:\n{0}", c2.ToString()); All it needs is to call the XmlSerializer.Deserialize method. It has several overloaded versions, one of which takes a TextReader instance as an input parameter. Because StringReader is derived from TextReader, you just pass an instance of StringReader to read from the XML string. The Deserialize method returns an object, so it is necessary to cast it to the correct type. Of course, there’s a price to pay. XML serialization is less flexible than working with the XML APIs directly—with serialization you decide exactly what XML elements and attributes you expect to see when you write the code. If you need to be able to adapt dynamically to elements whose names you only learn at runtime, you will need to stick with the XML-aware APIs. Customizing XML Serialization Using Attributes By default, all public read/write properties are serialized as child elements. You can customize your classes by specifying the type of XML node you want for each of your public properties, as shown in Example 12-9. Example 12-9. Customizing XML serialization with attributes using System; using System.IO; using System.Xml.Serialization; namespace Programming_CSharp { // Simple customer class public class Customer XML Serialization | 469 { [XmlAttribute] public string FirstName { get; set; } [XmlIgnore] public string LastName { get; set; } public string EmailAddress { get; set; } // Overrides the Object.ToString() to provide a // string representation of the object properties. public override string ToString() { return string.Format("{0} {1}\nEmail: {2}", FirstName, LastName, EmailAddress); } } // Main program public class Tester { static void Main() { Customer c1 = new Customer { FirstName = "Orlando", LastName = "Gee", EmailAddress = "orlando0@hotmail.com" }; //XmlSerializer serializer = new XmlSerializer(c1.GetType()); XmlSerializer serializer = new XmlSerializer(typeof(Customer)); StringWriter writer = new StringWriter(); serializer.Serialize(writer, c1); string xml = writer.ToString(); Console.WriteLine("Customer in XML:\n{0}\n", xml); Customer c2 = serializer.Deserialize(new StringReader(xml)) as Customer; Console.WriteLine("Customer in Object:\n{0}", c2.ToString()); Console.ReadKey(); } } } Output: Customer in XML: orlando0@hotmail.com 470 | Chapter 12: XML Customer in Object: Orlando Email: orlando0@hotmail.com The only changes in this example are a couple of XML serialization attributes added in the Customer class: [XmlAttribute] public string FirstName { get; set; } The first change is to specify that you want to serialize the FirstName property into an attribute of the Customer element by adding the XmlAttributeAttribute to the property: [XmlIgnore] public string LastName { get; set; } The other change is to tell XML serialization that you in fact do not want the Last Name property to be serialized at all. You do this by adding the XmlIgnoreAttribute to the property. As you can see from the sample output, the Customer object is serialized without LastName, exactly as we asked. However, you have probably noticed that when the object is deserialized, its Last Name property is lost. Because it is not serialized, the XmlSerializer is unable to assign it any value. Therefore, its value is left as the default, which is an empty string. So in practice, you would exclude from serialization only those properties you don’t need or can compute or can retrieve in other ways. Summary In this chapter, we saw how to use the LINQ to XML classes to build objects repre- senting the structure of an XML document, which can then be converted into an XML document, and we saw how the same classes can be used to load XML from a string or file back into memory as objects. These classes support LINQ, both for building new XML documents and for searching for information in existing XML documents. And we also saw how XML serialization can hide some of the details of XML handling behind ordinary C# classes in situations where you know exactly what structure of XML to expect. Summary | 471 CHAPTER 13 Networking Most interesting computer systems are distributed these days—it’s increasingly un- usual for a program to run in isolation on a single machine. So .NET provides various ways to communicate across networks. The array of networking options looks a little bewildering at first: there are 10 namespaces whose names start with System.Net con- taining more than 250 classes, and that’s not even the complete set—there’s an even bigger API for producing and consuming web services. Fortunately, it’s simpler than this makes it seem—despite the large API surface area most of the options fall into three categories. There’s WCF—the Windows Commu- nication Foundation, a framework for building and using web services. There are lower- level APIs for working directly with web protocols. Or you can use sockets if you need very low-level control. We’ll start by discussing how to choose the most appropriate style of communication for your application, and then we’ll look at these three options in more detail. Choosing a Networking Technology The first step in choosing the right networking API is to decide on the nature of the communication your application requires. There are many different styles of distrib- uted applications. Perhaps you are building a public-facing web service designed to be used by a diverse range of clients. Conversely, you might be writing client code that uses someone else’s web service. Or maybe you’re writing software that runs at both ends of the connection, but even then there are some important questions. Are you connecting a user interface to a service in a tightly controlled environment where you can easily deploy updates to the client and the server at the same time? Or perhaps you have very little control over client updates—maybe you’re selling software to thou- sands of customers whose own computers will connect back to your service, and you expect to have many different versions of the client program out there at any one time. Maybe it doesn’t even make sense to talk about clients and servers—you might be creating a peer-to-peer system. Or maybe your system is much simpler than that, and has just two computers talking to each other. 473 The variations are endless, so no single approach can work well for all systems. The next few sections will look at some common scenarios, and discuss the pros and cons of the various networking options .NET offers. Even within a specific scenario there will often be more than one way to make things work. There are no hard-and-fast rules, because each project has different requirements. So this section won’t tell you what to do—it’ll just describe the issues you’ll need to consider. Ultimately, only you can decide on the right solution for your system. We’ll start with a very common web-based scenario. Web Application with Client-Side Code Web user interfaces have been getting smarter lately. A few years ago, most of a web application’s logic would live on the server, with client-side code in the web browser typically doing little more than making buttons light up and menus fly out in response to the mouse. But now, we expect more from our web user interfaces. Whether you use AJAX (Asynchronous JavaScript and XML), or a RIA (Rich Internet Application) technology such as Silverlight or Flash, web applications often communicate constantly with the web server, and not just when navigating between pages. If you’re writing the server-side parts of this sort of application in C#, you will typically use ASP.NET to provide a web user interface. But what should you use for program- matic communication—the messages that flow between the web UI and the server once a page is already loaded? WCF is a flexible choice here, because as Figure 13-1 illustrates, you can make a single set of remote services accessible to many common browser-based user interface tech- nologies. A WCF service can be configured to communicate in several different ways simultaneously. You could use JSON (JavaScript Object Notation), which is widely used in AJAX-based user interfaces because it’s is a convenient message format for JavaScript client code. Or you could use XML-based web services. Note that using WCF on the server does not require WCF on the client. These services could be used by clients written in other technologies such as Java, as long as they also support the same web service standards as WCF. Looking specifically at the case where your web application uses C# code on the client side, this would mean using either Silverlight or WPF. (You can put WPF in a web page by writing an XBAP—a Xaml Browser Application. This will work only if the end user has WPF installed.) If you’re using C# on both the client and the server, the most straightforward choice is likely to be WCF on both ends. What if your server isn’t running .NET, but you still want to use .NET on the web client? There are some restrictions on WCF in this scenario. Silverlight’s version of WCF is much more limited than the version in the full .NET Framework—whereas the full version can be configured to use all manner of different protocols, Silverlight’s WCF supports just two options. There’s the so-called basic profile for web services, in which only a narrow set of features is available, and there’s a binary protocol unique to WCF, 474 | Chapter 13: Networking which offers the same narrow set of features but makes slightly more efficient use of network bandwidth than the XML-based basic profile. So if you want a Silverlight client to use WCF to communicate with a non-.NET web service, as Figure 13-2 illustrates, this will work only if your service supports the basic profile. Figure 13-1. Web application clients and a WCF service Figure 13-2. Silverlight client and non-.NET web service More surprisingly, similar restrictions exist with a WPF XBAP. Even though XBAPs use the full version of the .NET Framework, certain features of WCF are disabled for se- curity purposes—client code in web browsers shouldn’t have complete freedom to connect to anywhere on the Internet, because that would make life too easy for hackers. Choosing a Networking Technology | 475 So WCF offers only a very limited version of its services to .NET applications running inside web browsers, meaning that XBAPs have similar WCF limitations to Silverlight. If you’re writing a Silverlight client and you want to talk to a service that does not conform to the web services basic profile, that’s not necessarily a showstopper. It just rules out WCF—you will need to use the lower-level web-based APIs instead, or even the socket APIs, depending on the service. Note that while WCF is usually a good default choice on the server side for web ap- plications with client-side code, there are a few cases where you might not want to use it. ASP.NET provides its own mechanism for supporting AJAX clients, and while it’s considerably less flexible than WCF, you might not need the flexibility. The simplicity of using just one framework on the server instead of two might end up looking like a better option. There’s a subtler reason why WCF might not always be the best fit: the style of com- munication. If you use WCF in a web application, the communication it supports will tend to involve the following steps: 1. Some code in the client (browser script, C# code, or Flash ActionScript) decides to send a message to the server. 2. The server receives the message and runs some code that does whatever it needs to do to process the message. 3. Once the code has finished, the server