深入理解 C# 第3版 英文版


MANNING IN DEPTH Jon Skeet FOREWORD BY ERIC LIPPERT THIRD EDITION Praise for the Second Edition A masterpiece about C#. —Kirill Osenkov, Microsoft C# Team If you are looking to master C# then this book is a must-read. —Tyson S. Maxwell Sr. Software Engineer, Raytheon We're betting that this will be the best C# 4.0 book out there. —Nikander Bruggeman and Margriet Bruggeman .NET consultants, Lois & Clark IT Services A useful and engaging insight into the evolution of C# 4. —Joe Albahari Author of LINQPad and C# 4.0 in a Nutshell One of the best C# books I have ever read. —Aleksey Nudelman CEO, C# Computing, LLC This book should be required reading for all professional C# developers. —Stuart Caborn Senior Developer, BNP Paribas A highly focused, master-level resource on language updates across all major C# releases. This book is a must-have for the expert developer wanting to stay current with new features of the C# language. —Sean Reilly, Programmer/Analyst Point2 Technologies Why read the basics over and over again? Jon focuses on the chewy, new stuff! —Keith Hill, Software Architect Agilent Technologies Everything you didn’t realize you needed to know about C#. —Jared Parsons Senior Software Development Engineer Microsoft Praise for the First Edition Simply put, C# in Depth is perhaps the best computer book I’ve read. —Craig Pelkie, Author, System iNetwork I have been developing in C# from the very beginning and this book had some nice sur- prises even for me. I was especially impressed with the excellent coverage of delegates, anonymous methods, covariance and contravariance. Even if you are a seasoned devel- oper, C# in Depth will teach you something new about the C# language... This book truly has depth that no other C# language book can touch. —Adam J. Wolf Southeast Valley .NET User Group I enjoyed reading the whole book; it is well-written—the samples are easy to understand. I actually found it very easy to engage into the whole lambda expressions topic and really liked the chapter about lambda expressions. —Jose Rolando Guay Paz Web Developer, CSW Solutions This book wraps up the author’s great knowledge of the inner workings of C# and hands it over to readers in a well-written, concise, usable book. —Jim Holmes Author of Windows Developer Power Tools Every term is used appropriately and in the right context, every example is spot-on and contains the least amount of code that shows the full extent of the feature...this is a rare treat. —Franck Jeannin, Amazon UK reviewer If you have developed using C# for several years now, and would like to know the inter- nals, this book is absolutely right for you. —Golo Roden Author, Speaker, and Trainer for .NET and related technologies The best C# book I’ve ever read. —Chris Mullins, C# MVP C# in Depth THIRD EDITION JON SKEET MANNING SHELTER ISLAND For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 261 Shelter Island, NY 11964. ©2014 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. Development editor Jeff Bleiel 20 Baldwin Road Copyeditor: Andy Carroll PO Box 261 Proofreader: Katie Tennant Shelter Island, NY 11964 Typesetter: Dottie Marsico Cover designer: Marija Tudor ISBN 9781617291340 Printed in the United States of America 1 2 3 4 5 6 7 8 9 10 – MAL – 18 17 16 15 14 13 To my boys, Tom, Robin, and William vii brief contents PART 1 PREPARING FOR THE JOURNEY.......................................1 1 ■ The changing face of C# development 3 2 ■ Core foundations: building on C# 1 29 PART 2 C# 2: SOLVING THE ISSUES OF C# 1 .............................57 3 ■ Parameterized typing with generics 59 4 ■ Saying nothing with nullable types 105 5 ■ Fast-tracked delegates 133 6 ■ Implementing iterators the easy way 159 7 ■ Concluding C# 2: the final features 182 PART 3 C# 3: REVOLUTIONIZING DATA ACCESS ......................205 8 ■ Cutting fluff with a smart compiler 207 9 ■ Lambda expressions and expression trees 232 10 ■ Extension methods 262 11 ■ Query expressions and LINQ to Objects 285 12 ■ LINQ beyond collections 328 BRIEF CONTENTSviii PART 4 C# 4: PLAYING NICELY WITH OTHERS .........................369 13 ■ Minor changes to simplify code 371 14 ■ Dynamic binding in a static language 409 PART 5 C# 5: ASYNCHRONY MADE SIMPLE ..............................461 15 ■ Asynchrony with async/await 463 16 ■ C# 5 bonus features and closing thoughts 519 ix contents foreword xix preface xxi acknowledgments xxii about this book xxiv about the author xxix about the cover illustration xxx PART 1 PREPARING FOR THE JOURNEY .............................1 1 The changing face of C# development 3 1.1 Starting with a simple data type 4 The Product type in C# 1 5 ■ Strongly typed collections in C# 2 6 Automatically implemented properties in C# 3 7 ■ Named arguments in C# 4 8 1.2 Sorting and filtering 9 Sorting products by name 9 ■ Querying collections 12 1.3 Handling an absence of data 14 Representing an unknown price 14 ■ Optional parameters and default values 16 1.4 Introducing LINQ 16 Query expressions and in-process queries 17 ■ Querying XML 18 ■ LINQ to SQL 19 CONTENTSx 1.5 COM and dynamic typing 20 Simplifying COM interoperability 20 ■ Interoperating with a dynamic language 21 1.6 Writing asynchronous code without the heartache 22 1.7 Dissecting the .NET platform 23 C#, the language 24 ■ Runtime 24 ■ Framework libraries 24 1.8 Making your code super awesome 25 Presenting full programs as snippets 25 ■ Didactic code isn’t production code 26 ■ Your new best friend: the language specification 27 1.9 Summary 28 2 Core foundations: building on C# 1 29 2.1 Delegates 30 A recipe for simple delegates 30 ■ Combining and removing delegates 35 ■ A brief diversion into events 36 ■ Summary of delegates 37 2.2 Type system characteristics 38 C#’s place in the world of type systems 38 ■ When is C# 1’s type system not rich enough? 41 ■ Summary of type system characteristics 44 2.3 Value types and reference types 44 Values and references in the real world 45 ■ Value and reference type fundamentals 46 ■ Dispelling myths 47 ■ Boxing and unboxing 49 ■ Summary of value types and reference types 50 2.4 Beyond C# 1: new features on a solid base 51 Features related to delegates 51 ■ Features related to the type system 53 ■ Features related to value types 55 2.5 Summary 56 PART 2 C# 2: SOLVING THE ISSUES OF C# 1 ...................57 3 Parameterized typing with generics 59 3.1 Why generics are necessary 60 3.2 Simple generics for everyday use 62 Learning by example: a generic dictionary 62 ■ Generic types and type parameters 64 ■ Generic methods and reading generic declarations 67 CONTENTS xi 3.3 Beyond the basics 70 Type constraints 71 ■ Type inference for type arguments of generic methods 76 ■ Implementing generics 77 3.4 Advanced generics 83 Static fields and static constructors 84 ■ How the JIT compiler handles generics 85 ■ Generic iteration 87 ■ Reflection and generics 90 3.5 Limitations of generics in C# and other languages 94 Lack of generic variance 94 ■ Lack of operator constraints or a “numeric” constraint 99 ■ Lack of generic properties, indexers, and other member types 101 ■ Comparison with C++ templates 101 ■ Comparison with Java generics 103 3.6 Summary 104 4 Saying nothing with nullable types 105 4.1 What do you do when you just don’t have a value? 106 Why value type variables can’t be null 106 Patterns for representing null values in C# 1 107 4.2 System.Nullable and System.Nullable 109 Introducing Nullable 109 ■ Boxing Nullable and unboxing 112 ■ Equality of Nullable instances 113 Support from the nongeneric Nullable class 114 4.3 C# 2’s syntactic sugar for nullable types 114 The ? modifier 115 ■ Assigning and comparing with null 116 Nullable conversions and operators 118 ■ Nullable logic 121 Using the as operator with nullable types 123 ■ The null coalescing operator 123 4.4 Novel uses of nullable types 126 Trying an operation without using output parameters 127 Painless comparisons with the null coalescing operator 129 4.5 Summary 131 5 Fast-tracked delegates 133 5.1 Saying goodbye to awkward delegate syntax 134 5.2 Method group conversions 136 5.3 Covariance and contravariance 137 Contravariance for delegate parameters 138 ■ Covariance of delegate return types 139 ■ A small risk of incompatibility 141 CONTENTSxii 5.4 Inline delegate actions with anonymous methods 142 Starting simply: acting on a parameter 142 ■ Returning values from anonymous methods 145 ■ Ignoring delegate parameters 146 5.5 Capturing variables in anonymous methods 148 Defining closures and different types of variables 148 Examining the behavior of captured variables 149 ■ What’s the point of captured variables? 151 ■ The extended lifetime of captured variables 152 ■ Local variable instantiations 153 Mixtures of shared and distinct variables 155 ■ Captured variable guidelines and summary 156 5.6 Summary 158 6 Implementing iterators the easy way 159 6.1 C# 1: The pain of handwritten iterators 160 6.2 C# 2: Simple iterators with yield statements 163 Introducing iterator blocks and yield return 163 ■ Visualizing an iterator’s workflow 165 ■ Advanced iterator execution flow 167 ■ Quirks in the implementation 170 6.3 Real-life iterator examples 172 Iterating over the dates in a timetable 172 ■ Iterating over lines in a file 173 ■ Filtering items lazily using an iterator block and a predicate 176 6.4 Pseudo-synchronous code with the Concurrency and Coordination Runtime 178 6.5 Summary 180 7 Concluding C# 2: the final features 182 7.1 Partial types 183 Creating a type with multiple files 184 ■ Uses of partial types 186 ■ Partial methods—C# 3 only! 188 7.2 Static classes 190 7.3 Separate getter/setter property access 192 7.4 Namespace aliases 193 Qualifying namespace aliases 194 ■ The global namespace alias 195 ■ Extern aliases 196 CONTENTS xiii 7.5 Pragma directives 197 Warning pragmas 197 ■ Checksum pragmas 198 7.6 Fixed-size buffers in unsafe code 199 7.7 Exposing internal members to selected assemblies 201 Friend assemblies in the simple case 201 ■ Why use InternalsVisibleTo? 202 ■ InternalsVisibleTo and signed assemblies 203 7.8 Summary 204 PART 3 C# 3: REVOLUTIONIZING DATA ACCESS.............205 8 Cutting fluff with a smart compiler 207 8.1 Automatically implemented properties 208 8.2 Implicit typing of local variables 211 Using var to declare a local variable 211 ■ Restrictions on implicit typing 213 ■ Pros and cons of implicit typing 214 Recommendations 215 8.3 Simplified initialization 216 Defining some sample types 216 ■ Setting simple properties 217 Setting properties on embedded objects 219 ■ Collection initializers 220 ■ Uses of initialization features 223 8.4 Implicitly typed arrays 224 8.5 Anonymous types 225 First encounters of the anonymous kind 225 ■ Members of anonymous types 227 ■ Projection initializers 228 ■ What’s the point? 229 8.6 Summary 231 9 Lambda expressions and expression trees 232 9.1 Lambda expressions as delegates 234 Preliminaries: Introducing the Func<…> delegate types 234 First transformation to a lambda expression 235 ■ Using a single expression as the body 236 ■ Implicitly typed parameter lists 236 Shortcut for a single parameter 237 9.2 Simple examples using List and events 238 Filtering, sorting, and actions on lists 238 ■ Logging in an event handler 240 CONTENTSxiv 9.3 Expression trees 241 Building expression trees programmatically 242 ■ Compiling expression trees into delegates 243 ■ Converting C# lambda expressions to expression trees 244 ■ Expression trees at the heart of LINQ 248 ■ Expression trees beyond LINQ 249 9.4 Changes to type inference and overload resolution 251 Reasons for change: streamlining generic method calls 252 Inferred return types of anonymous functions 253 ■ Two-phase type inference 254 ■ Picking the right overloaded method 258 Wrapping up type inference and overload resolution 260 9.5 Summary 260 10 Extension methods 262 10.1 Life before extension methods 263 10.2 Extension method syntax 265 Declaring extension methods 265 ■ Calling extension methods 267 ■ Extension method discovery 268 ■ Calling a method on a null reference 269 10.3 Extension methods in .NET 3.5 271 First steps with Enumerable 271 ■ Filtering with Where and chaining method calls together 273 ■ Interlude: haven’t we seen the Where method before? 275 ■ Projections using the Select method and anonymous types 276 ■ Sorting using the OrderBy method 277 ■ Business examples involving chaining 278 10.4 Usage ideas and guidelines 280 “Extending the world” and making interfaces richer 280 ■ Fluent interfaces 280 ■ Using extension methods sensibly 282 10.5 Summary 284 11 Query expressions and LINQ to Objects 285 11.1 Introducing LINQ 286 Fundamental concepts in LINQ 286 ■ Defining the sample data model 291 11.2 Simple beginnings: selecting elements 292 Starting with a source and ending with a selection 293 ■ Compiler translations as the basis of query expressions 293 ■ Range variables and nontrivial projections 296 ■ Cast, OfType, and explicitly typed range variables 298 CONTENTS xv 11.3 Filtering and ordering a sequence 300 Filtering using a where clause 300 ■ Degenerate query expressions 301 ■ Ordering using an orderby clause 302 11.4 Let clauses and transparent identifiers 304 Introducing an intermediate computation with let 305 Transparent identifiers 306 11.5 Joins 307 Inner joins using join clauses 307 ■ Group joins with join...into clauses 311 ■ Cross joins and flattening sequences using multiple from clauses 314 11.6 Groupings and continuations 318 Grouping with the group...by clause 318 ■ Query continuations 321 11.7 Choosing between query expressions and dot notation 324 Operations that require dot notation 324 ■ Query expressions where dot notation may be simpler 325 ■ Where query expressions shine 325 11.8 Summary 326 12 LINQ beyond collections 328 12.1 Querying a database with LINQ to SQL 329 Getting started: the database and model 330 ■ Initial queries 332 ■ Queries involving joins 334 12.2 Translations using IQueryable and IQueryProvider 336 Introducing IQueryable and related interfaces 337 ■ Faking it: interface implementations to log calls 338 ■ Gluing expressions together: the Queryable extension methods 341 ■ The fake query provider in action 342 ■ Wrapping up IQueryable 344 12.3 LINQ-friendly APIs and LINQ to XML 344 Core types in LINQ to XML 345 ■ Declarative construction 347 Queries on single nodes 349 ■ Flattened query operators 351 Working in harmony with LINQ 352 12.4 Replacing LINQ to Objects with Parallel LINQ 353 Plotting the Mandelbrot set with a single thread 353 ■ Introducing ParallelEnumerable, ParallelQuery, and AsParallel 354 Tweaking parallel queries 356 CONTENTSxvi 12.5 Inverting the query model with LINQ to Rx 357 IObservable and IObserver 358 ■ Starting simply (again) 360 ■ Querying observables 360 ■ What’s the point? 363 12.6 Extending LINQ to Objects 364 Design and implementation guidelines 364 ■ Sample extension: selecting a random element 365 12.7 Summary 367 PART 4 C# 4: PLAYING NICELY WITH OTHERS ...............369 13 Minor changes to simplify code 371 13.1 Optional parameters and named arguments 372 Optional parameters 372 ■ Named arguments 378 ■ Putting the two together 382 13.2 Improvements for COM interoperability 387 The horrors of automating Word before C# 4 387 ■ The revenge of optional parameters and named arguments 388 ■ When is a ref parameter not a ref parameter? 389 ■ Calling named indexers 390 ■ Linking primary interop assemblies 391 13.3 Generic variance for interfaces and delegates 394 Types of variance: covariance and contravariance 394 ■ Using variance in interfaces 396 ■ Using variance in delegates 399 Complex situations 399 ■ Restrictions and notes 401 13.4 Teeny tiny changes to locking and field-like events 405 Robust locking 405 ■ Changes to field-like events 406 13.5 Summary 407 14 Dynamic binding in a static language 409 14.1 What? When? Why? How? 411 What is dynamic typing? 411 ■ When is dynamic typing useful, and why? 412 ■ How does C# 4 provide dynamic typing? 413 14.2 The five-minute guide to dynamic 414 14.3 Examples of dynamic typing 416 COM in general, and Microsoft Office in particular 417 ■ Dynamic languages such as IronPython 419 Dynamic typing in purely managed code 423 CONTENTS xvii 14.4 Looking behind the scenes 429 Introducing the Dynamic Language Runtime 429 ■ DLR core concepts 431 ■ How the C# compiler handles dynamic 434 The C# compiler gets even smarter 438 ■ Restrictions on dynamic code 441 14.5 Implementing dynamic behavior 444 Using ExpandoObject 444 ■ Using DynamicObject 448 Implementing IDynamicMetaObjectProvider 455 14.6 Summary 459 PART 5 C# 5: ASYNCHRONY MADE SIMPLE ....................461 15 Asynchrony with async/await 463 15.1 Introducing asynchronous functions 465 First encounters of the asynchronous kind 465 ■ Breaking down the first example 467 15.2 Thinking about asynchrony 468 Fundamentals of asynchronous execution 468 ■ Modeling asynchronous methods 471 15.3 Syntax and semantics 472 Declaring an async method 472 ■ Return types from async methods 473 ■ The awaitable pattern 474 ■ The flow of await expressions 477 ■ Returning from an async method 481 Exceptions 482 15.4 Asynchronous anonymous functions 490 15.5 Implementation details: compiler transformation 492 Overview of the generated code 493 ■ Structure of the skeleton method 495 ■ Structure of the state machine 497 ■ One entry point to rule them all 498 ■ Control around await expressions 500 ■ Keeping track of a stack 501 ■ Finding out more 503 15.6 Using async/await effectively 503 The task-based asynchronous pattern 504 ■ Composing async operations 507 ■ Unit testing asynchronous code 511 The awaitable pattern redux 515 ■ Asynchronous operations in WinRT 516 15.7 Summary 517 CONTENTSxviii 16 C# 5 bonus features and closing thoughts 519 16.1 Changes to captured variables in foreach loops 520 16.2 Caller information attributes 520 Basic behavior 521 ■ Logging 522 ■ Implementing INotifyPropertyChanged 523 ■ Using caller information attributes without .NET 4.5 524 16.3 Closing thoughts 525 appendix A LINQ standard query operators 527 appendix B Generic collections in .NET 540 appendix C Version summaries 554 index 563 xix foreword There are two kinds of pianists. There are some pianists who play, not because they enjoy it, but because their par- ents force them to take lessons. Then there are those who play the piano because it pleases them to create music. They don’t need to be forced; on the contrary, they sometimes don’t know when to stop. Of the latter kind, there are some who play the piano as a hobby. Then there are those who play for a living. That requires a higher level of dedication, skill, and talent. They may have some degree of freedom about what genre of music they play and the stylistic choices they make in playing it, but fundamentally those choices are driven by the needs of the employer or the tastes of the audience. Of the latter kind, there are some who do it primarily for the money. Then there are those professionals who would want to play the piano in public even if they weren’t being paid. They enjoy using their skills and talents to make music for others. That they can have fun and get paid for it is so much the better. Of the latter kind, there are some who are self-taught, who play by ear, who might have great talent and ability, but can’t communicate that intuitive understanding to others except through the music itself. Then there are those who have formal training in both theory and practice. They can explain what techniques the composer used to achieve the intended emotional effect, and use that knowledge to shape their inter- pretation of the piece. Of the latter kind, there are some who have never looked inside their pianos. Then there are those who are fascinated by the clever escapements that lift the damper felts a fraction of a second before the hammers strike the strings. They own key levelers FOREWORDxx and capstan wrenches. They take delight and pride in being able to understand the mechanisms of an instrument that has 5–10,000 moving parts. Of the latter kind, there are some who are content to master their craft and exer- cise their talents for the pleasure and profit it brings. Then there are those who are not just artists, theorists, and technicians; somehow they find the time to pass that knowledge on to others as mentors. I have no idea if Jon Skeet is a pianist or musician of any sort. But from my email conversations with him as one of the C# team’s Most Valuable Professionals over the years, from reading his blog, and from reading every word of each of his books at least three times, it has become clear to me that Jon is that latter kind of software developer: enthusiastic, knowledgeable, talented, curious, analytical—and a teacher of others. C# is a highly pragmatic and rapidly evolving language. Through the addition of query comprehensions, richer type inference, a compact syntax for anonymous func- tions, and so on, I hope that we have enabled a whole new style of programming while still staying true to the statically typed, component-oriented approach that has made C# a success. Many of these new stylistic elements have the paradoxical quality of feeling very old (lambda expressions go back to the foundations of computer science in the first half of the twentieth century) and yet at the same time feeling new and unfamiliar to developers used to a more modern object-oriented approach. Jon gets all that. This book is ideal for professional developers who have a need to understand the what and how of the latest revision to C#. But it is also for those devel- opers whose understanding is enriched by exploring the why of the language’s design principles. Being able to take advantage of all that new power requires new ways of thinking about data, functions, and the relationship between them. It’s not unlike trying to play jazz after years of classical training—or vice versa. Either way, I’m looking forward to finding out what sorts of functional compositions the next generation of C# pro- grammers come up with. Happy composing, and thanks for choosing the key of C# to do it in. ERIC LIPPERT C# ANALYSIS ARCHITECT COVERITY xxi preface Oh boy. When writing this preface, I started off with the preface to the second edition, which began by saying how long it felt since writing the preface to the first edition. The second edition is now a distant memory, and the first edition seems like a whole different life. I’m not sure whether that says more about the pace of modern life or my memory, but it’s a sobering thought either way. The development landscape has changed enormously since the first edition, and even since the second. This has been driven by many factors, with the rise of mobile devices probably being the most obvious. But many challenges have remained the same. It’s still hard to write properly internationalized applications. It’s still hard to handle errors gracefully in all situations. It’s still fairly hard to write correct multi- threaded applications, although this task has been made significantly simpler by both language and library improvements over the years. Most importantly in the context of this preface, I believe developers still need to know the language they’re using at a level where they’re confident in how it will behave. They may not know the fine details of every API call they’re using, or even some of the obscure corner cases of the language that they don’t happen to use,1 but the core of the language should feel like a solid friend that the developer can rely on to behave predictably. In addition to the letter of the language you’re developing in, I believe there’s great benefit in understanding its spirit. While you may occasionally find you have a fight on your hands however hard you try, if you attempt to make your code work in the way the language designers intended, your experience will be a much more pleasant one. 1 I have a confession to make: I know very little about unsafe code and pointers in C#. I’ve simply never needed to find out about them. xxii acknowledgments You might expect that putting together a third edition—and one where the main change consists of two new chapters—would be straightforward. Indeed, writing the “green field” content of chapters 15 and 16 was the easy part. But there’s a lot more to it than that—tweaking little bits of language throughout the rest of the book, check- ing for any aspects which were fine a few years ago but don’t quite make sense now, and generally making sure the whole book is up to the high standards I expect readers to hold it to. Fortunately, I have been lucky enough to have a great set of people sup- porting me and keeping the book on the straight and narrow. Most importantly, my family have been as wonderful as ever. My wife Holly is a chil- dren’s author herself, so our kids are used to us having to lock ourselves away for a while to meet editorial deadlines, but they’ve remained cheerfully encouraging throughout. Holly herself takes all of this in stride, and I’m grateful that she’s never reminded me just how many books she’s started from scratch and completed in the time I’ve been working on this third edition. The formal peer reviewers are listed later on, but I’d like to add a note of personal thanks to all those who ordered early access copies of this third edition, finding typos and suggesting changes...also constantly asking when the book was coming out. The very fact that I had readers who were eager to get their hands on the finished book was a huge source of encouragement. I always get on well with the team at Manning, and it’s been a pleasure to work with some familiar friends from the first edition as well as newcomers. Mike Stephens and Jeff Bleiel have guided the whole process smoothly, as we decided what to change from the earlier editions and what to keep. They’ve generally put the whole thing into ACKNOWLEDGMENTS xxiii the right shape. Andy Carroll and Katie Tennant provided expert copyediting and proofreading, respectively, never once expressing irritation with my Englishness, pick- iness, or general bewilderment. The production team has worked its magic in the background, as ever, but I’m grateful to them nonetheless: Dottie Marsico, Janet Vail, Marija Tudor, and Mary Piergies. Finally, I’d like to thank the publisher, Marjan Bace, for allowing me a third edition and exploring some interesting future options. Peer review is immensely important, not only for getting the technical details of the book right, but also the balance and tone. Sometimes the comments we received have merely shaped the overall book; in other cases I’ve made very specific changes in response. Either way, all feedback has been welcome. So thanks to the follow- ing reviewers for making the book better for all of us: Andy Kirsch, Bas Pennings, Bret Colloff, Charles M. Gross, Dror Helper, Dustin Laine, Ivan Todorovic´, Jon Parish, Sebastian Martín Aguilar, Tiaan Geldenhuys, and Timo Bredenoort. I’d particularly like to thank Stephen Toub and Stephen Cleary, whose early reviews of chapter 15 were invaluable. Asynchrony is a particularly tricky topic to write about clearly but accurately, and their expert advice made a very significant difference to the chapter. Without the C# team, this book would have no cause to exist, of course. Their ded- ication to the language in design, implementation and testing is exemplary, and I look forward to seeing what they come up with next. Since the second edition was pub- lished, Eric Lippert has left the C# team for a new fabulous adventure, but I’m enor- mously grateful that he was still able to act as the tech reviewer for this third edition. I also thank him for the foreword that he originally wrote to the first edition and that is included again this time. I refer to Eric’s thoughts on various matters throughout the book, and if you aren’t already reading his blog (http://ericlippert.com), you really should be. xxiv about this book This is a book about C# from version 2 onward—it’s as simple as that. I barely cover C# 1 and only cover the .NET Framework libraries and Common Language Runtime (CLR) when they’re related to the language. This is a deliberate decision, and the result is a book quite different from most of the C# and .NET books I’ve seen. By assuming a reasonable amount of knowledge of C# 1, I avoid spending hun- dreds of pages covering material that I think most people already understand. This gives me room to expand on the details of later versions of C#, which is what I hope you’re reading the book for. When I wrote the first edition of this book, even C# 2 was relatively unknown to some readers. By now, almost all C# developers have some expe- rience with the features introduced in C# 2, but I’ve still kept that material in this edi- tion, as it’s so fundamental to what comes later. Who should read this book? This book is squarely aimed at developers who already know some C#. For absolute maximum value, you’d know C# 1 well but know very little about later versions. There aren’t many readers in that sweet spot any more, but I believe there are still lots of developers who can benefit from digging deeper into C# 2 and 3, even if they’ve already been using them for a while...and many developers haven’t yet used C# 4 or 5 to any extent. If you don’t know any C# at all, this probably isn’t the book for you. You could struggle through, looking up aspects you’re not familiar with, but it wouldn’t be a very efficient way of learning. You’d be better off starting with a different book, and then gradually adding C# in Depth to the mix. There’s a wide variety of books that cover C# ABOUT THIS BOOK xxv from scratch, in many different styles. The C# in a Nutshell series (O’Reilly) has always been good in this respect, and Essential C# 5.0 (Addison-Wesley Professional) is also a good introduction. I’m not going to claim that reading this book will make you a fabulous coder. There’s so much more to software engineering than knowing the syntax of the lan- guage you happen to be using. I give some words of guidance, but ultimately there’s a lot more gut instinct in development than most of us would like to admit. What I will claim is that if you read and understand this book, you should feel comfortable with C# and free to follow your instincts without too much apprehension. It’s not about being able to write code that no one else will understand because it uses unknown cor- ners of the language; it’s about being confident that you know the options available to you, and know which path the C# idioms are encouraging you to follow. Roadmap The book’s structure is simple. There are five parts and three appendixes. The first part serves as an introduction, including a refresher on topics in C# 1 that are impor- tant for understanding later versions of the language, and that are often misunder- stood. The second part covers the new features introduced in C# 2, the third part covers C# 3, and so on. There are occasions when organizing the material this way means we'll come back to a topic a couple of times—in particular, delegates are improved in C# 2 and then again in C# 3—but there is method in my madness. I anticipate that a number of read- ers will be using different versions for different projects; for example, you may be using C# 4 at work, but experimenting with C# 5 at home. That means it’s useful to clarify what is in which version. It also provides a feeling of context and evolution—it shows how the language has developed over time. Chapter 1 sets the scene by taking a simple piece of C# 1 code and evolving it, see- ing how later versions allow the source to become more readable and powerful. We'll look at the historical context in which C# has grown, and the technical context in which it operates as part of a complete platform; C# as a language builds on frame- work libraries and a powerful runtime to turn abstraction into reality. Chapter 2 looks back at C# 1, and at three specific aspects: delegates, the type sys- tem characteristics, and the differences between value types and reference types. These topics are often understood “just well enough” by C# 1 developers, but as C# has evolved and developed them significantly, a solid grounding is required in order to make the most of the new features. Chapter 3 tackles the biggest feature of C# 2, and potentially the hardest to grasp: generics. Methods and types can be written generically, with type parameters standing in for real types that are specified in the calling code. Initially it’s as confusing as this description makes it sound, but once you understand generics, you’ll wonder how you survived without them. ABOUT THIS BOOKxxvi If you’ve ever wanted to represent a null integer, chapter 4 is for you. It introduces nullable types: a feature, built on generics, that takes advantage of support in the lan- guage, runtime, and framework. Chapter 5 shows the improvements to delegates in C# 2. Until now, you may have only used delegates for handling events such as button clicks. C# 2 makes it easier to create delegates, and library support makes them more useful for situations other than events. In chapter 6 we'll examine iterators, and the easy way to implement them in C# 2. Few developers use iterator blocks, but as LINQ to Objects is built on iterators, they’ll become more and more important. The lazy nature of their execution is also a key part of LINQ. Chapter 7 shows a number of smaller features introduced in C# 2, each making life a little more pleasant. The language designers have smoothed over a few rough places in C# 1, allowing more flexible interaction with code generators, better support for utility classes, more granular access to properties, and more. Chapter 8 once again looks at a few relatively simple features—but this time in C# 3. Almost all the new syntax is geared toward the common goal of LINQ, but the build- ing blocks are also useful in their own right. With anonymous types, automatically implemented properties, implicitly typed local variables, and greatly enhanced initial- ization support, C# 3 gives a far richer language with which your code can express its behavior. Chapter 9 looks at the first major topic of C# 3—lambda expressions. Not content with the reasonably concise syntax discussed in chapter 5, the language designers have made delegates even easier to create than in C# 2. Lambdas are capable of more— they can be converted into expression trees, a powerful way of representing code as data. In chapter 10 we’ll examine extension methods, which provide a way of fooling the compiler into believing that methods declared in one type actually belong to another. At first glance this appears to be a readability nightmare, but with careful consider- ation it can be an extremely powerful feature—and one that’s vital to LINQ. Chapter 11 combines the previous three chapters in the form of query expres- sions, a concise but powerful way of querying data. Initially we’ll concentrate on LINQ to Objects, but you’ll see how the query expression pattern is applied in a way that allows other data providers to plug in seamlessly. Chapter 12 is a quick tour of various different uses of LINQ. First we’ll look at the benefits of query expressions combined with expression trees—how LINQ to SQL is able to convert what appears to be normal C# into SQL statements. We’ll then move on to see how libraries can be designed to mesh well with LINQ, taking LINQ to XML as an example. Parallel LINQ and Reactive Extensions show two alternative approaches to in-process querying, and the chapter closes with a discussion of how you can extend LINQ to Objects with your own LINQ operators. ABOUT THIS BOOK xxvii Coverage of C# 4 begins in chapter 13, where we’ll look at named arguments and optional parameters, COM interop improvements, and generic variance. In some ways these are very separate features, but named arguments and optional parameters con- tribute to COM interop as well as the more specific abilities that are only available when working with COM objects. Chapter 14 describes the single biggest feature in C# 4: dynamic typing. The ability to bind members dynamically at execution time instead of statically at compile time is a huge departure for C#, but it’s applied selectively—only code that involves a dynamic value will be executed dynamically. Chapter 15 is all about asynchrony. C# 5 only contains one major feature—the abil- ity to write asynchronous functions. This single feature is simultaneously brain- bustingly complicated to understand thoroughly and awe-inspiringly elegant to use. At long last, we can write asynchronous code that doesn’t read like spaghetti. We’ll wind down in chapter 16 with the remaining features of C# 5 (both of which are tiny) and some thoughts about the future. The appendixes are all reference material. In appendix A, I cover the LINQ stan- dard query operators, with some examples. Appendix B looks at the core generic col- lection classes and interfaces. Appendix C provides a brief look at the different versions of .NET, including the different flavors such as the Compact Framework and Silverlight. Terminology, typography, and downloads Most of the terminology of the book is explained as it goes along, but there are a few definitions that are worth highlighting here. I use C# 1, C# 2, C# 3, C# 4, and C# 5 in a reasonably obvious manner—but you may see other books and websites referring to C# 1.0, C# 2.0, C# 3.0, C# 4.0, and C# 5.0. The extra “.0” seems redundant to me, which is why I’ve omitted it—I hope the meaning is clear. I’ve appropriated a pair of terms from a C# book by Mark Michaelis. To avoid the confusion between runtime being an execution environment (as in “the Common Lan- guage Runtime”) and a point in time (as in “overriding occurs at runtime”), Mark uses execution time for the latter concept, usually in comparison with compile time. This seems to me to be a thoroughly sensible idea, and one that I hope catches on in the wider community. I’m doing my bit by following his example in this book. I frequently refer to “the language specification” or just “the specification”—unless I indicate otherwise, this means the C# language specification. However, multiple ver- sions of the specification are available, partly due to different versions of the language itself and partly due to the standardization process. Any section numbers provided are from the C# 5.0 language specification from Microsoft. This book contains numerous pieces of code, which appear in a fixed-width font like this; output from the listings appears in the same way. Code annotations accompany some listings, and at other times particular sections of the code are shown in bold to highlight a change, improvement, or addition. Almost all of the code ABOUT THIS BOOKxxviii appears in snippet form, allowing it to stay compact but still runnable—within the right environment. That environment is Snippy, a custom tool that is introduced in section 1.8. Snippy is available for download, along with all of the code from the book (in the form of snippets, full Visual Studio solutions, or more often both) from the book’s website at csharpindepth.com, as well as from the publisher's website at man- ning.com/CSharpinDepthThirdEdition. Author Online and the C# in Depth website Purchase of C# in Depth, Third Edition includes free access to a private web forum run by Manning Publications where you can make comments about the book, ask techni- cal questions, and receive help from the author and other users. To access the forum and subscribe to it, point your web browser to www.manning.com/CSharpinDepth- ThirdEdition. This page provides information on how to get on the forum once you are registered, what kind of help is available, and the rules of conduct on the forum. The Author Online forum and the archives of previous discussions will be accessi- ble from the publisher’s website as long as the book is in print. In addition to Manning’s own website, I have set up a companion website for the book at csharpindepth.com, containing information that didn’t quite fit into the book, downloadable source code for all the listings in the book, and links to other resources. xxix about the author I’m not a typical C# developer, I think it’s fair to say. For the last five years, almost all of my time working with C# has been for fun—effectively as a somewhat obsessive hobby. At work, I’ve been writing server-side Java in Google London, and I can safely claim that few things help you to appreciate new language features more than having to code in a language that doesn’t have them, but is similar enough to remind you of their absence. I’ve tried to keep in touch with what other developers find hard about C# by keep- ing a careful eye on Stack Overflow, posting oddities to my blog, and occasionally talk- ing about C# and related topics just about anywhere that will provide people to listen to me. Additionally, I’m actively developing an open source .NET date and time API called Noda Time (see http://nodatime.org). In short, C# is still coursing through my veins as strongly as ever. For all these oddities—and despite my ever-surprising micro-celebrity status due to Stack Overflow—I’m a very ordinary developer in many other ways. I write plenty of code that makes me grimace when I come back to it. My unit tests don’t always come first...and sometimes they don’t even exist. I make off-by-one errors every so often. The type inference section of the C# specification still confuses me, and there are some uses of Java wildcards that make me want to have a little lie-down. I’m a deeply flawed programmer. That’s the way it should be. For the next few hundred pages, I’ll try to pretend oth- erwise: I’ll espouse best practices as if I always followed them myself, and frown on dirty shortcuts as if I’d never dream of taking them. Don’t believe a word of it. The truth of the matter is, I’m probably just like you. I happen to know a bit more about how C# works, that’s all...and even that state of affairs will only last until you’ve fin- ished the book. xxx about the cover illustration The caption for the illustration on the cover of C# in Depth, Third Edition is “Musician.” The illustration is taken from a collection of costumes of the Ottoman Empire pub- lished on January 1, 1802, by William Miller of Old Bond Street, London. The title page is missing from the collection and we have been unable to track it down to date. The book’s table of contents identifies the figures in both English and French, and each illustration bears the names of two artists who worked on it, both of whom would no doubt be surprised to find their art gracing the front cover of a computer program- ming book...two hundred years later. The collection was purchased by a Manning editor at an antiquarian flea market in the “Garage” on West 26th Street in Manhattan. The seller was an American based in Ankara, Turkey, and the transaction took place just as he was packing up his stand for the day. The Manning editor didn't have on his person the substantial amount of cash that was required for the purchase and a credit card and check were both politely turned down. With the seller flying back to Ankara that evening, the situation was get- ting hopeless. What was the solution? It turned out to be nothing more than an old- fashioned verbal agreement sealed with a handshake. The seller simply proposed that the money be transferred to him by wire and the editor walked out with the bank information on a piece of paper and the portfolio of images under his arm. Needless to say, we transferred the funds the next day, and we remain grateful and impressed by this unknown person’s trust in one of us. It recalls something that might have hap- pened a long time ago. We at Manning celebrate the inventiveness, the initiative, and, yes, the fun of the computer business with book covers based on the rich diversity of regional life of two centuries ago‚ brought back to life by the pictures from this collection. Part 1 Preparing for the journey Every reader will come to this book with a different set of expectations and a different level of experience. Are you an expert looking to fill some holes, how- ever small, in your present knowledge? Perhaps you consider yourself an average developer, with a bit of experience in using generics and lambda expressions, but a desire to better understand how they work. Maybe you’re reasonably confi- dent with C# 2 and 3 but have no experience with C# 4 or 5. As an author, I can’t make every reader the same—and I wouldn’t want to, even if I could. But I hope that all readers have two things in common: the desire for a deeper relationship with C# as a language, and at least a basic knowl- edge of C# 1. If you can bring those elements to the party, I’ll provide the rest. The potentially huge range of skill levels is the main reason why this part of the book exists. You may already know what to expect from later versions of C#— or it could all be brand new to you. You could have a rock-solid understanding of C# 1, or you might be rusty on some of the details—some of which will become increasingly important as you learn about the later versions. By the end of part 1, I won’t have leveled the playing field entirely, but you should be able to approach the rest of the book with confidence and an idea of what’s coming later. In the first two chapters, we’ll look both forward and back. One of the key themes of the book is evolution. Before introducing any feature into the lan- guage, the C# design team carefully considers that feature in the context of what’s already present and the general goals for the future. This brings a feeling of consistency to the language even in the midst of change. To understand how and why the language is evolving, you need to see where it’s come from and where it’s going. Chapter 1 presents a bird’s-eye view of the rest of the book, taking a brief look at some of the biggest features of C# beyond version 1. I’ll show a progression of code from C# 1 onward, applying new features one by one until the code is almost unrecog- nizable from its humble beginnings. We’ll also look at some of the terminology I’ll use in the rest of the book, as well as the format for the sample code. Chapter 2 is heavily focused on C# 1. If you’re an expert in C# 1, you can skip this chapter, but it does tackle some of the areas of C# 1 that tend to be misunderstood. Rather than try to explain the whole of the language, the chapter concentrates on fea- tures that are fundamental to the later versions of C#. From this solid base, you can move on and look at C# 2 in part 2 of the book. 3 The changing face of C# development Do you know what I really like about dynamic languages such as Python, Ruby, and Groovy? They suck away fluff from your code, leaving just the essence of it—the bits that really do something. Tedious formality gives way to features such as generators, lambda expressions, and list comprehensions. The interesting thing is that few of the features that tend to give dynamic lan- guages their lightweight feel have anything to do with being dynamic. Some do, of course—duck typing and some of the magic used in Active Record, for example— but statically typed languages don’t have to be clumsy and heavyweight. Enter C#. In some ways, C# 1 could have been seen as a nicer version of the Java language, circa 2001. The similarities were all too clear, but C# had a few extras: properties as a first-class feature in the language, delegates and events, foreach This chapter covers  An evolving example  The composition of .NET  Using the code in this book  The C# language specification 4 CHAPTER 1 The changing face of C# development loops, using statements, explicit method overriding, operator overloading, and cus- tom value types, to name a few. Obviously, language preference is a personal issue, but C# 1 definitely felt like a step up from Java when I first started using it. Since then, things have only gotten better. Each new version of C# has added sig- nificant features to reduce developer angst, but always in a carefully considered way, and with little backward incompatibility. Even before C# 4 gained the ability to use dynamic typing where it’s genuinely useful, many features traditionally associated with dynamic and functional languages had made it into C#, leading to code that’s easier to write and maintain. Similarly, while the features around asynchrony in C# 5 aren’t exactly the same as those in F#, it feels to me like there’s a definite influence. In this book, I’ll take you through those changes one by one, in enough detail to make you feel comfortable with some of the miracles the C# compiler is now prepared to perform on your behalf. All that comes later, though—in this chapter I’ll whiz through as many features as I can, barely taking a breath. I’ll define what I mean when I talk about C# as a language compared with .NET as a platform, and I’ll offer a few important notes about the sample code for the rest of the book. Then we can dive into the details. We won’t be looking at all the changes made to C# in this single chapter, but you’ll see generics, properties with different access modifiers, nullable types, anonymous methods, automatically implemented properties, enhanced collection initializers, enhanced object initializers, lambda expressions, extension methods, implicit typing, LINQ query expressions, named arguments, optional parameters, simpler COM interop, dynamic typing, and asynchronous functions. These will carry us from C# 1 all the way up to the latest release, C# 5. Obviously that’s a lot to get through, so let’s get started. 1.1 Starting with a simple data type In this chapter I’ll let the C# compiler do amazing things without telling you how and barely mentioning the what or the why. This is the only time that I won’t explain how things work or try to go one step at a time. Quite the opposite, in fact—the plan is to impress rather than educate. If you read this entire section without getting at least a little excited about what C# can do, maybe this book isn’t for you. With any luck, though, you’ll be eager to get to the details of how these magic tricks work, and that’s what the rest of the book is for. The example I’ll use is contrived—it’s designed to pack as many new features into as short a piece of code as possible. It’s also clichéd, but at least that makes it familiar. Yes, it’s a product/name/price example, the e-commerce alternative to “hello, world.” We’ll look at how various tasks can be achieved, and how, as we move forward in ver- sions of C#, you can accomplish them more simply and elegantly than before. You won’t see any of the benefits of C# 5 until right at the end, but don’t worry—that doesn’t make it any less important. 5Starting with a simple data type 1.1.1 The Product type in C# 1 We’ll start off with a type representing a product, and then manipulate it. You won’t see anything particularly impressive yet—just the encapsulation of a couple of proper- ties. To make life simpler for demonstration purposes, this is also where we’ll create a list of predefined products. Listing 1.1 shows the type as it might be written in C# 1. We’ll then move on to see how the code might be rewritten for each later version. This is the pattern we’ll follow for each of the other pieces of code. Given that I’m writing this in 2013, it’s likely that you’re already familiar with code that uses some of the features I’ll introduce, but it’s worth looking back so you can see how far the language has come. using System.Collections; public class Product { string name; public string Name { get { return name; }} decimal price; public decimal Price { get { return price; }} public Product(string name, decimal price) { this.name = name; this.price = price; } public static ArrayList GetSampleProducts() { ArrayList list = new ArrayList(); list.Add(new Product("West Side Story", 9.99m)); list.Add(new Product("Assassins", 14.99m)); list.Add(new Product("Frogs", 13.99m)); list.Add(new Product("Sweeney Todd", 10.99m)); return list; } public override string ToString() { return string.Format("{0}: {1}", name, price); } } Nothing in listing 1.1 should be hard to understand—it’s just C# 1 code, after all. There are three limitations that it demonstrates, though:  An ArrayList has no compile-time information about what’s in it. You could accidentally add a string to the list created in GetSampleProducts, and the com- piler wouldn’t bat an eyelid.  You’ve provided public getter properties, which means that if you wanted matching setters, they’d have to be public, too. Listing 1.1 The Product type (C# 1) 6 CHAPTER 1 The changing face of C# development  There’s a lot of fluff involved in creating the properties and variables—code that complicates the simple task of encapsulating a string and a decimal. Let’s see what C# 2 can do to improve matters. 1.1.2 Strongly typed collections in C# 2 Our first set of changes (shown in the following listing) tackles the first two items listed previously, including the most important change in C# 2: generics. The parts that are new are in bold. public class Product { string name; public string Name { get { return name; } private set { name = value; } } decimal price; public decimal Price { get { return price; } private set { price = value; } } public Product(string name, decimal price) { Name = name; Price = price; } public static List GetSampleProducts() { List list = new List(); list.Add(new Product("West Side Story", 9.99m)); list.Add(new Product("Assassins", 14.99m)); list.Add(new Product("Frogs", 13.99m)); list.Add(new Product("Sweeney Todd", 10.99m)); return list; } public override string ToString() { return string.Format("{0}: {1}", name, price); } } You now have properties with private setters (which you use in the constructor), and it doesn’t take a genius to guess that List is telling the compiler that the list contains products. Attempting to add a different type to the list would result in a com- piler error, and you also don’t need to cast the results when you fetch them from the list. Listing 1.2 Strongly typed collections and private setters (C# 2) 7Starting with a simple data type The changes in C# 2 leave only one of the original three difficulties unanswered, and C# 3 helps out there. 1.1.3 Automatically implemented properties in C# 3 We’re starting off with some fairly tame features from C# 3. The automatically imple- mented properties and simplified initialization shown in the following listing are rela- tively trivial compared with lambda expressions and the like, but they can make code a lot simpler. using System.Collections.Generic; class Product { public string Name { get; private set; } public decimal Price { get; private set; } public Product(string name, decimal price) { Name = name; Price = price; } Product() {} public static List GetSampleProducts() { return new List { new Product { Name="West Side Story", Price = 9.99m }, new Product { Name="Assassins", Price=14.99m }, new Product { Name="Frogs", Price=13.99m }, new Product { Name="Sweeney Todd", Price=10.99m} }; } public override string ToString() { return string.Format("{0}: {1}", Name, Price); } } Now the properties don’t have any code (or visible variables!) associated with them, and you’re building the hardcoded list in a very different way. With no name and price variables to access, you’re forced to use the properties everywhere in the class, improv- ing consistency. You now have a private parameterless constructor for the sake of the new property-based initialization. (This constructor is called for each item before the properties are set.) In this example, you could’ve removed the public constructor completely, but then no outside code could’ve created other product instances. Listing 1.3 Automatically implemented properties and simpler initialization (C# 3) 8 CHAPTER 1 The changing face of C# development 1.1.4 Named arguments in C# 4 For C# 4, we’ll go back to the original code when it comes to the properties and con- structor, so that it’s fully immutable again. A type with only private setters can’t be pub- licly mutated, but it can be clearer if it’s not privately mutable either.1 There’s no shortcut for read-only properties, unfortunately, but C# 4 lets you specify argument names for the constructor call, as shown in the following listing, which gives you the clarity of C# 3 initializers without the mutability. using System.Collections.Generic; public class Product { readonly string name; public string Name { get { return name; }} readonly decimal price; public decimal Price { get { return price; }} public Product(string name, decimal price) { this.name = name; this.price = price; } public static List GetSampleProducts() { return new List { new Product( name: "West Side Story", price: 9.99m), new Product( name: "Assassins", price: 14.99m), new Product( name: "Frogs", price: 13.99m), new Product( name: "Sweeney Todd", price: 10.99m) }; } public override string ToString() { return string.Format("{0}: {1}", name, price); } } The benefits of specifying the argument names explicitly are relatively minimal in this particular example, but when a method or constructor has several parameters, it can make the meaning of the code much clearer—particularly if they’re of the same type, or if you’re passing in null for some arguments. You can choose when to use this feature, of course, only specifying the names for arguments when it makes the code easier to understand. Figure 1.1 summarizes how the Product type has evolved so far. I’ll include a simi- lar diagram after each task, so you can see the pattern of how the evolution of C# 1 The C# 1 code could’ve been immutable too—I only left it mutable to simplify the changes for C# 2 and 3. Listing 1.4 Named arguments for clear initialization code (C# 4) 9Sorting and filtering improves the code. You’ll notice that C# 5 is missing from all of the block diagrams; that’s because the main feature of C# 5 (asynchronous functions) is aimed at an area that really hasn’t evolved much in terms of language support. We’ll take a peek at it before too long, though. So far, the changes are relatively minimal. In fact, the addition of generics (the List syntax) is probably the most important part of C# 2, but you’ve only seen part of its usefulness so far. There’s nothing to get the heart racing yet, but we’ve only just started. Our next task is to print out the list of products in alphabetical order. 1.2 Sorting and filtering In this section, we won’t change the Product type at all—instead, we’ll take the sample products and sort them by name, and then find the expensive ones. Neither of these tasks is exactly difficult, but you’ll see how much simpler they become over time. 1.2.1 Sorting products by name The easiest way to display a list in a particular order is to sort the list and then run through it, displaying items. In .NET 1.1, this involved using ArrayList.Sort, and optionally providing an IComparer implementation to specify a particular compari- son. You could make the Product type implement IComparable, but that would only allow you to define one sort order, and it’s not a stretch to imagine that you might want to sort by price at some stage, as well as by name. The following listing implements IComparer, and then sorts the list and displays it. class ProductNameComparer : IComparer { public int Compare(object x, object y) { Product first = (Product)x; Product second = (Product)y; return first.Name.CompareTo(second.Name); Listing 1.5 Sorting an ArrayList using IComparer (C# 1) C# 1 Read-only properties Weakly typed collections C# 4 Named arguments for clearer constructor and method calls C# 3 Automatically implemented properties Enhanced collection and object initialization C# 2 Private property setters Strongly typed collections Figure 1.1 Evolution of the Product type, showing greater encapsulation, stronger typing, and ease of initialization over time 10 CHAPTER 1 The changing face of C# development } } ... ArrayList products = Product.GetSampleProducts(); products.Sort(new ProductNameComparer()); foreach (Product product in products) { Console.WriteLine (product); } The first thing to spot in listing 1.5 is that you had to introduce an extra type to help with the sorting. That’s not a disaster, but it’s a lot of code if you only want to sort by name in one place. Next, look at the casts in the Compare method. Casts are a way of telling the compiler that you know more information than it does, and that usually means there’s a chance you’re wrong. If the ArrayList you returned from Get- SampleProducts did contain a string, that’s where the code would go bang—where the comparison tries to cast the string to a Product. You also have a cast in the code that displays the sorted list. It’s not obvious, because the compiler puts it in automatically, but the foreach loop implicitly casts each element of the list to Product. Again, that cast could fail at execution time, and once more generics come to the rescue in C# 2. The following listing shows the previ- ous code with the use of generics as the only change. class ProductNameComparer : IComparer { public int Compare(Product x, Product y) { return x.Name.CompareTo(y.Name); } } ... List products = Product.GetSampleProducts(); products.Sort(new ProductNameComparer()); foreach (Product product in products) { Console.WriteLine(product); } The code for the comparer in listing 1.6 is simpler because you’re given products to start with. No casting is necessary. Similarly, the invisible cast in the foreach loop is effectively gone now. The compiler still has to consider the conversion from the source type of the sequence to the target type of the variable, but it knows that in this case both types are Product, so it doesn’t need to emit any code for the conversion. That’s an improvement, but it’d be nice if you could sort the products by simply specifying the comparison to make, without needing to implement an interface to do so. The following listing shows how to do precisely this, telling the Sort method how to compare two products using a delegate. Listing 1.6 Sorting a List using IComparer (C# 2) 11Sorting and filtering List products = Product.GetSampleProducts(); products.Sort(delegate(Product x, Product y) { return x.Name.CompareTo(y.Name); } ); foreach (Product product in products) { Console.WriteLine(product); } Behold the lack of the ProductNameComparer type. The statement in bold font creates a delegate instance, which you provide to the Sort method in order to perform the comparisons. You’ll learn more about this feature (anonymous methods) in chapter 5. You’ve now fixed all the problems identified in the C# 1 version. That doesn’t mean that C# 3 can’t do better, though. First, you’ll replace the anonymous method with an even more compact way of creating a delegate instance, as shown in the fol- lowing listing. List products = Product.GetSampleProducts(); products.Sort((x, y) => x.Name.CompareTo(y.Name)); foreach (Product product in products) { Console.WriteLine(product); } You’ve gained even more strange syntax (a lambda expression), which still creates a Comparison delegate, just as listing 1.7 did, but this time with less fuss. You didn’t have to use the delegate keyword to introduce it, or even specify the types of the parameters. There’s more, though: with C# 3, you can easily print out the names in order with- out modifying the original list of products. The next listing shows this using the OrderBy method. List products = Product.GetSampleProducts(); foreach (Product product in products.OrderBy(p => p.Name) ) { Console.WriteLine (product); } In this listing, you appear to be calling an OrderBy method on the list, but if you look in MSDN, you’ll see that it doesn’t even exist in List. You’re able to call it due to the presence of an extension method, which you’ll see in more detail in chapter 10. You’re not actually sorting the list “in place” anymore, just retrieving the contents Listing 1.7 Sorting a List using Comparison (C# 2) Listing 1.8 Sorting using Comparison from a lambda expression (C# 3) Listing 1.9 Ordering a List using an extension method (C# 3) 12 CHAPTER 1 The changing face of C# development of the list in a particular order. Sometimes you’ll need to change the actual list; some- times an ordering without any other side effects is better. The important point is that this code is much more compact and readable (once you understand the syntax, of course). We wanted the list ordered by name, and that’s exactly what the code says. It doesn’t say to sort by comparing the name of one prod- uct with the name of another, like the C# 2 code did, or to sort by using an instance of another type that knows how to compare one product with another. It just says to order by name. This simplicity of expression is one of the key benefits of C# 3. When the individual pieces of data querying and manipulation are so simple, larger transfor- mations can remain compact and readable in one piece of code. That, in turn, encourages a more data-centric way of looking at the world. You’ve seen more of the power of C# 2 and 3 in this section, with a lot of (as yet) unexplained syntax, but even without understanding the details you can see the prog- ress toward clearer, simpler code. Figure 1.2 shows that evolution. That’s it for sorting.2 Let’s do a different form of data manipulation now—querying. 1.2.2 Querying collections Your next task is to find all the elements of the list that match a certain criterion—in particular, those with a price greater than $10. The following listing shows how, in C# 1, you need to loop around, testing each element and printing it out when appropriate. ArrayList products = Product.GetSampleProducts(); foreach (Product product in products) { if (product.Price > 10m) { Console.WriteLine(product); } } This code is not difficult to understand. But it’s worth bearing in mind how inter- twined the three tasks are—looping with foreach, testing the criterion with if, and 2 C# 4 does provide one feature that can be relevant when sorting, called generic variance, but giving an example here would require too much explanation. You can find the details near the end of chapter 13. Listing 1.10 Looping, testing, printing out (C# 1) C# 2 Strongly typed comparator Delegate comparisons Anonymous methods C# 3 Lambda expressions Extension methods Option of leaving list unsorted C# 1 Weakly typed comparator No delegate sorting option Figure 1.2 Features involved in making sorting easier in C# 2 and 3 13Sorting and filtering then displaying the product with Console.WriteLine. The dependency is obvious because of the nesting. The following listing demonstrates how C# 2 lets you flatten things out a bit. List products = Product.GetSampleProducts(); Predicate test = delegate(Product p) { return p.Price > 10m; }; List matches = products.FindAll(test); Action print = Console.WriteLine; matches.ForEach(print); The test variable is initialized using the anonymous method feature you saw in the previous section. The print variable initialization uses another new C# 2 feature called method group conversions that makes it easier to create delegates from existing methods. I’m not going to claim that this code is simpler than the C# 1 code, but it is a lot more powerful.3 In particular, the technique of separating the two concerns like this makes it very easy to change the condition you’re testing for and the action you take on each of the matches independently. The delegate variables involved (test and print) could be passed into a method, and that same method could end up testing radically different conditions and taking radically different actions. Of course, you could put all the test- ing and printing into one statement, as shown in the following listing. List products = Product.GetSampleProducts(); products.FindAll(delegate(Product p) { return p.Price > 10;}) .ForEach(Console.WriteLine); In some ways, this version is better, but the delegate(Product p) is getting in the way, as are the braces. They’re adding noise to the code, which hurts readability. I still prefer the C# 1 version in cases where I only ever want to use the same test and per- form the same action. (It may sound obvious, but it’s worth remembering that there’s nothing stopping you from using the C# 1 code with a later compiler version. You wouldn’t use a bulldozer to plant tulip bulbs, which is the kind of overkill used in the last listing.) The next listing shows how C# 3 improves matters dramatically by removing a lot of the fluff surrounding the actual logic of the delegate. Listing 1.11 Separating testing from printing (C# 2) 3 In some ways, this is cheating. You could’ve defined appropriate delegates in C# 1 and called them within the loop. The FindAll and ForEach methods in .NET 2.0 just encourage you to consider separation of concerns. Listing 1.12 Separating testing from printing redux (C# 2) 14 CHAPTER 1 The changing face of C# development List products = Product.GetSampleProducts(); foreach (Product product in products.Where(p => p.Price > 10)) { Console.WriteLine(product); } The combination of the lambda expression putting the test in just the right place and a well-named method means you can almost read the code out loud and understand it without thinking. You still have the flexibility of C# 2—the argument to Where could come from a variable, and you could use an Action instead of the hard- coded Console.WriteLine call if you wanted to. This task has emphasized what you already knew from sorting—anonymous meth- ods make writing a delegate simple, and lambda expressions are even more concise. In both cases, that brevity means that you can include the query or sort operation inside the first part of the foreach loop without losing clarity. Figure 1.3 summarizes the changes we’ve just looked at. C# 4 doesn’t offer any- thing to simplify this task any further. Now that you’ve displayed the filtered list, let’s consider a change to your initial assumptions about the data. What happens if you don’t always know the price of a product? How can you cope with that within the Product class? 1.3 Handling an absence of data We’ll look at two different forms of missing data. First we’ll deal with the scenario where you genuinely don’t have the information, and then see how you can actively remove information from method calls, using default values. 1.3.1 Representing an unknown price I won’t present much code this time, but I’m sure it’ll be a familiar problem to you, especially if you’ve done a lot of work with databases. Imagine your list of products contains not just products on sale right now, but ones that aren’t available yet. In some cases, you may not know the price. If decimal were a reference type, you could just use null to represent the unknown price, but since it’s a value type, you can’t. How would you represent this in C# 1? Listing 1.13 Testing with a lambda expression (C# 3) C# 2 Separate condition from action invoked. Anonymous methods make delegates simple. C# 3 Lambda expressions make the condition even easier to read. C# 1 Strong coupling between condition and action. Both are hardcoded. Figure 1.3 Anonymous methods and lambda expressions in C# 2 and 3 aid separation of concerns and readability. 15Handling an absence of data There are three common alternatives:  Create a reference type wrapper around decimal.  Maintain a separate Boolean flag indicating whether the price is known.  Use a “magic value” (decimal.MinValue, for example) to represent the unknown price. I hope you’ll agree that none of these holds much appeal. Time for a little magic: you can solve the problem by adding a single character in the variable and property decla- rations. .NET 2.0 makes matters a lot simpler by introducing the Nullable struc- ture, and C# 2 provides some additional syntactic sugar that lets you change the property declaration to this block of code: decimal? price; public decimal? Price { get { return price; } private set { price = value; } } The constructor parameter changes to decimal?, and then you can pass in null as the argument, or say Price = null; within the class. The meaning of the null changes from “a special reference that doesn’t refer to any object” to “a special value of any nullable type representing the absence of other data,” where all reference types and all Nullable-based types count as nullable types. That’s a lot more expressive than any of the other solutions. The rest of the code works as is—a product with an unknown price will be considered to be less expensive than $10, due to the way nullable values are handled in greater-than comparisons. To check whether a price is known, you can compare it with null or use the HasValue property, so to show all the products with unknown prices in C# 3, you’d write the fol- lowing code. List products = Product.GetSampleProducts(); foreach (Product product in products.Where(p => p.Price == null)) { Console.WriteLine(product.Name); } The C# 2 code would be similar to that in listing 1.12, but you’d need to check for null in the anonymous method: List products = Product.GetSampleProducts(); products.FindAll(delegate(Product p) { return p.Price == null; }) .ForEach(Console.WriteLine); C# 3 doesn’t offer any changes here, but C# 4 has a feature that’s at least tangentially related. Listing 1.14 Displaying products with an unknown price (C# 3) 16 CHAPTER 1 The changing face of C# development 1.3.2 Optional parameters and default values Sometimes you don’t want to tell a method everything it needs to know, such as when you almost always use the same value for a particular parameter. Traditionally the solution has been to overload the method in question, but C# 4 introduced optional parameters to make this simpler. In the C# 4 version of the Product type, you have a constructor that takes the name and the price. You can make the price a nullable decimal, just as in C# 2 and 3, but let’s suppose that most of the products don’t have prices. It would be nice to be able to initialize a product like this: Product p = new Product("Unreleased product"); Prior to C# 4, you would’ve had to introduce a new overload in the Product construc- tor for this purpose. C# 4 allows you to declare a default value (in this case, null) for the price parameter: public Product(string name, decimal? price = null) { this.name = name; this.price = price; } You always have to specify a constant value when you declare an optional parameter. It doesn’t have to be null; that just happens to be the appropriate default in this situa- tion. The requirement that the default value is a constant applies to any type of parameter, although for reference types other than strings you are limited to null as the only constant value available. Figure 1.4 summarizes the evolution we’ve looked at across different versions of C#. So far the features have been useful, but perhaps nothing to write home about. Next we’ll look at something rather more exciting: LINQ. 1.4 Introducing LINQ LINQ (Language-Integrated Query) is at the heart of the changes in C# 3. As its name suggests, LINQ is all about queries—the aim is to make it easy to write queries against multiple data sources with consistent syntax and features, in a readable and compos- able fashion. C# 2 / 3 Nullable types make the "extra work" option simple, and syntactic sugar improves matters even further. C# 1 Choice between extra work maintaining a flag, changing to reference type semantics, or the hack of a magic value. C# 4 Optional parameters allow simple defaulting. Figure 1.4 Options for working with missing data 17Introducing LINQ Whereas the features in C# 2 are arguably more about fixing annoyances in C# 1 than setting the world on fire, almost everything in C# 3 builds toward LINQ, and the result is rather special. I’ve seen features in other languages that tackle some of the same areas as LINQ, but nothing quite so well-rounded and flexible. 1.4.1 Query expressions and in-process queries If you’ve seen any LINQ code before, you’ve probably seen query expressions that allow you to use a declarative style to create queries on various data sources. The reason none of this chapter’s examples have used query expressions so far is that the exam- ples have all been simpler without using the extra syntax. That’s not to say you couldn’t use it anyway—the following listing, for example, is equivalent to listing 1.13. List products = Product.GetSampleProducts(); var filtered = from Product p in products where p.Price > 10 select p; foreach (Product product in filtered) { Console.WriteLine(product); } Personally, I find the earlier listing easier to read—the only benefit to this query expression version is that the where clause is simpler. I’ve snuck in one extra feature here—implicitly typed local variables, which are declared using the var contextual key- word. These allow the compiler to infer the type of a variable from the value that it’s initially assigned—in this case, the type of filtered is IEnumerable. I’ll use var fairly extensively in the rest of the examples in this chapter; it’s particularly useful in books, where space in listings is at a premium. But if query expressions are no good, why does everyone make such a fuss about them, and about LINQ in general? The first answer is that although query expressions aren’t particularly beneficial for simple tasks, they’re very good for more complicated situations that would be hard to read if written out in the equivalent method calls (and would be fiendish in C# 1 or 2). Let’s make things a little harder by introducing another type—Supplier. Each supplier has a Name (string) and a SupplierID (int). I’ve also added SupplierID as a property in Product and adapted the sample data appropriately. Admittedly that’s not a very object-oriented way of giving each product a supplier—it’s much closer to how the data would be represented in a database. It does make this particular feature easier to demonstrate for now, but you’ll see in chapter 12 that LINQ allows you to use a more natural model too. Now let’s look at the code (listing 1.16) that joins the sample products with the sample suppliers (obviously based on the supplier ID), applies the same price filter as before to the products, sorts by supplier name and then product name, and prints out Listing 1.15 First steps with query expressions: filtering a collection 18 CHAPTER 1 The changing face of C# development the name of both the supplier and the product for each match. That was a mouthful, and in earlier versions of C# it would’ve been a nightmare to implement. In LINQ, it’s almost trivial. List products = Product.GetSampleProducts(); List suppliers = Supplier.GetSampleSuppliers(); var filtered = from p in products join s in suppliers on p.SupplierID equals s.SupplierID where p.Price > 10 orderby s.Name, p.Name select new { SupplierName = s.Name, ProductName = p.Name }; foreach (var v in filtered) { Console.WriteLine("Supplier={0}; Product={1}", v.SupplierName, v.ProductName); } You might have noticed that this looks remarkably like SQL. Indeed, the reaction of many people on first hearing about LINQ (but before examining it closely) is to reject it as merely trying to put SQL into the language for the sake of talking to databases. Fortunately, LINQ has borrowed the syntax and some ideas from SQL, but as you’ve seen, you needn’t be anywhere near a database in order to use it. None of the code you’ve seen so far has touched a database at all. Indeed, you could be getting data from any number of sources: XML, for example. 1.4.2 Querying XML Suppose that instead of hardcoding your suppliers and products, you’d used the fol- lowing XML file: The file is simple enough, but what’s the best way of extracting the data from it? How do you query it? Join on it? Surely it’s going to be somewhat harder than what you did in listing 1.16, right? The following listing shows how much work you have to do in LINQ to XML. Listing 1.16 Joining, filtering, ordering, and projecting (C# 3) 19Introducing LINQ XDocument doc = XDocument.Load("data.xml"); var filtered = from p in doc.Descendants("Product") join s in doc.Descendants("Supplier") on (int)p.Attribute("SupplierID") equals (int)s.Attribute("SupplierID") where (decimal)p.Attribute("Price") > 10 orderby (string)s.Attribute("Name"), (string)p.Attribute("Name") select new { SupplierName = (string)s.Attribute("Name"), ProductName = (string)p.Attribute("Name") }; foreach (var v in filtered) { Console.WriteLine("Supplier={0}; Product={1}", v.SupplierName, v.ProductName); } This approach isn’t quite as straightforward, because you need to tell the system how it should understand the data (in terms of what attributes should be used as what types), but it’s not far off. In particular, there’s an obvious relationship between each part of the two listings. If it weren’t for the line-length limitations of books, you’d see an exact line-by-line correspondence between the two queries. Impressed yet? Not quite convinced? Let’s put the data where it’s much more likely to be—in a database. 1.4.3 LINQ to SQL There’s some work involved in letting LINQ to SQL know what to expect in what table, but it’s all fairly straightforward and much of it can be automated. We’ll skip straight to the querying code, which is shown in the following listing. If you want to see the details of LinqDemoDataContext, they’re all in the downloadable source code. using (LinqDemoDataContext db = new LinqDemoDataContext()) { var filtered = from p in db.Products join s in db.Suppliers on p.SupplierID equals s.SupplierID where p.Price > 10 orderby s.Name, p.Name select new { SupplierName = s.Name, ProductName = p.Name }; foreach (var v in filtered) { Console.WriteLine("Supplier={0}; Product={1}", v.SupplierName, v.ProductName); } } Listing 1.17 Complex processing of an XML file with LINQ to XML (C# 3) Listing 1.18 Applying a query expression to a SQL database (C# 3) 20 CHAPTER 1 The changing face of C# development By now, this should be looking incredibly familiar. Everything below the join line is cut and pasted directly from listing 1.16 with no changes. That’s impressive enough, but if you’re performance-conscious, you may be won- dering why you’d want to pull down all the data from the database and then apply these .NET queries and orderings. Why not get the database to do it? That’s what it’s good at, isn’t it? Well, indeed—and that’s exactly what LINQ to SQL does. The code in listing 1.18 issues a database request, which is basically the query translated into SQL. Even though you’ve expressed the query in C# code, it’s been executed as SQL. You’ll see later that there’s a more relation-oriented way of approaching this kind of join when the schema and the entities know about the relationship between suppli- ers and products. The result is the same, though, and it shows just how similar LINQ to Objects (the in-memory LINQ operating on collections) and LINQ to SQL can be. LINQ is extremely flexible—you can write your own provider to talk to a web ser- vice or translate a query into your own specific representation. In chapter 13, we’ll look at how broad the term LINQ really is, and how it can go beyond what you might expect in terms of querying collections. 1.5 COM and dynamic typing Next, I’d like to demonstrate some features that are specific to C# 4. Whereas LINQ was the major focus of C# 3, interoperability was the biggest theme in C# 4. This includes working with both the old technology of COM and also the brave new world of dynamic languages executing on the Dynamic Language Runtime (DLR). We’ll start by exporting the product list to an Excel spreadsheet. 1.5.1 Simplifying COM interoperability There are various ways of making data available to Excel, but using COM to control it gives you the most power and flexibility. Unfortunately, previous incarnations of C# made it quite difficult to work with COM; VB had much better support. C# 4 largely rectifies that situation. The following listing shows some code to save your data to a new spreadsheet. var app = new Application { Visible = false }; Workbook workbook = app.Workbooks.Add(); Worksheet worksheet = app.ActiveSheet; int row = 1; foreach (var product in Product.GetSampleProducts() .Where(p => p.Price != null)) { worksheet.Cells[row, 1].Value = product.Name; worksheet.Cells[row, 2].Value = product.Price; row++; } workbook.SaveAs(Filename: "demo.xls", FileFormat: XlFileFormat.xlWorkbookNormal); app.Application.Quit(); Listing 1.19 Saving data to Excel using COM (C# 4) 21COM and dynamic typing This may not be quite as nice as you’d like, but it’s a lot better than it would’ve been using earlier versions of C#. In fact, you already know about some of the C# 4 fea- tures shown here—but there are a couple of others that aren’t so obvious. Here’s the full list:  The SaveAs call uses named arguments.  Various calls omit arguments for optional parameters—in particular, SaveAs would normally have an extra 10 arguments!  C# 4 can embed the relevant parts of the primary interop assembly (PIA) into the calling code, so you no longer need to deploy the PIA separately.  In C# 3, the assignment to worksheet would fail without a cast, because the type of the ActiveSheet property is represented as object. When using the embed- ded PIA feature, the type of ActiveSheet becomes dynamic, which leads to an entirely different feature. Additionally, C# 4 supports named indexers when working with COM—a feature not demonstrated in this example. I’ve already mentioned the final feature: dynamic typing in C# using the dynamic type. 1.5.2 Interoperating with a dynamic language Dynamic typing is such a big topic that the entirety of chapter 14 is dedicated to it. I’ll just show you one small example of what it can do here. Suppose your products aren’t stored in a database, or in XML, or in memory. They’re accessible via a web service of sorts, but you only have Python code to access it, and that code uses the dynamic nature of Python to build results without declaring a type containing all the properties you need to access on each result. Instead, the results let you ask for any property, and try to work out what you mean at execution time. In a language like Python, there’s nothing unusual about that. But how can you access your results from C#? The answer comes in the form of dynamic—a new type4 that the C# compiler allows you to use dynamically. If an expression is of type dynamic, you can call meth- ods on it, access properties, pass it around as a method argument, and so on—and most of the normal binding process happens at execution time instead of compile time. You can implicitly convert a value from dynamic to any other type (which is why the worksheet cast in listing 1.19 worked) and do all kinds of other fun stuff. This behavior can also be useful even within pure C# code, with no interop involved, but it’s fun to see it working with other languages. The following listing shows how you can get the list of products from IronPython and print it out. This includes all the setup code to run the Python code in the same process. 4 Sort of, anyway. It’s a type as far as the C# compiler is concerned, but the CLR doesn’t know anything about it. 22 CHAPTER 1 The changing face of C# development ScriptEngine engine = Python.CreateEngine(); ScriptScope scope = engine.ExecuteFile("FindProducts.py"); dynamic products = scope.GetVariable("products"); foreach (dynamic product in products) { Console.WriteLine("{0}: {1}", product.ProductName, product.Price); } Both products and product are declared to be dynamic, so the compiler is happy to let you iterate over the list of products and print out the properties, even though it doesn’t know whether it’ll work. If you make a typo, using product.Name instead of product.ProductName, for example, that would only show up at execution time. This is completely contrary to the rest of C#, which is statically typed. But dynamic typing only comes into play when expressions with a type of dynamic are involved; most C# code is likely to remain statically typed throughout. 1.6 Writing asynchronous code without the heartache Finally you get to see C# 5’s big feature: asynchronous functions, which allow you to pause code execution without blocking a thread. This topic is big—really big—but I’ll give you just a snippet for now. As I’m sure you’re aware, there are two golden rules when it comes to threading in Windows Forms: you mustn’t block the UI thread, and you mustn’t access UI elements on any other thread, except in a few well-specified ways. The following listing shows a single method that handles a button click in a Windows Forms application and displays information about a product, given its ID. private async void CheckProduct(object sender, EventArgs e) { try { productCheckButton.Enabled = false; string id = idInput.Text; Task productLookup = directory.LookupProductAsync(id); Task stockLookup = warehouse.LookupStockLevelAsync(id); Product product = await productLookup; if (product == null) { return; } nameValue.Text = product.Name; priceValue.Text = product.Price.ToString("c"); int stock = await stockLookup; stockValue.Text = stock.ToString(); } Listing 1.20 Running IronPython and extracting properties dynamically (C# 4) Listing 1.21 Displaying products in Windows Forms using an asynchronous function 23Dissecting the .NET platform finally { productCheckButton.Enabled = true; } } The full method is a little longer than the one shown in listing 1.22, displaying status messages and clearing the results at the start, but this listing contains all the important parts. The new parts of syntax are in bold—the method has the new async modifier, and there are two await expressions. If you squint and ignore those for the moment, you can probably understand the general flow of the code. It starts off performing lookups on both the product direc- tory and warehouse to find out the product details and current stock. The method then waits until it has the product information, and quits if the directory has no entry for the given ID. Otherwise, it fills in the UI elements for the name and price, and then waits to get the stock information, and displays that too. Both the product and stock lookups are asynchronous—they could be database operations or web service calls. It doesn’t matter—when you await the results, you’re not actually blocking the UI thread, even though all the code in the method runs on that thread. When the results come back, the method continues from where it left off. The example also demonstrates that normal flow control (try/finally) operates exactly as you’d expect it to. The really surprising thing about this method is that it has managed to achieve exactly the kind of asynchrony you want without any of the normal messing around starting other threads or BackgroundWorkers, calling Control .BeginInvoke, or attaching callbacks to asynchronous events. Of course you still need to think—asynchrony doesn’t become easy using async/await, but it becomes less tedious, with far less boilerplate code to distract you from the inherent complexity you’re trying to control. Are you dizzy yet? Relax, I’ll slow down considerably for the rest of the book. In particular, I’ll explain some of the corner cases, going into more detail about why vari- ous features were introduced, and giving some guidance as to when it’s appropriate to use them. So far I’ve been showing you features of C#. Some of these features require library assistance, and some of them require runtime assistance. I’ll say this sort of thing a lot, so let’s clear up what I mean. 1.7 Dissecting the .NET platform When it was originally introduced, .NET was used as a catchall term for a vast range of technologies coming from Microsoft. For instance, Windows Live ID was called .NET Passport, despite there being no clear relationship between that and what you cur- rently know as .NET. Fortunately, things have calmed down somewhat since then. In this section, we’ll look at the various parts of .NET. 24 CHAPTER 1 The changing face of C# development In several places in this book, I’ll refer to three different kinds of features: features of C# as a language, features of the runtime that provides the “engine,” if you will, and features of the .NET framework libraries. This book is heavily focused on the language of C#, and I’ll generally only discuss runtime and framework features when they relate to features of C# itself. Often features will overlap, but it’s important to understand where the boundaries lie. 1.7.1 C#, the language The language of C# is defined by its specification, which describes the format of C# source code, including both syntax and behavior. It doesn’t describe the platform that the compiler output will run on, beyond a few key points where the two interact. For instance, the C# language requires a type called System.IDisposable, which contains a method called Dispose. These are required in order to define the using statement. Likewise, the platform needs to be able to support (in one form or another) both value types and reference types, along with garbage collection. In theory, any platform that supports the required features could have a C# compiler targeting it. For example, a C# compiler could legitimately produce output in a form other than the Intermediate Language (IL), which is the typical output at the time of this writing. A runtime could interpret the output of a C# compiler, or convert it all to native code in one step rather than JIT-compiling it. Though these options are relatively uncommon, they do exist in the wild; for example, the Micro Framework uses an inter- preter, as can Mono (http://mono-project.net). At the other end of the spectrum, ahead-of-time compilation is used by NGen and by Xamarin.iOS (http://xamarin .com/ios)—a platform for building applications for the iPhone and other iOS devices. 1.7.2 Runtime The runtime aspect of the .NET platform is the relatively small amount of code that’s responsible for making sure that programs written in IL execute according to the Com- mon Language Infrastructure (CLI) specification (ECMA-335 and ISO/IEC 23271), parti- tions I to III. The runtime part of the CLI is called the Common Language Runtime (CLR). When I refer to the CLR in the rest of the book, I mean Microsoft’s implementation. Some elements of the C# language never appear at the runtime level, but others cross the divide. For instance, enumerators aren’t defined at a runtime level, and nei- ther is any particular meaning attached to the IDisposable interface, but arrays and delegates are important to the runtime. 1.7.3 Framework libraries Libraries provide code that’s available to your programs. The framework libraries in .NET are largely built as IL themselves, with native code used only where necessary. This is a mark of the strength of the runtime: your own code isn’t expected to be a second-class citizen—it can provide the same kind of power and performance as the 25Making your code super awesome libraries it utilizes. The amount of code in the libraries is much greater than that of the runtime, in the same way that there’s much more to a car than the engine. The framework libraries are partially standardized. Partition IV of the CLI specifi- cation provides a number of different profiles (compact and kernel) and libraries. Parti- tion IV comes in two parts—a general textual description of the libraries identifying, among other things, which libraries are required within which profiles, and another part containing the details of the libraries themselves in XML format. This is the same form of documentation produced when you use XML comments within C#. There’s much within .NET that’s not within the base libraries. If you write a pro- gram that only uses libraries from the specification, and uses them correctly, you should find that your code works flawlessly on any implementation—Mono, .NET, or anything else. But in practice, almost any program of any size will use libraries that aren’t standardized—Windows Forms or ASP.NET, for instance. The Mono project has its own libraries that aren’t part of .NET, such as GTK#, and it implements many of the nonstandardized libraries. The term .NET refers to the combination of the runtime and libraries provided by Microsoft, and it also includes compilers for C# and VB.NET. It can be seen as a whole development platform built on top of Windows. Each aspect of .NET is versioned sepa- rately, which can be a source of confusion. Appendix C gives a quick rundown of which version of what came out when and with what features. If that’s all clear, I have one last bit of housekeeping to go through before we really start diving into C#. 1.8 Making your code super awesome I apologize for the misleading heading. This section (in itself) will not make your code super awesome. It won’t even make it refreshingly minty. It will help you make the most of this book, though—and that’s why I wanted to make sure you actually read it. There’s more of this sort of thing in the front matter (the bit before page 1), but I know that many readers skip over that, heading straight for the meat of the book. I can understand that, so I’ll make this as quick as possible. 1.8.1 Presenting full programs as snippets One of the challenges when writing a book about a computer language (other than scripting languages) is that complete programs—ones that the reader can compile and run with no source code other than what’s presented—get long pretty quickly. I wanted to get around this, to provide you with code that you could easily type in and experiment with. I believe that actually trying something is a much better way of learn- ing than just reading about it. With the right assembly references and the right using directives, you can accom- plish a lot with a fairly short amount of C# code, but the killer is the fluff involved in writing those using directives, declaring a class, and declaring a Main method before you’ve written the first line of useful code. My examples are mostly in the form of 26 CHAPTER 1 The changing face of C# development snippets, which ignore the fluff that gets in the way of simple programs, concentrat- ing on the important parts. The snippets can be run directly in a small tool I’ve built, called Snippy. If a snippet doesn’t contain an ellipsis (...), then all of the code should be consid- ered to be the body of the Main method of a program. If there is an ellipsis, then everything before it is treated as declarations of methods and nested types, and every- thing after the ellipsis goes in the Main method. For example, consider this snippet: static string Reverse(string input) { char[] chars = input.ToCharArray(); Array.Reverse(chars); return new string(chars); } ... Console.WriteLine(Reverse("dlrow olleH")); This is expanded by Snippy into the following: using System; public class Snippet { static string Reverse(string input) { char[] chars = input.ToCharArray(); Array.Reverse(chars); return new string(chars); } [STAThread] static void Main() { Console.WriteLine(Reverse("dlrow olleH")); } } In reality, Snippy includes far more using directives, but the expanded version was already getting long. Note that the containing class will always be called Snippet, and any types declared within the snippet will be nested within that class. There are more details about how to use Snippy on the book’s website (http:// mng.bz/Lh82), along with all the examples as both snippets and expanded versions in Visual Studio solutions. Additionally, there’s support for LINQPad (http:// www.linqpad.net)—a similar tool developed by Joe Albahari, with particularly helpful features for exploring LINQ. Next, let’s look at what’s wrong with the code we’ve just seen. 1.8.2 Didactic code isn’t production code It’d be lovely if you could take all the examples from this book and use them directly in your own applications with no further thought involved…but I strongly suggest you don’t. Most examples are presented to demonstrate a specific point—and that’s usu- 27Making your code super awesome ally the limit of the intent. Most snippets don’t include argument validation, access modifiers, unit tests, or documentation. They may also fail when used outside their intended context. For example, let’s consider the body of the method previously shown for reversing a string. I use this code several times in the course of the book: char[] chars = input.ToCharArray(); Array.Reverse(chars); return new string(chars); Leaving aside argument validation, this succeeds in reversing the sequence of UTF-16 code points within a string, but in some cases that’s not good enough. For example, if a single displayed glyph is composed of an e followed by a combining character repre- senting an acute accent, you don’t want to switch the sequence of the code points; the accent will end up on the wrong character. Or suppose your string contains a charac- ter outside the basic multilingual plane, formed from a surrogate pair—reordering the code points will lead to a string that’s effectively invalid UTF-16. Fixing these prob- lems would lead to much more complicated code, distracting from the point it’s meant to be demonstrating. You’re welcome to use the code from the book, but please bear this section in mind if you do so—it’d be much better to take inspiration from it than to copy it ver- batim and assume it’ll meet your particular requirements. Finally, there’s another book you should download in order to make the absolute most of this one. 1.8.3 Your new best friend: the language specification I’ve tried extremely hard to be accurate in this book, but I’d be amazed if there were no errors at all—indeed, you’ll find a list of known errors on the book’s website (http://mng.bz/m1Hh). If you think you’ve found a mistake, I’d be grateful if you could email me (skeet@pobox.com) or add a note on the author forum (http:// mng.bz/TQmF). But you may not want to wait for me to get back to you, or you may have a question that isn’t covered in the book. Ultimately, the definitive source for the intended behavior of C# is the language specification. There are two important forms of the spec—the international standard from ECMA, and the Microsoft specification. As I write this, the ECMA specification (ECMA- 334 and ISO/IEC 23270) only covers C# 2, despite being the fourth edition. It’s unclear whether or when this will be updated, but the Microsoft version is complete and freely available. This book’s website has links to all the available versions of both specification flavors (http://mng.bz/8s38), and Visual Studio ships with a copy too.5 When I refer to sections of the specification within this book, I’ll use numbering from 5 The exact location of the specification will depend on your system, but on my Visual Studio 2012 Professional installation, it’s in C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC#\Specifications\1033. 28 CHAPTER 1 The changing face of C# development the Microsoft C# 5 specification, even when I’m talking about earlier versions of the language. I strongly recommend that you download this version and have it on hand whenever you find yourself eager to check out a weird corner case. One of my aims is to make the spec mostly redundant for developers—to provide a more developer-oriented form covering everything you’re likely to see in everyday code, without the huge level of detail required by compiler authors. Having said that, it’s extremely readable as specifications go, and you shouldn’t be daunted by it. If you find the spec interesting, there are annotated versions available for C# 3 and C# 4. Both contain fascinating comments from the C# team and other contributors. (Dis- claimer: I’m one of the “other contributors” for the C# 4 edition…but all the other comments are great!) 1.9 Summary In this chapter, I’ve shown (but not explained) some of the features that are tackled in depth in the rest of the book. There are plenty more that I haven’t shown here, and many of the features you’ve seen so far have further subfeatures associated with them. Hopefully what you’ve seen here has whetted your appetite for the rest of the book. Although features have taken up most of the chapter, we’ve also looked at some areas that should help you get the most out of the book. I’ve clarified what I mean when I refer to the language, runtime, and libraries, and I’ve also explained how code will be laid out in the book. There’s one more area we need to cover before we dive into the features of C# 2, and that’s C# 1. Obviously, as an author I have no idea how knowledgeable you are about C# 1, but I do have some understanding of which areas of C# often cause con- ceptual problems. Some of these areas are critical to getting the most out of the later versions of C#, so in the next chapter I’ll go over them in some detail. 29 Core foundations: building on C# 1 This isn’t a refresher on the whole of C# 1. Let’s get that out of the way immediately. I couldn’t do justice to any topic in C# if I had to cover the whole of the first version in a single chapter. I’ve written this book assuming that you’re at least reasonably competent in C# 1. What counts as “reasonably competent” is, of course, somewhat subjective, but I’ll assume you’d at least be happy to walk into an interview for a junior C# developer role and answer technical questions appropriate to that job. You may well have more experience, but that’s the level of knowledge I’m assuming. In this chapter, we’ll focus on three areas of C# 1 that are particularly important in order to understand the features of later versions. This should raise the lowest common denominator a little, so that I can make slightly greater assumptions later in the book. Given that it is a lowest common denominator, you may find you already have a perfect understanding of all the concepts in this chapter. If you believe that’s the case without even reading the chapter, then feel free to skip it. This chapter covers  Delegates  Type system characteristics  Value/reference types 30 CHAPTER 2 Core foundations: building on C# 1 You can always come back later if it turns out something isn’t as simple as you thought. If you’re not certain you know everything in this chapter, you might want to look at the summary at the end of each section, which highlights the important points—if any of those sound unfamiliar, it’s worth reading that section in detail. We’ll start off by looking at delegates, then consider how the C# type system com- pares with some other possibilities, and finally look at the differences between value types and reference types. For each topic, I’ll describe the ideas and behavior, as well as take the opportunity to define terms so that I can use them later on. After we’ve looked at how C# 1 works, I’ll show you a quick preview of how many of the new fea- tures in later versions relate to the topics examined in this chapter. 2.1 Delegates I’m sure you already have an instinctive idea about what a delegate is, even though it can be hard to articulate. If you’re familiar with C and had to describe delegates to another C programmer, the term function pointer would no doubt crop up. Essentially, delegates provide a level of indirection: instead of specifying behavior to be executed immediately, the behavior can somehow be “contained” in an object. That object can then be used like any other, and one operation you can perform with it is to execute the encapsulated action. Alternatively, you can think of a delegate type as a single- method interface, and a delegate instance as an object implementing that interface. If that’s just gobbledygook to you, maybe an example will help. It’s slightly morbid, but it does capture what delegates are all about. Consider your will—your last will and testament. It’s a set of instructions: “pay the bills, make a donation to charity, leave the rest of my estate to the cat,” for instance. You write it before your death, and leave it in an appropriately safe place. After your death, your attorney will (you hope!) act on those instructions. A delegate in C# acts like your will does in the real world—it allows you to specify a sequence of actions to be executed at the appropriate time. Delegates are typically used when the code that wants to execute the actions doesn’t know the details of what those actions should be. For instance, the only reason why the Thread class knows what to run in a new thread when you start it is because you provide the constructor with a ThreadStart or ParameterizedThreadStart delegate instance. We’ll start our tour of delegates with the four absolute basics, without which none of the rest would make sense. 2.1.1 A recipe for simple delegates In order for a delegate to do anything, four things need to happen:  The delegate type needs to be declared.  The code to be executed must be contained in a method.  A delegate instance must be created.  The delegate instance must be invoked. Let’s take each step of this recipe in turn. 31Delegates DECLARING THE DELEGATE TYPE A delegate type is effectively a list of parameter types and a return type. It specifies what kind of action can be represented by instances of the type. For instance, consider a delegate type declared like this: delegate void StringProcessor(string input); The code says that if you want to create an instance of StringProcessor, you’ll need a method with one parameter (a string) and a void return type (the method doesn’t return anything). It’s important to understand that StringProcessor really is a type, deriving from System.MulticastDelegate, which in turn derives from System.Delegate. It has methods, you can create instances of it and pass around references to instances, the whole works. There are obviously a few special features, but if you’re ever stuck won- dering what’ll happen in a particular situation, first think about what would happen if you were using a normal reference type. SOURCE OF CONFUSION: THE AMBIGUOUS TERM DELEGATE Delegates can be mis- understood because the word delegate is often used to describe both a delegate type and a delegate instance. The distinction between these two is exactly the same as between any other type and instances of that type—the string type itself is different from a particular sequence of characters, for example. I’ve used the terms delegate type and delegate instance throughout this chapter to try to keep clear exactly what I’m talking about at any point. We’ll use the StringProcessor delegate type when we consider the next ingredient. FINDING AN APPROPRIATE METHOD FOR THE DELEGATE INSTANCE’S ACTION The next ingredient is to find (or write) a method that does what you want and has the same signature as the delegate type you’re using. The idea is to make sure that when you try to invoke a delegate instance, the parameters you use will all match up, and you’ll be able to use the return value (if any) in the way you expect—just like a normal method call. Consider these five method signatures as candidates to be used for a String- Processor instance: void PrintString(string x) void PrintInteger(int x) void PrintTwoStrings(string x, string y) int GetStringLength(string x) void PrintObject(object x) The first method has everything right, so you can use it to create a delegate instance. The second method has one parameter, but it’s not string, so it’s incompatible with StringProcessor. The third method has the correct first parameter type, but it has another parameter as well, so it’s still incompatible. The fourth method has the right parameter list but a nonvoid return type. (If your delegate type had a return type, the return type of the method would have to match that too.) 32 CHAPTER 2 Core foundations: building on C# 1 The fifth method is interesting—any time you invoke a StringProcessor instance, you could call the PrintObject method with the same arguments, because string derives from object. It would make sense to be able to use it for an instance of StringProcessor, but in C# 1 the delegate must have exactly the same parameter types.1 C# 2 changes this situation—see chapter 5 for more details. In some ways, the fourth method is similar, since you could always ignore the unwanted return value. But void and nonvoid return types are currently always deemed to be incompatible. This is partly because other aspects of the system (particularly the JIT) need to know whether a value will be left on the stack as a return value when a method is executed.2 Let’s assume you have a method body for the compatible signature (PrintString) and move on to the next ingredient—the delegate instance itself. CREATING A DELEGATE INSTANCE Now that you have a delegate type and a method with the right signature, you can cre- ate an instance of that delegate type, specifying that this method be executed when the delegate instance is invoked. No official terminology has been defined for this, but for this book I’ll call it the action of the delegate instance. The exact form of the expression used to create the delegate instance depends on whether the action uses an instance method or a static method. Suppose PrintString is a static method in a type called StaticMethods and an instance method in a type called InstanceMethods. Here are two examples of creating an instance of String- Processor: StringProcessor proc1, proc2; proc1 = new StringProcessor(StaticMethods.PrintString); InstanceMethods instance = new InstanceMethods(); proc2 = new StringProcessor(instance.PrintString); When the action is a static method, you only need to specify the type name. When the action is an instance method, you need an instance of the type (or a derived type), as you normally would. This object is called the target of the action, and when the dele- gate instance is invoked, the method will be called on that object. If the action is within the same class (as it often is, particularly when you’re writing event handlers in UI code), you don’t need to qualify it either way—the this reference is used implicitly for instance methods.3 Again, these rules act just as if you were calling the method directly. UTTER GARBAGE! (OR NOT, AS THE CASE MAY BE) It’s worth being aware that a delegate instance will prevent its target from being garbage collected if the delegate instance itself can’t be collected. This can result in apparent 1 In addition to the parameter types, you have to match whether the parameter is in (the default), out, or ref. It’s reasonably rare to use out and ref parameters with delegates, though. 2 This is a deliberately vague use of the word stack to avoid going into too much irrelevant detail. See Eric Lip- pert’s blog post “The void is invariant” for more information (http://mng.bz/4g58). 3 Of course, if the action is an instance method and you’re trying to create a delegate instance from within a static method, you’ll still need to provide a reference to be the target. 33Delegates memory leaks, particularly when a short-lived object subscribes to an event in a long-lived object, using itself as the target. The long-lived object indirectly holds a reference to the short-lived one, prolonging its lifetime. There’s not much point in creating a delegate instance if it doesn’t get invoked at some point. Let’s look at the last step—the invocation. INVOKING A DELEGATE INSTANCE Invoking a delegate instance is the really easy bit:4 it’s just a matter of calling a method on the delegate instance. The method itself is called Invoke, and it’s always present in a delegate type with the same list of parameters and return type that the delegate type declaration specifies. In our continuing example, there’s a method like this: void Invoke(string input) Calling Invoke will execute the action of the delegate instance, passing on whatever arguments you’ve speci- fied in the call to Invoke, and (if the return type isn’t void) returning the return value of the action. As simple as this is, C# makes it even easier; if you have a variable5 whose type is a delegate type, you can treat it as if it were a method itself. It’s easiest to see this happening as a chain of events occurring at differ- ent times, as shown in figure 2.1. As you can see, that’s simple too. All the ingredi- ents are now in place, so you can preheat your CLR to 200°C, stir everything together, and see what happens. A COMPLETE EXAMPLE AND SOME MOTIVATION It’s easiest to see all this in action in a complete example—finally, something you can actually run! As there are lots of bits and pieces involved, I’ve included the whole source code this time rather than using snippets. There’s nothing mind-blowing in the following listing, so don’t expect to be amazed—it’s just useful to have concrete code to discuss. using System; delegate void StringProcessor(string input); class Person { string name; public Person(string name) { this.name = name; } 4 For synchronous invocation, anyway. You can use BeginInvoke and EndInvoke to invoke a delegate instance asynchronously, but that’s beyond the scope of this chapter. 5 Or any other kind of expression—but it’s usually a variable. Listing 2.1 Using delegates in a variety of simple ways Declares delegate typeB proc1("Hello"); proc1.Invoke("Hello"); PrintString("Hello"); Compiles to... Which at execution time invokes... Figure 2.1 Processing a call to a delegate instance that uses the C# shorthand syntax 34 CHAPTER 2 Core foundations: building on C# 1 public void Say(string message) { Console.WriteLine("{0} says: {1}", name, message); } } class Background { public static void Note(string note) { Console.WriteLine("({0})", note); } } class SimpleDelegateUse { static void Main() { Person jon = new Person("Jon"); Person tom = new Person("Tom"); StringProcessor jonsVoice, tomsVoice, background; jonsVoice = new StringProcessor(jon.Say); tomsVoice = new StringProcessor(tom.Say); background = new StringProcessor(Background.Note); jonsVoice("Hello, son."); tomsVoice.Invoke("Hello, Daddy!"); background("An airplane flies past."); } } To start with, you declare the delegate type B. Next, you create two methods (C and D) that are both compatible with the delegate type. You have one instance method (Person.Say) and one static method (Background.Note), so you’ll see how they’re used differently when you create the delegate instances E. Listing 2.1 includes two instances of the Person class, so you can see the difference that the tar- get of a delegate makes. When jonsVoice is invoked F, it calls the Say method on the Person object with the name Jon; likewise, when tomsVoice is invoked, it uses the object with the name Tom. This code includes both ways of invoking delegate instances that you’ve seen— calling Invoke explicitly and using the C# shorthand—just for interest’s sake. Nor- mally you’d use the shorthand. The output of listing 2.1 is fairly obvious: Jon says: Hello, son. Tom says: Hello, Daddy! (An airplane flies past.) Frankly, there’s an awful lot of code in listing 2.1 to display three lines of output. Even if you wanted to use the Person class and the Background class, there’s no real need to use delegates here. So what’s the point? Why not just call the methods directly? The answer lies in our original example of an attorney executing a will—just because you want something to happen doesn’t mean you’re always there at the right time and Declares compatible instance methodC Declares compatible static method D Creates three delegate instances E Invokes delegate instances F 35Delegates place to make it happen. Sometimes you need to give instructions—to delegate respon- sibility, as it were. I should stress that back in the world of software, this isn’t a matter of objects leav- ing dying wishes. Often the object that first creates a delegate instance is still alive and well when the delegate instance is invoked. Instead, it’s about specifying some code to be executed at a particular time, when you may not be able (or may not want) to change the code that’s running at that point. If I want something to happen when a button is clicked, I don’t want to have to change the code of the button—I just want to tell the button to call one of my methods, which will take the appropriate action. It’s a matter of adding a level of indirection, as so much of object-oriented programming is. As you’ve seen, this adds complexity (look at how many lines of code it took to pro- duce so little output!) but also flexibility. Now that you understand more about simple delegates, we’ll take a brief look at combining delegates together to execute a whole bunch of actions instead of just one. 2.1.2 Combining and removing delegates So far, all the delegate instances we’ve looked at have had a single action. In reality, life is a bit more complicated: a delegate instance actually has a list of actions associ- ated with it called the invocation list. The static Combine and Remove methods of the System.Delegate type are responsible for creating new delegate instances by respec- tively splicing together the invocation lists of two delegate instances or removing the invocation list of one delegate instance from another. DELEGATES ARE IMMUTABLE Once you’ve created a delegate instance, noth- ing about it can be changed. This makes it safe to pass around references to delegate instances and combine them with others without worrying about consistency, thread safety, or anyone trying to change their actions. This is like strings, which are also immutable, and Delegate.Combine is just like String.Concat—they both combine existing instances together to form a new one without changing the original objects at all. In the case of delegate instances, the original invocation lists are concatenated together. Note that if you ever try to combine null with a delegate instance, the null is treated as if it were a delegate instance with an empty invoca- tion list. You’ll rarely see an explicit call to Delegate .Combine in C# code—usually the + and += opera- tors are used. Figure 2.2 shows the translation process, where x and y are both variables of the same (or compatible) delegate types. All of this is done by the C# compiler. void Dump(int x, int y = 20, int z required parameter default values optional parameters Figure 2.2 The transformation process used for the C# shorthand syntax for combining delegate instances 36 CHAPTER 2 Core foundations: building on C# 1 As you can see, it’s a straightforward transformation, but it makes the code a lot neater. Just as you can combine delegate instances, you can remove one from another with the Delegate.Remove method, and C# uses the shorthand of the - and -= opera- tors in the obvious way. Delegate.Remove(source, value) creates a new delegate whose invocation list is the one from source, with the list from value having been removed. If the result would have an empty invocation list, null is returned. When a delegate instance is invoked, all its actions are executed in order. If the delegate’s signature has a nonvoid return type, the value returned by Invoke is the value returned by the last action executed. It’s rare to see a nonvoid delegate instance with more than one action in its invocation list because it means the return values of all the other actions are never seen unless the invoking code explicitly executes the actions one at a time, using Delegate.GetInvocationList to fetch the list of actions. If any of the actions in the invocation list throws an exception, that prevents any of the subsequent actions from being executed. For example, if a delegate instance with an invocation list [a, b, c] is invoked, and action b throws an exception, the excep- tion will be propagated immediately and action c won’t be executed. Combining and removing delegate instances is particularly useful when it comes to events. Now that you understand what combining and removing involves, we can talk about events. 2.1.3 A brief diversion into events You probably have an instinctive idea about the overall point of events, particularly if you’ve written any UIs. The idea is that an event allows code to react when something happens—saving a file when the appropriate button is clicked, for example. In this case, the event is the clicking of the button, and the action is the saving of the file. Understanding the reason for the concept isn’t the same as understanding how C# defines events in language terms, though. Developers often confuse events and delegate instances, or events and fields declared with delegate types. The difference is important: events aren’t fields. The rea- son for the confusion is that C# provides a shorthand in the form of field-like events. We’ll come to those in a minute, but first let’s consider what events consist of as far as the C# compiler is concerned. It’s helpful to think of events as being similar to properties. To start with, both of them are declared to be of a certain type—an event is forced to be a delegate type. When you use properties, it looks like you’re fetching or assigning values directly to fields, but you’re actually calling methods (getters and setters). The property imple- mentation can do what it likes within those methods—it just happens that most prop- erties are implemented with simple fields backing them, sometimes with some validation in the setter and sometimes with some thread safety thrown in for good measure. Likewise, when you subscribe to or unsubscribe from an event, it looks like you’re using a field whose type is a delegate type, with the += and -= operators. Again, 37Delegates though, you’re actually calling methods (add and remove).6 That’s all you can do with an event—subscribe to it (add an event handler) or unsubscribe from it (remove an event handler). It’s up to the event methods to do something useful, such as taking notice of the event handlers you’re trying to add and remove, and making them avail- able elsewhere within the class. The reason for having events in the first place is much like the reason for having properties—they add a layer of encapsulation, implementing the publish/subscribe pattern (see my article, “Delegates and Events,” here: http://mng.bz/HPx6). Just as you don’t want other code to be able to set field values without the owner at least hav- ing the option of validating the new value, you often don’t want code outside a class to be able to arbitrarily change (or call) the handlers for an event. Of course, a class can add methods to give extra access—for instance, to reset the list of handlers for an event, or to raise the event (in other words, to call its event handlers). For example, BackgroundWorker.OnProgressChanged just calls the ProgressChanged event han- dlers. But if you only expose the event itself, code outside the class only has the ability to add and remove handlers. Field-like events make the implementation of all of this much simpler to look at—a single declaration and you’re done. The compiler turns the declaration into both an event with default add/remove implementations and a private field of the same type. Code inside the class sees the field; code outside the class only sees the event. This makes it look as if you can invoke an event, but what you actually do to call the event handlers is invoke the delegate instance stored in the field. The details of events are outside the scope of this chapter—events themselves haven’t changed much in later versions of C#,7 but I wanted to draw attention to the difference between delegate instances and events now, to prevent confusion later on. 2.1.4 Summary of delegates Let’s summarize what we’ve covered on delegates:  Delegates encapsulate behavior with a particular return type and set of parame- ters, similar to a single-method interface.  The type signature described by a delegate type declaration determines which methods can be used to create delegate instances, and the signature for invocation.  Creating a delegate instance requires a method and (for instance methods) a target to call the method on.  Delegate instances are immutable.  Delegate instances each contain an invocation list—a list of actions. 6 These aren’t their names in the compiled code; otherwise you could only have one event per type. The com- piler creates two methods with names that aren’t used elsewhere, and includes a special piece of metadata to let other types know that there’s an event with the given name, and what its add/remove methods are called. 7 There are very small changes to field-like events in C# 4. See section 4.2 for details. 38 CHAPTER 2 Core foundations: building on C# 1  Delegate instances can be combined with and removed from each other.  Events aren’t delegate instances—they’re just add/remove method pairs (think property getters/setters). Delegates are one specific feature of C# and .NET—a detail in the grand scheme of things. Both of the other reminder sections in this chapter deal with much broader topics. First, we’ll consider what it means to talk about C# being a statically typed lan- guage and the implications that has. 2.2 Type system characteristics Almost every programming language has a type system of some kind. Over time, these have been classified as strong/weak, safe/unsafe, static/dynamic, and no doubt some more esoteric variations. It’s obviously important to understand the type system you’re working with, and it’s reasonable to expect that knowing the categories into which a language falls would give you a lot of information on that front. But because the terms are used by different people to mean somewhat different things, miscom- munication is almost inevitable. I’ll try to explain exactly what I mean by each term to minimize confusion. One important thing to note is that this section is only applicable to safe code, which means all C# code that isn’t explicitly within an unsafe context. As you might judge from the name, code within an unsafe context can do various things that safe code can’t, and that may violate some aspects of normal type safety, although the type system is still safe in many other ways. Most developers are unlikely ever to need to write unsafe code, and the characteristics of the type system are far simpler to describe and understand when only safe code is considered. This section shows what restrictions are and aren’t enforced in C# 1 while defining some terms to describe that behavior. We’ll then look at a few things you can’t do with C# 1—first from the point of view of what you can’t tell the compiler, and then from the point of view of what you might wish you didn’t have to tell the compiler. Let’s start off with what C# 1 does, and with the terminology that’s usually used to describe that kind of behavior. 2.2.1 C#’s place in the world of type systems It’s easiest to begin by making a statement and then clarifying what it means and what the alternatives might be: C# 1’s type system is static, explicit, and safe. You might have expected the word strong to appear in the list, and I had half a mind to include it. But although most people can reasonably agree on whether a language has the characteristics I listed, deciding whether a language is strongly typed can cause heated debate because the definitions vary so wildly. Some meanings (those prevent- ing any conversions, explicit or implicit) would clearly rule C# out, whereas others are quite close to (or even the same as) statically typed, which would include C# 1. Most of 39Type system characteristics the articles and books I’ve read that describe C# as a strongly typed language are effec- tively using “strongly typed” to mean statically typed. Let’s take the terms in the definition one at a time and shed some light on them. STATIC TYPING VERSUS DYNAMIC TYPING C# 1 is statically typed: each variable is of a particular type, and that type is known at compile time.8 Only operations that are known for that type are allowed, and this is enforced by the compiler. Consider this example of enforcement: object o = "hello"; Console.WriteLine(o.Length); As you look at the code, it’s obvious that the value of o refers to a string, and that the string type has a Length property, but the compiler only thinks of o as being of type object. If you want to get to the Length property, you have to tell the compiler that the value of o refers to a string: object o = "hello"; Console.WriteLine(((string)o).Length); The compiler is then able to find the Length property of System.String. It uses this to validate that the call is correct, emit the appropriate IL, and work out the type of the larger expression. The compile-time type of an expression is also known as its static type—so you might say, “The static type of o is System.Object.” WHY IS IT CALLED STATIC TYPING? The word static is used to describe this kind of typing because the analysis of what operations are available is per- formed using unchanging data: the compile-time types of expressions. Sup- pose a variable is declared to be of type Stream; the type of the variable doesn’t change even if the value of the variable varies from a reference to a MemoryStream, a FileStream, or no stream at all (with a null reference). Even within static type systems, there can be some dynamic behavior; the actual implementation executed by a virtual method call will depend on the value it’s called on. The idea of unchanging information is also the motiva- tion behind the static modifier, but it’s generally simpler to think of a static member as one belonging to the type itself rather than to any particu- lar instance of the type. For most practical purposes, you can think of the two uses of the word as unrelated. The alternative to static typing is dynamic typing, which can take a variety of guises. The essence of dynamic typing is that variables just have values—they aren’t restricted to particular types, so the compiler can’t perform the same sort of checks. Instead, the execution environment attempts to understand expressions in an appropriate manner for the values involved. For example, if C# 1 were dynamically typed, you could do this: 8 This applies to most expressions too, but not quite all of them. Certain expressions don’t have a type, such as void method invocations, but this doesn’t affect C# 1’s status of being statically typed. I’ve used the word vari- able throughout this section to avoid unnecessary brain strain. 40 CHAPTER 2 Core foundations: building on C# 1 o = "hello"; Console.WriteLine(o.Length); o = new string[] {"hi", "there"}; Console.WriteLine(o.Length); This would invoke two completely unrelated Length properties—String.Length and Array.Length—by examining the types dynamically at execution time. Like many aspects of type systems, there are different levels of dynamic typing. Some languages allow you to specify types where you want to—possibly still treating them dynamically apart from assignment—but let you use untyped variables elsewhere. Although I’ve specified C# 1 repeatedly in this description, C# was entirely stati- cally typed up to and including C# 3. You’ll see later that C# 4 introduced some dynamic typing, although the vast majority of code in most C# 4 applications will still use static typing. EXPLICIT TYPING VERSUS IMPLICIT TYPING The distinction between explicit typing and implicit typing is only relevant in statically typed languages. With explicit typing, the type of every variable must be explicitly stated in the declaration. Implicit typing allows the compiler to infer the type of the variable based on its use. For example, the language could dictate that the type of the variable is the type of the expression used to assign the initial value. Consider a hypothetical language that uses the keyword var to indicate type infer- ence.9 Table 2.1 shows how code in such a language could be written in C# 1. The code in the left column is not allowed in C# 1, but the code in the right column is the equivalent valid code. Hopefully it’s clear why this is only relevant for statically typed situations: for both implicit and explicit typing, the type of the variable is known at compile time, even if it’s not explicitly stated. In a dynamic context, the variable doesn’t even have a compile-time type to state or infer. TYPE-SAFE VERSUS TYPE-UNSAFE The easiest way of describing a type-safe system is to describe its opposite. Some lan- guages (I’m thinking particularly of C and C++) allow you to do some really devious things. They’re potentially powerful in the right situations, but with great power comes a free box of donuts, or however the expression goes, and the right situations 9 Okay, not so hypothetical. See section 8.2 for C# 3’s implicitly typed local variable capabilities. Table 2.1 An example showing the differences between implicit and explicit typing Invalid C# 1—implicit typing Valid C# 1—explicit typing var s = "hello"; string s = "hello"; var x = s.Length; int x = s.Length; var twiceX = x * 2; int twiceX = x * 2; WHAT IF? 41Type system characteristics are relatively rare. Some of these devious things can shoot you in the foot if you get them wrong. Abusing the type system is one of them. With the right voodoo rituals, you can persuade these languages to treat a value of one type as if it were a value of a completely different type without applying any conver- sions. I don’t just mean calling a method that happens to have the same name, as in the dynamic typing example earlier. I mean code that looks at the raw bytes within a value and interprets them in the “wrong” way. The following listing gives a simple C example of what I mean. #include int main(int argc, char**argv) { char *first_arg = argv[1]; int *first_arg_as_int = (int *)first_arg; printf ("%d", *first_arg_as_int); } If you compile listing 2.2 and run it with a simple argument of "hello", you’ll see a value of 1819043176—at least on a little-endian architecture with a compiler treating int as 32 bits and char as 8 bits, and where text is represented in ASCII or UTF-8. The code is treating the char pointer as an int pointer, so dereferencing it returns the first 4 bytes of text, treating them as a number. In fact, this tiny example is tame compared with other potential abuses—casting between completely unrelated structs can easily result in total mayhem. It’s not that this happens in real life very often, but some elements of the C typing system often require you to tell the compiler what to do, leaving it no option but to trust you even at execution time. Fortunately, none of this occurs in C#. Yes, there are plenty of conversions avail- able, but you can’t pretend that data for one particular type of object is actually data for a different type. You can try by adding a cast to give the compiler this extra (and incorrect) information, but if the compiler spots that it’s actually impossible for that cast to work, it’ll trigger a compilation error—and if it’s theoretically allowed but actu- ally incorrect at execution time, the CLR will throw an exception. Now that you know a little about how C# 1 fits into the bigger picture of type sys- tems, I’d like to mention a few downsides of its choices. That’s not to say the choices are wrong—they’re just limiting in some ways. Often language designers have to choose between different paths that add different limitations or have other undesir- able consequences. I’ll start with the case where you want to give the compiler more information, but there’s no way of doing so. 2.2.2 When is C# 1’s type system not rich enough? There are two common situations where you might want to expose more information to the caller of a method, or perhaps force the caller to limit what it provides in its Listing 2.2 Demonstrating a type-unsafe system with C code 42 CHAPTER 2 Core foundations: building on C# 1 arguments. The first involves collections, and the second involves inheritance and overriding methods or implementing interfaces. We’ll examine each in turn. COLLECTIONS, STRONG AND WEAK Having avoided the terms strong and weak for the C# type system in general, I’ll use them when talking about collections. The terms are used almost everywhere in this context, with little room for ambiguity. Broadly speaking, three kinds of collection types are built into .NET 1.1:  Arrays—strongly typed—in both the language and the runtime  Weakly typed collections in the System.Collections namespace  Strongly typed collections in the System.Collections.Specialized namespace Arrays are strongly typed,10 so at compile time you can’t set an element of a string[] to be a FileStream, for instance. But reference type arrays also support covariance, which provides an implicit conversion from one type of array to another, as long as there’s a conversion between the element types. Checks occur at execution time to make sure that the wrong type of reference isn’t being stored, as shown in the follow- ing listing. string[] strings = new string[5]; object[] objects = strings; objects[0] = new Button(); If you run listing 2.3, you’ll see that an ArrayTypeMismatchException is thrown C. This is because the conversion from string[] to object[] B returns the original ref- erence—both strings and objects refer to the same array. The array itself knows it’s a string array and will reject attempts to store references to nonstrings. Array covari- ance is occasionally useful, but it comes at the cost of implementing some of the type safety at execution time instead of compile time. Let’s compare this with the situation that weakly typed collections, such as Array- List and Hashtable, put you in. The API of these collections uses object as the type of keys and values. When you write a method that takes an ArrayList, for example, there’s no way of making sure at compile time that the caller will pass in a list of strings. You can document it, and the type safety of the runtime will enforce it if you cast each element of the list to string, but you don’t get compile-time type safety. Likewise, if you return an ArrayList, you can indicate in the documentation that it’ll just contain strings, but callers will have to trust that you’re telling the truth, and they’ll have to insert casts when they access the elements of the list. 10 At least, the language allows them to be. You can use the Array type for weakly typed access to arrays, though. Listing 2.3 Demonstration of array covariance and execution-time checking Applies covariant conversionB Attempts to store a ButtonC 43Type system characteristics Finally, consider strongly typed collections, such as StringCollection. These pro- vide a strongly typed API, so you can be confident that when you receive a String- Collection as a parameter or return value, it’ll only contain strings, and you don’t need to cast when fetching elements of the collection. It sounds ideal, but there are two problems. First, it implements IList, so you can still try to add nonstrings to it (although you’ll fail at execution time). Second, it only deals with strings. There are other specialized collections, but all told they don’t cover much ground. There’s the CollectionBase type, which can be used to build your own strongly typed collec- tions, but that means creating a new collection type for each element type, which is also not ideal. Now that you’ve seen the problem with collections, let’s consider the issue that can occur when you’re overriding methods and implementing interfaces. It’s related to the idea of covariance, which we’ve already seen with arrays. LACK OF COVARIANT RETURN TYPES ICloneable is one of the simplest interfaces in the framework. It has a single method, Clone, which should return a copy of the object that the method is called on. Now, leaving aside the issue of whether this should be a deep or shallow copy, let’s look at the signature of the Clone method: object Clone() It’s a straightforward signature, certainly—but as I said, the method should return a copy of the object it’s called on. That means it needs to return an object of the same type, or at least a compatible one (where that meaning will vary depending on the type). It would make sense to be able to override the method with a signature that gives a more accurate description of what the method actually returns. For example, in a Person class it’d be nice to be able to implement ICloneable with public Person Clone() That wouldn’t break anything—code expecting any old object would still work fine. This feature is called return type covariance but, unfortunately, interface implementa- tion and method overriding don’t support it. Instead, the normal workaround for interfaces is to use explicit interface implementation to achieve the desired effect: public Person Clone() { [Implementation goes here] } object ICloneable.Clone() { return Clone(); } Any code that calls Clone() on an expression with a static type of Person will call the top method; if the type of the expression is ICloneable, it’ll call the bottom method. This works, but it’s really ugly. The mirror image of this situation also occurs with Implements interface explicitly Calls noninterface method 44 CHAPTER 2 Core foundations: building on C# 1 parameters, where if you had an interface or virtual method with a signature of, say, void Process(string x), it’d seem logical to be able to implement or override the method with a less demanding signature, such as void Process(object x). This is called parameter type contravariance; it’s just as unsupported as return type covariance, and you have to use the same workaround for interfaces and normal overloading for virtual methods. It’s not a showstopper, but it’s irritating. Of course, C# 1 developers put up with all of these issues for a long time, and Java developers had a similar situation for far longer. Although compile-time type safety is a great feature in general, I can’t remember seeing many bugs where people actually put the wrong type of element in a collection. I can live with the workaround for the lack of covariance and contravariance. But there’s such a thing as elegance and mak- ing your code clearly express what you mean, preferably without needing explanatory comments. Even if bugs don’t strike, enforcing the documented contract that a collec- tion must only contain strings (for example) can be expensive and fragile in the face of mutable collections. This is the sort of contract you really want the type system itself to enforce. You’ll see later that C# 2 isn’t flawless either, but it makes large improvements. There are more changes in C# 4, but even so, return type covariance and parameter contravariance are missing.11 2.2.3 Summary of type system characteristics In this section, you’ve learned some of the differences between type systems, and in particular which characteristics apply to C# 1:  C# 1 is statically typed—the compiler knows what members to let you use.  C# 1 is explicit—you have to state the type of every variable.  C# 1 is safe—you can’t treat one type as if it were another unless there’s a genu- ine conversion available.  Static typing doesn’t allow a single collection to be a strongly typed list of strings or list of integers without a lot of code duplication for different element types.  Method overriding and interface implementation don’t allow covariance or contravariance. The next section covers one of the most fundamental aspects of C#’s type system beyond its high-level characteristics—the differences between structs and classes. 2.3 Value types and reference types It would be hard to overstate how important the subject of this section is. Everything you do in .NET will deal with either a value type or a reference type, and yet it’s curi- ously possible to develop for a long time with only a vague idea of what the difference is. Worse yet, there are plenty of myths to confuse things further. The unfortunate fact is that it’s easy to make a short but incorrect statement that’s close enough to the truth 11 C# 4 introduced limited generic covariance and contravariance, but that’s not quite the same thing. 45Value types and reference types to be plausible but inaccurate enough to be misleading—but it’s relatively tricky to come up with a concise but accurate description. This section isn’t a complete breakdown of how types are handled, marshaling between application domains, interoperability with native code, and the like. Instead, it’s a brief look at the absolute basics of the topic (as applied to C# 1) that are crucial in order to come to grips with later versions of C#. We’ll start off by seeing how the fundamental differences between value types and reference types appear naturally in the real world, as well as in .NET. 2.3.1 Values and references in the real world Suppose you’re reading something fantastic, and you want a friend to read it too. Let’s further suppose that it’s a document in the public domain, just to avoid any accusa- tions of supporting copyright violation. What do you need to give your friend so that he can read it too? It depends entirely on what you’re reading. First we’ll deal with the case where you have real paper in your hands. To give your friend a copy, you’d need to photocopy all the pages and then give it to him. At that point, he has his own complete copy of the document. In this situation, you’re dealing with value type behavior. All the information is directly in your hands—you don’t need to go anywhere else to get it. Your copy of the information is also independent of your friend’s after you’ve made the copy. You could add some notes to your pages, and his pages wouldn’t be changed at all. Compare that with the situation where you’re reading a web page. This time, all you have to give your friend is the URL of the web page. This is reference type behavior, with the URL taking the place of the reference. In order to read the document, you have to navigate the reference by putting the URL in your browser and asking it to load the page. If the web page changes for some reason (imagine it’s a wiki page and you’ve added your notes to the page), both you and your friend will see that change the next time each of you loads the page. These differences in the real world illustrate the heart of the distinction between value types and reference types in C# and .NET. Most types in .NET are reference types, and you’re likely to create far more reference than value types. The most common cases are classes (declared using class), which are reference types, and structures (declared using struct), which are value types. The other situations are as follows:  Array types are reference types, even if the element type is a value type (so int[] is still a reference type, even though int is a value type).  Enumerations (declared using enum) are value types.  Delegate types (declared using delegate) are reference types.  Interface types (declared using interface) are reference types, but they can be implemented by value types. Now that you have a basic idea of what reference types and value types are about, we’ll look at a few of the most important details. 46 CHAPTER 2 Core foundations: building on C# 1 2.3.2 Value and reference type fundamentals The key concept to grasp when it comes to value types and reference types is what the value of a particular expression is. To keep things concrete, I’ll use variables as the most common examples of expressions, but the same thing applies to properties, method calls, indexers, and other expressions. As we discussed in section 2.2.1, most expressions have a static type associated with them. The value of a value type expression is the value, plain and simple. For instance, the value of the expression "2 + 3" is 5. The value of a reference type expression, though, is a reference—it’s not the object that the reference refers to. The value of the expres- sion String.Empty is not an empty string—it’s a reference to an empty string. In every- day discussions and even in documentation, we tend to blur this distinction. For instance, I might describe String.Concat as returning “a string that’s the concatena- tion of all the parameters.” Using precise terminology here would be time consuming and distracting, and there’s no problem as long as everyone involved understands that only a reference is returned. To demonstrate this further, consider a Point type that stores two integers, x and y. It could have a constructor that takes the two values. This type could be implemented as either a struct or a class. Figure 2.3 shows the result of executing the following lines of code: Point p1 = new Point(10, 20); Point p2 = p1; The left side of figure 2.3 indicates the values involved when Point is a class (a refer- ence type), and the right side shows the situation when Point is a struct (a value type). In both cases, p1 and p2 have the same value after the assignment. But in the case where Point is a reference type, that value is a reference: both p1 and p2 refer to the same object. When Point is a value type, the value of p1 is the whole of the data for a point—the x and y values. Assigning the value of p1 to p2 copies all of that data. The values of variables are stored wherever they’re declared. Local variable values are always stored on the stack,12 and instance variable values are always stored 12 This is only totally true for C# 1. You’ll see later that local variables can end up on the heap in certain situa- tions in later versions. ref p1 10p2 ref 20 y When Point is a reference type When Point is a value type p1 p2 x 10 20 yx 10 20 yx Figure 2.3 Comparing value type and reference type behaviors, particularly with regard to assignment 47Value types and reference types wherever the instance itself is stored. Reference type instances (objects) are always stored on the heap, as are static variables. Another difference between the two kinds of type is that value types can’t be derived from. One consequence of this is that the value doesn’t need any extra infor- mation about what type that value actually is. Compare that with reference types, where each object contains a block of data at the start identifying the type of the object, along with some other information. You can never change the type of an object—when you perform a simple cast, the runtime just takes a reference, checks whether the object it refers to is a valid object of the desired type, and returns the ref- erence if it’s valid or throws an exception otherwise. The reference itself doesn’t know the type of the object, so the same reference value can be used for multiple variables of different types. For instance, consider the following code: Stream stream = new MemoryStream(); MemoryStream memoryStream = (MemoryStream) stream; The first line creates a new MemoryStream object and sets the value of the stream variable to be a reference to that new object. The second line checks whether the value of stream refers to a MemoryStream (or derived type) object and sets the value of memoryStream to be the same as stream. Once you understand these basic points, you can apply them when thinking about some of the falsehoods that are often stated about value types and reference types. 2.3.3 Dispelling myths Various myths do the rounds on a regular basis. I’m sure the misinformation is almost always passed on with no malice and with no idea of the inaccuracies involved, but it’s unhelpful nonetheless. In this section, I’ll tackle the most prominent myths, explain- ing the true situation as I go. MYTH #1: STRUCTS ARE LIGHTWEIGHT CLASSES This myth comes in a variety of forms. Some people believe that value types can’t or shouldn’t have methods or other significant behavior—they should be used as simple data transfer types, with just public fields or simple properties. The DateTime type is a good counterexample to this: it makes sense for it to be a value type, in terms of being a fundamental unit like a number or a character, and it also makes sense for it to be able to perform calculations based on its value. Looking at things from the other direction, data transfer types should often be reference types anyway—the decision should be based on the desired value or reference type semantics, not the simplicity of the type. Other people believe that value types are “lighter” than reference types in terms of performance. The truth is that in some cases value types are more performant— they don’t require garbage collection unless they’re boxed, don’t have the type identification overhead, and don’t require dereferencing, for example. But in other ways, reference types are more performant—parameter passing, assigning values to variables, returning values, and similar operations only require 4 or 8 bytes to be 48 CHAPTER 2 Core foundations: building on C# 1 copied (depending on whether you’re running the 32-bit or 64-bit CLR) rather than copying all the data. Imagine if ArrayList were somehow a “pure” value type, and passing an ArrayList expression to a method involved copying all its data! In almost all cases, performance isn’t really determined by this sort of decision anyway. Bottle- necks are almost never where you think they’ll be, and before you make a design deci- sion based on performance, you should measure the different options. It’s worth noting that the combination of the two beliefs doesn’t work either. It doesn’t matter how many methods a type has (whether it’s a class or a struct)—the memory taken per instance isn’t affected. (There’s a cost in terms of the memory taken up for the code itself, but that’s incurred once rather than for each instance.) MYTH #2: REFERENCE TYPES LIVE ON THE HEAP; VALUE TYPES LIVE ON THE STACK This one is often caused by laziness on the part of the person repeating it. The first part is correct—an instance of a reference type is always created on the heap. It’s the second part that causes problems. As I’ve already noted, a variable’s value lives wher- ever it’s declared, so if you have a class with an instance variable of type int, that vari- able’s value for any given object will always be where the rest of the data for the object is—on the heap. Only local variables (variables declared within methods) and method parameters live on the stack. In C# 2 and later, even some local variables don’t really live on the stack, as you’ll see when we look at anonymous methods in chapter 5. ARE THESE CONCEPTS RELEVANT NOW? It’s arguable that if you’re writing man- aged code, you should let the runtime worry about how memory is best used. Indeed, the language specification makes no guarantees about what lives where; a future runtime may be able to create some objects on the stack if it knows it can get away with it, or the C# compiler could generate code that hardly uses the stack at all. The next myth is usually just a terminology issue. MYTH #3: OBJECTS ARE PASSED BY REFERENCE IN C# BY DEFAULT This is probably the most widely propagated myth. Again, the people who make this claim often (though not always) know how C# actually behaves, but they don’t know what “pass by reference” really means. Unfortunately, this is confusing for people who do know what it means. The formal definition of pass by reference is relatively complicated, involving l-values and similar computer-science terminology, but the important thing is that if you pass a variable by reference, the method you’re calling can change the value of the caller’s vari- able by changing its parameter value. Now, remember that the value of a reference type variable is the reference, not the object itself. You can change the contents of the object that a parameter refers to without the parameter itself being passed by refer- ence. For instance, the following method changes the contents of the StringBuilder object in question, but the caller’s expression will still refer to the same object as before: 49Value types and reference types void AppendHello(StringBuilder builder) { builder.Append("hello"); } When this method is called, the parameter value (a reference to a StringBuilder) is passed by value. If you were to change the value of the builder variable within the method—for example, with the statement builder = null;—that change wouldn’t be seen by the caller, contrary to the myth. It’s interesting to note that not only is the “by reference” bit of the myth inaccu- rate, but so is the “objects are passed” bit. Objects themselves are never passed, either by reference or by value. When a reference type is involved, either the variable is passed by reference or the value of the argument (the reference) is passed by value. Aside from anything else, this answers the question of what happens when null is used as a by-value argument—if objects were being passed around, that would cause issues, as there wouldn’t be an object to pass! Instead, the null reference is passed by value in the same way as any other reference would be. If this quick explanation has left you bewildered, you might want to look at my arti- cle, “Parameter passing in C#,” (http://mng.bz/otVt), which goes into much more detail. These myths aren’t the only ones around. Boxing and unboxing come in for their fair share of misunderstanding, which I’ll try to clear up next. 2.3.4 Boxing and unboxing Sometimes, you just don’t want a value type value. You want a reference. There are var- ious reasons why this can happen, and fortunately C# and .NET provide a mechanism called boxing that lets you create an object from a value type value and use a reference to that new object. Before we leap into an example, let’s start off by reviewing two important facts:  The value of a reference type variable is always a reference.  The value of a value type variable is always a value of that type. Given those two facts, the following three lines of code don’t seem to make much sense at first glance: int i = 5; object o = i; int j = (int) o; You have two variables: i is a value type variable, and o is a reference type variable. How does it make sense to assign the value of i to o? The value of o has to be a refer- ence, and the number 5 isn’t a reference—it’s an integer value. What’s actually hap- pening is boxing: the runtime creates an object (on the heap—it’s a normal object) that contains the value (5). The value of o is then a reference to that new object. The value in the object is a copy of the original value—changing the value of i won’t change the value in the box at all. 50 CHAPTER 2 Core foundations: building on C# 1 The third line performs the reverse operation—unboxing. You have to tell the com- piler which type to unbox the object as, and if you use the wrong type (if it’s a boxed uint or long, for example, or not a boxed value at all), an InvalidCastException is thrown. Again, unboxing copies the value that was in the box; after the assignment, there’s no further association between j and the object. That’s boxing and unboxing in a nutshell. The only remaining problem is knowing when boxing and unboxing occur. Unboxing is usually obvious, because the cast is pres- ent in the code. Boxing can be more subtle. You’ve seen the simple version, but it can also occur if you call the ToString, Equals, or GetHashCode methods on the value of a type that doesn’t override them,13 or if you use the value as an interface expression— assigning it to a variable whose type is an interface type or passing it as the value for a parameter with an interface type. For example, the statement IComparable x = 5; would box the number 5. It’s worth being aware of boxing and unboxing because of the potential perfor- mance penalty involved. A single box or unbox operation is cheap, but if you perform hundreds of thousands of them, you not only have the cost of the operations, but you’re also creating a lot of objects, which gives the garbage collector more work to do. This performance hit isn’t usually an issue, but it’s worth being aware of so you can measure the effect if you’re concerned. 2.3.5 Summary of value types and reference types In this section, we’ve looked at the differences between value types and reference types and at some of the myths surrounding them. Here are the key points:  The value of a reference type expression (a variable, for example) is a refer- ence, not an object.  References are like URLs—they’re small pieces of data that let you access the real information.  The value of a value type expression is the actual data.  There are times when value types are more efficient than reference types, and vice versa.  Reference type objects are always on the heap, but value type values can be on either the stack or the heap, depending on context.  When a reference type is used as a method parameter, by default the argument is passed by value, but the value itself is a reference.  Value type values are boxed when reference type behavior is needed; unboxing is the reverse process. Now that we’ve had a look at all the bits of C# 1 that you need to be comfortable with, it’s time to take a quick look forward and see where each of the features are enhanced by the later versions of C#. 13 Boxing will always occur when you call GetType() on a value type variable, because it can’t be overridden. You should already know the exact type if you’re dealing with the unboxed form, so you can just use typeof instead. 51Beyond C# 1: new features on a solid base 2.4 Beyond C# 1: new features on a solid base The three topics covered in this chapter are vital to all versions of C#. Almost all the new features relate to at least one of them, and they change the balance of how the language is used. Before we wrap up the chapter, let’s explore how the new features relate to the old ones. I won’t give many details (for some reason the publisher didn’t want a 600-page section), but it’s helpful to have an idea of where we’re going before we get to the nitty-gritty. We’ll look at them in the same order as we covered them ear- lier, starting with delegates. 2.4.1 Features related to delegates Delegates of all kinds get a boost in C# 2, and then they’re given even more special treatment in C# 3. Most of the features aren’t new to the CLR but are clever compiler tricks to make delegates work more smoothly within the language. The changes affect not just the syntax you can use, but the appearance and feeling of idiomatic C# code. Over time, C# is gaining a more functional approach. C# 1 has pretty clumsy syntax when it comes to creating a delegate instance. For one thing, even if you need to accomplish something straightforward, you have to write a whole separate method to create a delegate instance for it. C# 2 fixed this with anonymous methods and introduced a simpler syntax for the cases where you still want to use a normal method to provide the action for the delegate. You can also cre- ate delegate instances using methods with compatible signatures—the method signa- ture no longer has to be exactly the same as the delegate’s declaration. The following listing demonstrates all these improvements. static void HandleDemoEvent(object sender, EventArgs e) { Console.WriteLine ("Handled by HandleDemoEvent"); } ... EventHandler handler; handler = new EventHandler(HandleDemoEvent); handler(null, EventArgs.Empty); handler = HandleDemoEvent; handler(null, EventArgs.Empty); handler = delegate(object sender, EventArgs e) { Console.WriteLine ("Handled anonymously"); }; handler(null, EventArgs.Empty); handler = delegate { Console.WriteLine ("Handled anonymously again"); }; handler(null, EventArgs.Empty); Listing 2.4 Improvements in delegate instantiation brought in by C# 2 Specifies delegate type and method B Implicitly converts to delegate instance C Specifies action with anonymous method D Uses anonymous method shortcut E 52 CHAPTER 2 Core foundations: building on C# 1 MouseEventHandler mouseHandler = HandleDemoEvent; mouseHandler(null, new MouseEventArgs(MouseButtons.None, 0, 0, 0, 0)); The first part of the main code B is just C# 1 code, kept for comparison. The remaining delegates all use new features of C# 2. Method group conversions C make event subscription code read a lot more pleasantly—lines such as saveButton.Click += SaveDocument; are straightforward, with no extra fluff to distract the eye. The anonymous method syntax D is a little cumbersome, but it does allow the action to be clear at the point of creation, rather than being another method to look at before you understand what’s going on. A shortcut is available when using anonymous meth- ods E, but this form can only be used when you don’t need the parameters. Anony- mous methods have other powerful features as well, but we’ll see those later. The final delegate instance created F is an instance of MouseEventHandler rather than just EventHandler, but the HandleDemoEvent method can still be used due to con- travariance, which specifies parameter compatibility. Covariance specifies return type compatibility. We’ll look at both of these in more detail in chapter 5. Event handlers are probably the biggest beneficiaries of this, because suddenly the Microsoft guide- line to make all delegate types used in events follow the same convention makes a lot more sense. In C# 1, it didn’t matter whether two different event handlers looked quite similar—you had to have a method with an exactly matching signature in order to create a delegate instance. In C# 2, you may find yourself able to use the same method to handle many different kinds of events, particularly if the purpose of the method is fairly event independent, such as logging. C# 3 provides special syntax for instantiating delegate types, using lambda expres- sions. To demonstrate these, we’ll use a new delegate type. When the CLR gained generics in .NET 2.0, generic delegate types became available and were used in a num- ber of API calls in generic collections. .NET 3.5 takes things a step further, introducing a group of generic delegate types called Func that all take parameters of specified types and return a value of another specified type. The following listing shows the use of a Func delegate type as well as lambda expressions. Func func = (x, y) => (x * y).ToString(); Console.WriteLine(func(5, 20)); Func is a delegate type that takes two integers and returns a string. The lambda expression in listing 2.5 specifies that the delegate instance (held in func) should multiply the two integers together and call ToString(). The syntax is much more straightforward than that of anonymous methods, and there are other benefits in terms of the amount of type inference the compiler is prepared to perform for you. Lambda expressions are absolutely crucial to LINQ, and you should get ready to make them a core part of your language toolkit. They’re not restricted to working Listing 2.5 Lambda expressions—like improved anonymous methods Uses delegate contravariance F 53Beyond C# 1: new features on a solid base with LINQ, though—any use of anonymous methods from C# 2 can use lambda expressions in C# 3, and that will almost always lead to shorter code. To summarize, the new features related to delegates are as follows:  Generics (generic delegate types)—C# 2  Delegate instance creation expressions—C# 2  Anonymous methods—C# 2  Delegate covariance/contravariance—C# 2  Lambda expression—C# 3 Additionally, C# 4 allows generic covariance and contravariance for delegates, which goes beyond what you’ve just seen. Indeed, generics form one of the principal enhancements to the type system, which we’ll look at next. 2.4.2 Features related to the type system The primary new feature in C# 2 regarding the type system is the inclusion of gener- ics. It largely addresses the issues I raised in section 2.2.2 about strongly typed collec- tions, although generic types are useful in a number of other situations too. As a feature, it’s elegant, it solves a real problem, and despite a few wrinkles it generally works well. You’ve seen examples of this in quite a few places already, and it’s described fully in the next chapter, so I won’t go into any more detail here. Generics form probably the most important feature in C# 2 with respect to the type system, and you’ll see generic types throughout the rest of the book. C# 2 doesn’t tackle the issues of return type covariance and parameter contravari- ance for overriding members or implementing interfaces. But it does improve the situ- ation for creating delegate instances in certain situations, as you saw in section 2.4.1. C# 3 introduced a wealth of new concepts in the type system, most notably anony- mous types, implicitly typed local variables, and extension methods. Anonymous types them- selves are mostly present for the sake of LINQ, where it’s useful to be able to effectively create a data transfer type with a bunch of read-only properties without having to actu- ally write the code for them. There’s nothing to stop them from being used outside LINQ, though, which makes life easier for demonstrations. Listing 2.6 shows both fea- tures in action. var jon = new { Name = "Jon", Age = 31 }; var tom = new { Name = "Tom", Age = 4 }; Console.WriteLine ("{0} is {1}", jon.Name, jon.Age); Console.WriteLine ("{0} is {1}", tom.Name, tom.Age); The first two lines each show implicit typing (the use of var) and anonymous object initializers (the new {...} bit), which create instances of anonymous types. There are two things worth noting at this stage, long before we get into the details—points that have caused people to worry needlessly before. The first is that C# 3 is still statically typed. The C# compiler has declared jon and tom to be of a particular Listing 2.6 Demonstration of anonymous types and implicit typing 54 CHAPTER 2 Core foundations: building on C# 1 type, just as normal, and when you use the properties of the objects, they’re normal properties—no dynamic lookup is going on. It’s just that you (as a source code author) can’t tell the compiler what type to use in the variable declaration because the com- piler will be generating the type itself. The properties are also statically typed—here the Age property is of type int, and the Name property is of type string. The second point is that we haven’t created two different anonymous types here. The variables jon and tom both have the same type because the compiler uses the prop- erty names, types, and order to work out that it can generate just one type and use it for both statements. This is done on a per-assembly basis, and makes life a lot simpler in terms of being able to assign the value of one variable to another (for example, jon = tom; would be permitted in the previous code) and similar operations. Extension methods are also there for the sake of LINQ but can be useful outside it. Think of all the times you’ve wished that a framework type had a certain method, and you’ve had to write a static utility method to implement it. For instance, to create a new string by reversing an existing one, you might write a static StringUtil.Reverse method. Well, the extension method feature effectively lets you call that static method as if it existed on the string type itself, so you could write string x = "dlrow olleH".Reverse(); Extension methods also let you appear to add methods with implementations to interfaces, and LINQ relies on this heavily, allowing calls to all kinds of methods on IEnumerable that have never previously existed. C# 4 has two features related to the type system. A relatively minor feature is cova- riance and contravariance for generic delegates and interfaces. This has been present in the CLR since .NET 2.0 came out, but only with the introduction of C# 4 and updates to the generic types in the Base Class Library (BCL) has it become usable for C# developers. A far bigger feature—although one many coders may never need—is dynamic typing in C#. Remember the introduction I gave to static typing, where I tried to use the Length property of an array and a string via the same variable? Well, in C# 4 it works—when you want it to. The following listing shows the same code except for the variable decla- ration, but working as valid C# 4 code. dynamic o = "hello"; Console.WriteLine(o.Length); o = new string[] {"hi", "there"}; Console.WriteLine(o.Length); By declaring the variable o as having a static type of dynamic (yes, you read that right), the compiler handles almost everything to do with o differently, leaving all the bind- ing decisions (such as what Length means) until execution time. Obviously we’re going to look at dynamic typing in greater depth, but I want to stress now that C# 4 is still a statically typed language for the most part. Unless you’re Listing 2.7 Dynamic typing in C# 4 55Beyond C# 1: new features on a solid base using the dynamic type (which acts as a static type denoting a dynamic value), every- thing works exactly the same way as before. Most C# developers will only rarely need dynamic typing, and for the rest of the time they can ignore it. When dynamic typing is handy, it can be really slick—and it lets you play nicely with code written in dynamic languages running on the Dynamic Language Runtime (DLR). I’d just advise you not to start using C# as a primarily dynamic language. If that’s what you want, use Iron- Python or something similar; languages that are designed to support dynamic typing from the ground up are likely to have fewer unexpected gotchas. Here’s the quick-view list of these features, along with which version of C# they’re introduced in:  Generics—C# 2  Limited delegate covariance/contravariance—C# 2  Anonymous types—C# 3  Implicit typing—C# 3  Extension methods—C# 3  Limited generic covariance/contravariance—C# 4  Dynamic typing—C# 4 After that fairly diverse set of features on the type system, let’s look at the features added to one specific part of typing in .NET—value types. 2.4.3 Features related to value types There are only two features to talk about here, both introduced in C# 2. The first goes back to generics yet again, and in particular to collections. One common complaint about using value types in collections with .NET 1.1 was that due to all of the general- purpose APIs being specified in terms of the object type, every operation that added a struct value to a collection would involve boxing it, and you’d have to unbox it when retrieving it. While boxing is pretty cheap for an individual call, it can cause a signifi- cant performance hit if it’s used every time with frequently accessed collections. It also takes more memory than it needs to, due to the per-object overhead. Generics fix both the speed and memory deficiencies by using the real type involved rather than a general-purpose object. For example, it would’ve been madness to read a file and store each byte as an element in an ArrayList in .NET 1.1, but in .NET 2.0 it wouldn’t be crazy to do the same with a List. The second feature addresses another common cause of complaint, particularly when talking to databases—the fact that you can’t assign null to a value type variable. There’s no such concept as an int value of null, for instance, even though a database integer field may well be nullable. That makes it hard to model the database table within a statically typed class without ugliness of some form or another. Nullable types are part of .NET 2.0, and C# 2 includes extra syntax to make them easy to use. The fol- lowing listing gives a brief example of this. 56 CHAPTER 2 Core foundations: building on C# 1 int? x = null; x = 5; if (x != null) { int y = x.Value; Console.WriteLine (y); } int z = x ?? 10; Listing 2.8 shows a number of the features of nullable types and the shorthand that C# provides for working with them. We’ll get around to the details of each feature in chapter 4, but the important point here is how much easier and cleaner all of this is than any of the workarounds used in the past. The list of enhancements is smaller this time, but they’re important features in terms of both performance and elegance of expression:  Generics—C# 2  Nullable types—C# 2 2.5 Summary This chapter has mostly been a revision exercise for C# 1. The aim wasn’t to cover any one topic in its entirety, but merely to get everyone on the same page so that I can describe the later features without worrying about the ground that I’m building on. All of the topics we’ve covered are core to C# and .NET, but I’ve seen a lot of mis- understandings around them within community discussions. Although this chapter hasn’t gone into much depth about any one point, it’ll hopefully have cleared up any confusion that would’ve made the rest of the book harder to understand. The three core topics we briefly covered in this chapter have all been significantly enhanced since C# 1, and some features touch on more than one topic. In particular, the addition of generics has an impact on almost every area we’ve covered in this chapter—it’s probably the most widely used and important feature in C# 2. Now that we’ve finished all our preparations, we can start looking at generics properly in the next chapter. Listing 2.8 Demonstration of a variety of nullable type features Declares, sets nullable variable Tests for presence of real value Obtains real value Uses null-coalescing operator Part 2 C# 2: Solving the issues of C# 1 In part 1 we took a quick look at a few of the features of C# 2. Now it’s time to do the job properly. You’ll see how C# 2 fixes various problems that developers ran into when using C# 1, and how C# 2 makes existing features more useful by streamlining them. This is no mean feat, and life with C# 2 is much more pleas- ant than with C# 1. The new features in C# 2 have a certain amount of independence. That’s not to say they’re not related at all; many of the features are based on—or at least interact with—the massive contribution that generics make to the language. But the different topics we’ll look at in the next five chapters don’t combine into one super-feature. The first four chapters of this part cover the biggest new features. We’ll look at the following:  Generics—The most important new feature in C# 2 (and indeed in the CLR for .NET 2.0), generics allow type and method parameterization in terms of the types they interact with.  Nullable types—Value types such as int and DateTime don’t have any con- cept of “no value present”; nullable types allow you to represent the absence of a meaningful value.  Delegates—Although delegates haven’t changed at the CLR level, C# 2 makes them a lot easier to work with. In addition to a few simple shortcuts, the introduction of anonymous methods begins the movement toward a more functional style of programming—a trend that continues in C# 3.  Iterators—Although using iterators has always been simple in C# with the foreach statement, it’s a pain to implement them in C# 1. The C# 2 compiler is happy to build a state machine for you behind the scenes, hiding a lot of the complexity involved. Once we’ve covered the major, complex new features of C# 2 with a chapter dedicated to each one, chapter 7 rounds off the coverage by introducing several simpler fea- tures. Simpler doesn’t necessarily mean less useful; partial types, in particular, are cru- cial for better designer support in versions of Visual Studio from 2005 onward. The same feature is beneficial for other generated code, too. Likewise, many C# develop- ers take the ability to write a property with a public getter and a private setter for granted these days, but it was only introduced in C# 2. When the first edition of this book was published, many developers still hadn’t used C# 2 at all. My impression in 2013 is that it’s rare to find someone who’s currently using C#, but who hasn’t at least dabbled with C# 2, probably 3, and quite often 4. The topics covered here are fundamental to how later versions of C# work; in particular, attempting to learn about LINQ without understanding generics and iterators would be tricky. The chapter on iterators is also related to C# 5’s asynchronous methods; the two features are very different on the face of it, but both involve state machines built by the compiler to change the conventional flow of execution. If you’ve been using C# 2 and upward for a while, you may find a lot of this part covers familiar ground, but I suspect you’ll still benefit from a deeper knowledge of the details presented. 59 Parameterized typing with generics True story:1 The other day my wife and I went out to do our weekly grocery shop- ping. Just before we left, she asked me if I had the list. I confirmed that I did have the list, and off we went. It was only when we got to the grocery store that our mis- take became obvious. My wife had been asking about the shopping list, whereas I’d brought the list of neat features in C# 2. When we asked an assistant whether we could buy any anonymous methods, we received a strange look. This chapter covers  Type inference for generic methods  Type constraints  Reflection and generics  CLR behavior  Limitations of generics  Comparisons with other languages 1 By which I mean “convenient for the purposes of introducing the chapter”—not necessarily accurate. 60 CHAPTER 3 Parameterized typing with generics If only we could’ve expressed ourselves more clearly! If only she’d had some way of saying that she wanted me to bring the list of items we wanted to buy! If only we’d had generics… For most developers, generics are the most important new feature of C# 2. They enhance performance, make your code more expressive, and move a lot of safety checks from execution time to compile time. Essentially, they allow you to parameterize types and methods. Just as normal method calls often have parameters to tell them what values to use, generic types and methods have type parameters to tell them what types to use. It all sounds confusing to start with—and if you’re completely new to generics, you can expect a certain amount of head scratching—but once you get the basic idea, you’ll come to love them. In this chapter, we’ll look at how to use generic types and methods that others have provided (whether in the framework or as third-party libraries), and how to write your own. Along the way, we’ll look at how generics work with the reflection calls in the API, and at a bit of detail around how the CLR handles generics. To conclude the chapter, I’ll present some of the most frequently encountered limitations of generics, along with possible workarounds, and compare generics in C# with similar features in other languages. First, though, you need to understand the problems that led to generics being devised in the first place. 3.1 Why generics are necessary If you still have any C# 1 code available, look at it and count the casts—particularly in code that uses collections extensively. Don’t forget that almost every use of foreach contains an implicit cast. When you use types that are designed to work with many dif- ferent types of data, that naturally leads to casting, quietly telling the compiler not to worry, that everything’s fine; just treat the expression over there as if it had this partic- ular type. Using almost any API that has object as either a parameter type or a return type will probably involve casts at some point. Having a single-class hierarchy with object as the root makes some things more straightforward, but the object type in itself is extremely dull, and to do anything genuinely useful with an object you almost always need to cast it. Casts are bad, m’kay? Not bad in an almost never do this kind of way (like mutable structs and nonprivate fields) but bad in a necessary evil kind of way. They’re an indica- tion that you ought to give the compiler more information somehow, and that you’re choosing to ask the compiler to trust you at compile time and to generate a check that will run at execution time to keep you honest. If you need to tell the compiler the information somehow, chances are that anyone reading your code is also going to need the same information. They can see it where you’re casting, of course, but that’s not terribly useful. The ideal place to keep such information is usually at the point where you declare a variable or method. This is even more important if you’re providing a type or method that other people will call 61Why generics are necessary without access to your code. Generics allow library providers to prevent their users from compiling code that calls the library with bad arguments. In C# 1, you had to rely on manually written documentation, which can easily become incomplete or inaccurate, as duplicate information so often is. When the extra information can be declared in code as part of a method or type declaration, everyone can work more productively. The compiler can do more checking; the IDE can present IntelliSense options based on the extra information (for instance, offer- ing the members of string as the next step when you access an element within a list of strings); callers of methods can be more confident that arguments passed in and values returned are correct; and anyone maintaining your code can better understand what was running through your head when you originally wrote it. WILL GENERICS REDUCE YOUR BUG COUNT? Every description of generics I’ve read (including my own) emphasizes the importance of compile-time type checking over execution-time type checking. I’ll let you in on a secret: I can’t remember ever fixing a bug in released code that was directly due to the lack of type checking. In other words, the casts we put in C# 1 code always worked, in my experience. Those casts were like warning signs, forcing us to think about the type safety explicitly rather than it flowing naturally in the code we wrote. But although generics may not radically reduce the number of type safety bugs you encounter, the greater readability they afford can reduce the number of bugs across the board. Code that’s simple to understand is simple to get right. Likewise, code that has to be robust in the face of malicious call- ers is much simpler to write correctly when the type system can provide appropriate guarantees. All of this would be enough to make generics worthwhile, but there are performance improvements, too. First, because the compiler can perform more enforcement, that leaves less to be checked at execution time. Second, the JIT can treat value types in a particularly clever way that manages to eliminate boxing and unboxing in many situa- tions. In some cases, this can make a huge difference in performance in terms of both speed and memory consumption. Many of the benefits of generics may strike you as being similar to the benefits of statically typed languages over dynamic ones: better compile-time checking, more information expressed directly in the code, more IDE support, better performance. The reason for this is fairly simple: when you’re using a general API (such as Array- List) that can’t differentiate between the different types, you effectively are in a dynamic situation in terms of access to that API. The reverse isn’t generally true, by the way—the benefits that dynamic languages provide rarely apply to the choice between generic and nongeneric APIs. When you can reasonably use generics, the decision to do so is usually a no-brainer. So, those are the goodies awaiting you in C# 2—now it’s time to start using generics. 62 CHAPTER 3 Parameterized typing with generics 3.2 Simple generics for everyday use The topic of generics has a lot of dark corners if you want to know everything about it. The C# language specification goes into a great deal of detail in order to make sure that the behavior is specified in pretty much every conceivable case. But you don’t need to understand most of those corner cases in order to be productive. (The same is true in other areas, in fact. For example, you don’t need to know all the exact rules about definite assignment—you just fix the code appropriately when the compiler complains.) This section will cover most of what you’ll need in your day-to-day use of generics, both for consuming generic APIs that other people have created and for creating your own. If you get stuck while reading this chapter but want to keep making progress, I suggest you concentrate on what you need to know in order to use generic types and methods within the framework and other libraries; writing your own generic types and methods crops up a lot less often than using the framework ones. We’ll start by looking at one of the collection classes introduced in .NET 2.0— Dictionary. 3.2.1 Learning by example: a generic dictionary Using generic types can be straightforward if you don’t happen to hit some of the lim- itations and start wondering what’s wrong. You don’t even need to know any of the ter- minology to make a pretty good guess as to what a piece of code will do, and with a bit of trial and error you could experiment your way to writing your own working code. (One of the benefits of generics is that more checking is done at compile time, so you’re more likely to have working code when it all compiles—this makes the experi- mentation simpler.) Of course, the aim of this chapter is to build your knowledge so that you won’t be using guesswork—you’ll know what’s going on at every stage. For now, let’s look at some code that’s straightforward, even if the syntax is unfa- miliar. The following listing uses a Dictionary (roughly the generic equivalent of the nongeneric Hashtable class) to count the frequencies of words in a given piece of text. static Dictionary CountWords(string text) { Dictionary frequencies; frequencies = new Dictionary(); string[] words = Regex.Split(text, @"\W+"); foreach (string word in words) { if (frequencies.ContainsKey(word)) { frequencies[word]++; } Listing 3.1 Using a Dictionary to count words in text Creates new map from word to frequency B Splits text into words C Adds to or updates map D 63Simple generics for everyday use else { frequencies[word] = 1; } } return frequencies; } ... string text = @"Do you like green eggs and ham? I do not like them, Sam-I-am. I do not like green eggs and ham."; Dictionary frequencies = CountWords(text); foreach (KeyValuePair entry in frequencies) { string word = entry.Key; int frequency = entry.Value; Console.WriteLine ("{0}: {1}", word, frequency); } The CountWords method first creates an empty map from string to int B. This will effectively count how often each word is used within the given text. You then use a reg- ular expression C to split the text into words. It’s crude—you end up with an empty string due to the period at the end of the text, and do and Do are counted separately. These issues are easily fixable, but I wanted to keep the code as simple as possible for this example. For each word, you check whether it’s already in the map. If it is, you increment the existing count; otherwise, you give the word an initial count of 1 D. Note how the incrementing code doesn’t need to do a cast to int in order to perform the addition; the value you retrieve is known to be an int at compile time. The step incrementing the count is actually performing a get on the indexer for the map, then incrementing, and then performing a set on the indexer. You may find it easier to keep this explicit, using frequencies[word] = frequencies[word] + 1; instead. The final part of the listing is familiar: enumerating through a Hashtable gives a similar (nongeneric) DictionaryEntry with Key and Value properties for each entry E. But in C# 1, you would’ve needed to cast both the word and the frequency, because the key and value would’ve been returned as just object. That also means that the frequency would’ve been boxed. Admittedly you don’t have to put the word and the frequency into variables—you could’ve had a single call to Console.Write- Line and passed entry.Key and entry.Value as arguments. I included the variables here to ram home the point that no casting is necessary. Now that you’ve seen an example, let’s look at what it means to talk about Dictionary in the first place. What are TKey and TValue, and why do they have angle brackets around them? Adds to or updates map D Prints each key/value pair from map E 64 CHAPTER 3 Parameterized typing with generics 3.2.2 Generic types and type parameters There are two forms of generics in C#: generic types (including classes, interfaces, dele- gates, and structures—there are no generic enums) and generic methods. Both are essentially ways of expressing an API (whether it’s for a single generic method or a whole generic type) such that in some places where you’d expect to see a normal type, you’ll see a type parameter instead. A type parameter is a placeholder for a real type. Type parameters appear in angle brackets within a generic declaration, using commas to separate them. So in Dictionary, the type parameters are TKey and TValue. When you use a generic type or method, you specify the real types you want to use. These are called the type arguments—for example, in listing 3.1 the type arguments were string (for TKey) and int (for TValue). JARGON ALERT! A lot of detailed terminology is involved in generics. I’ve included it for reference—and because occasionally it makes it easier to talk about topics precisely. It could also be useful if you ever need to consult the language specification, but you’re unlikely to need this terminology in day-to- day life. Just grin and bear it for the moment. A lot of this terminology is defined in section 4.4 of the C# 5 specification (“Constructed Types”)—look there for further details. The form of a generic type where none of the type parameters have been provided with type arguments is called an unbound generic type. When type arguments are specified, the type is said to be a constructed type. Unbound generic types are effectively blueprints for constructed types, much like how types (generic or not) can be regarded as blueprints for objects. It’s a sort of extra layer of abstraction. Figure 3.1 shows this graphically. Instance of Hashtable Instantiation Instance of Dictionary Specification of type arguments (etc.) Instance of Dictionary Nongeneric blueprints Generic blueprints Instantiation Instantiation Dictionary (unbound generic type) Hashtable Dictionary (constructed type) Dictionary (constructed type) Figure 3.1 Unbound generic types act as blueprints for constructed types, which then act as blueprints for actual objects, just as nongeneric types do. 65Simple generics for everyday use As a further complication, types can be open or closed. An open type is one that still involves a type parameter (as one of the type arguments, or as the array element type, for example), whereas a closed type is one that isn’t open; every aspect of the type is known precisely. All code actually executes in the context of a closed constructed type. The only time you’ll see an unbound generic type within C# code (other than as a declaration) is within the typeof operator, which you’ll meet in section 3.4.4. The idea of a type parameter “receiving” information and a type argument “pro- viding” the information—the dashed lines in figure 3.1—is exactly the same as with method parameters and arguments, although type arguments have to be types rather than just arbitrary values. The type argument has to be known at compile time, but it can be (or can involve) a type parameter from the relevant context. You can think of a closed type as having the API of the open type but with the type parameters being replaced with their corresponding type arguments.2 Table 3.1 shows some public method and property declarations from the open type Dictionary and the equivalent member in the closed type you built from it— Dictionary. One important thing to note is that none of the methods in table 3.1 are actually generic methods. They’re normal methods within a generic type, and they happen to use the type parameters declared as part of the type. We’ll look at generic methods in the next section. Now that you know what TKey and TValue mean, and what the angle brackets are for, you can see what the declarations in table 3.1 would look like within the class dec- laration. Here’s what the code for Dictionary might look like, although the actual method implementations are all missing, and there are more members in reality: namespace System.Collections.Generic { public class Dictionary : IEnumerable> 2 It doesn’t always work exactly that way—there are corner cases that break when you apply that simple rule— but it’s an easy way of thinking about generics that works in the vast majority of situations. Table 3.1 Examples of how method signatures in generic types contain placeholders, which are replaced when the type arguments are specified Method signature in generic type Method signature after type parameter substitution void Add(TKey key, TValue value) void Add(string key, int value) TValue this[TKey key] { get; set; } int this[string key] { get; set; } bool ContainsValue(TValue value) bool ContainsValue(int value) bool ContainsKey(TKey key) bool ContainsKey(string key) Declares generic class Implements generic interface 66 CHAPTER 3 Parameterized typing with generics { public Dictionary() {... } public void Add(TKey key, TValue value) {... } public TValue this[TKey key] { get {... } set {... } } public bool ContainsValue(TValue value) {... } public bool ContainsKey(TKey key) {... } [... other members ...] } } Note how Dictionary implements the generic interface IEnumerable > (and many other interfaces in real life). Whatever type arguments you specify for the class are applied to the interface where the same type parameters are used, so in this example, Dictionary implements IEnumerable>. That’s sort of a doubly generic inter- face—it’s the IEnumerable interface with the structure KeyValue- Pair as the type argument. It’s because it implements that interface that listing 3.1 was able to enumerate the keys and values as it did. It’s also worth pointing out that the constructor doesn’t list the type parameters in angle brackets. The type parameters belong to the type rather than to the particular constructor, so that’s where they’re declared. Members only declare type parameters when they’re introducing new ones—and only methods can do that. PRONOUNCING GENERICS If you ever need to describe a generic type to a col- league, it’s conventional to use “of” to introduce the type parameters or argu- ments—so List is pronounced “list of T,” for example. In VB, this is part of the language: the type itself would be written as List(Of T). When there are multiple type parameters, I find it makes sense to separate them with a word appropriate to the meaning of the overall type, so I’d talk about a “dic- tionary of string to int” in order to emphasize the mapping aspect, but a “tuple of string and int.” Generic types can effectively be overloaded on the number of type parameters, so you could define MyType, MyType, MyType, MyType, and so forth, all within the same namespace. The names of the type parameters aren’t used when con- sidering this—just how many there are. These types are unrelated except in name— there’s no default conversion from one to another, for instance. The same is true for generic methods: two methods can be exactly the same in signature other than the number of type parameters. Although this may sound like a recipe for disaster, it can be useful if you want to take advantage of generic type inference where the compiler can work out some of the type arguments for you. We’ll come back to that in section 3.3.2. Declares parameterless constructorDeclares method using type parameters 67Simple generics for everyday use NAMING CONVENTIONS FOR TYPE PARAMETERS Although you could have a type with type parameters T, U, and V, it wouldn’t give much indication of what they actually meant or how they should be used. Compare this with Dictionary , where it’s obvious that TKey represents the type of the keys and TValue represents the type of the values. Where you have a single type parameter and its meaning is clear, T is conventionally used (List is a good example of this). Multiple type parameters should usually be named according to meaning, using the prefix T to indicate a type parameter. Every so often, you may run into a type with multiple single-letter type parameters (SynchronizedKeyedCollection, for example), but you should try to avoid creating the same situation yourself. Now that you have an idea of what generic types do, let’s look at generic methods. 3.2.3 Generic methods and reading generic declarations I’ve mentioned generic methods a few times, but we haven’t actually met one yet. You may find the overall idea of generic methods more confusing than generic types— they’re somehow less natural for the brain—but it’s the same basic principle. You’re used to the parameters and return value of a method having firmly specified types, and you’ve seen how a generic type can use its type parameters in method declara- tions. Generic methods go one step further: even if you know exactly which con- structed type you’re dealing with, an individual method can have type parameters too. Don’t worry if you’re still none the wiser—the concept is likely to click at some point, after you’ve seen enough examples. Dictionary doesn’t have any generic methods, but its close neigh- bor List does. As you can imagine, List is just a list of items of whatever type is specified—List is a list of strings, for instance. Remembering that T is the type parameter for the whole class, let’s dissect a generic method declaration. Figure 3.2 identifies the different parts of the declaration of the ConvertAll method. When you look at a generic declaration—whether it’s for a generic type or a generic method—trying to work out what it means can be daunting, particularly if you have to deal with generic types of generic types, as you did when you saw the interface implemented by the dictionary. The key is to not panic—just take things calmly and pick an example situation. Use a different type for each type parameter, and apply them all consistently. List ConvertAll(Converter converter) Return type (a generic list) Parameter type (generic delegate)Method name Parameter nameType parameter Figure 3.2 The anatomy of a generic method declaration 68 CHAPTER 3 Parameterized typing with generics In this case, let’s start by replacing the type parameter of the type containing the method (the part of List). We’ll stick with the concept of a list of strings and replace T with string everywhere in the method declaration: List ConvertAll(Converter converter) That looks a bit better, but you’ve still got TOutput to deal with. You can tell that it’s a method’s type parameter (apologies for the confusing terminology) because it’s in angle brackets directly after the name of the method, so let’s try another familiar type—Guid—as the type argument for TOutput. Again you replace the type parameter with the type argument everywhere. You can now think of the method as if it were nongeneric, removing the type parameter part of the declaration: List ConvertAll(Converter converter) Now everything is expressed in terms of a concrete type, so it’s easier to think about. Even though the real method is generic, we’ll treat it as if it weren’t for the sake of understanding it better. Let’s go through the elements of this declaration from left to right:  The method returns a List.  The method’s name is ConvertAll.  The method has a single parameter: a Converter called converter. Now you just need to know what Converter is and you’re all done. Not surprisingly, Converter is a constructed generic delegate type (the unbound type is Converter), which is used to convert a string to a GUID. So you have a method that can operate on a list of strings, using a converter to pro- duce a list of GUIDs. Now that you understand the method’s signature, it’s easier to understand the documentation, which confirms that this method does the obvious thing and creates a new List, converts each element in the original list into the target type, adding it to the new list, and then returns that list. Thinking about the sig- nature in concrete terms gives you a clearer mental model, and makes it simpler to think about what the method might do. Although this technique may sound some- what simplistic, I find it useful for complicated methods even now. Some of the LINQ method signatures with four type parameters are fearsome beasts, but putting them into concrete terms tames them significantly. Just to prove I haven’t been leading you down the garden path, let’s take a look at the ConvertAll method in action. The following listing shows the conversion of a list of integers into a list of floating-point numbers, where each element of the second list is the square root of the corresponding element in the first list. After the conversion, the results are printed. 69Simple generics for everyday use static double TakeSquareRoot(int x) { return Math.Sqrt(x); } ... List integers = new List(); integers.Add(1); integers.Add(2); integers.Add(3); integers.Add(4); Converter converter = TakeSquareRoot; List doubles; doubles = integers.ConvertAll(converter); foreach (double d in doubles) { Console.WriteLine(d); } The creation and population of the list B is straightforward enough—it’s just a strongly typed list of integers. The assignment to converter C uses a feature of dele- gates (method group conversions) which is new to C# 2 and which we’ll discuss in more detail in section 5.2. Although I don’t like using a feature before describing it fully, the line would’ve been too long to fit on the page with the C# 1 delegate syntax. It does what you expect it to, though. At D you call the generic method, specifying the type argument for the method in the same way you’ve seen for generic types. This is one situation where you could’ve used type inference to avoid explicitly specifying the type argument, but I wanted to take it one step at a time. Writing out the list that has been returned is simple, and when you run the code you’ll see it print 1, 1.414…, 1.732…, and 2, as expected. What’s the point of all of this? We could’ve just used a foreach loop to go through the integers and printed out the square root immediately, of course, but it’s not uncommon to want to convert a list of one type to a list of another by performing some logic on it. The code to do it manually is simple, but it’s easier to read a version that does it in a single method call. That’s often the way with generic methods—they often do things that previously you’d have happily done “longhand” but that are sim- pler with a method call. Before generics, there could’ve been a similar operation to ConvertAll on ArrayList converting from object to object, but it would’ve been a lot less satisfactory. Anonymous methods (see section 5.4) also help here—if you hadn’t wanted to introduce an extra method, you could’ve specified the conversion inline. LINQ and lambda expressions take this pattern much further, as you’ll see in part 3 of the book. Note that generic methods can be part of nongeneric types as well. The following listing shows a generic method being declared and used within a normal nongeneric type. Listing 3.2 The List.ConvertAll method in action Creates, populates list of integers B Creates delegate instanceC Calls generic method to convert listD 70 CHAPTER 3 Parameterized typing with generics static List MakeList(T first, T second) { List list = new List(); list.Add(first); list.Add(second); return list; } ... List list = MakeList("Line 1", "Line 2"); foreach (string x in list) { Console.WriteLine (x); } The MakeList generic method only needs one type parameter (T). All it does is build a list containing the two parameters. It’s worth noting that you can use T as a type argument when you create the List in the method. Just as when we were looking at generic declarations, you can think of the implementation as (roughly speaking) replacing all of the mentions of T with string. When you call the method, you use the same syntax you’ve seen before to specify the type arguments. All okay so far? You should now have the hang of simple generics. There’s a bit more complexity to come, I’m afraid, but if you’re happy with the fundamental idea of generics, you’ve jumped the biggest hurdle. Don’t worry if it’s still a bit hazy (partic- ularly when it comes to the open/closed/unbound/constructed terminology), but now would be a good time to do some experimentation so you can see generics in action before you go any further. If you haven’t used the generic collections before, you might want to quickly look at appendix B, which describes what’s available. The collection types give you a simple starting point for playing with generics, and they’re widely used in almost every nontrivial .NET program. One thing you may find when you experiment is that it’s hard to go only part of the way. Once you make one part of an API generic, you’ll often find that you need to rework other code, either making that generic too or putting in the casts required by the new, more strongly typed method calls. An alternative would be to have a strongly typed implementation, using generic classes under the covers, but leaving a weakly typed API for the moment. As time goes on, you’ll become more confident about when it’s appropriate to use generics. 3.3 Beyond the basics The relatively simple uses of generics we’ve looked at so far can get you a long way, but there are some more features that can help you go further. We’ll start by examining type constraints, which give you more control over which type arguments can be specified. They’re useful when creating your own generic types and methods, and you’ll need to understand them in order to know what options are available when using the framework, too. Listing 3.3 Implementing a generic method in a nongeneric type 71Beyond the basics We’ll then examine type inference—a handy compiler trick that allows you to not explicitly state the type arguments when you’re using generic methods. You don’t have to use it, but it can make your code a lot easier to read when used appropriately. You’ll see in part 3 of the book that the C# compiler is gradually being allowed to infer a lot more information from your code, while still keeping the language safe and stati- cally typed.3 The last part of this section deals with obtaining the default value of a type param- eter and the comparisons that are available when you’re writing generic code. We’ll wrap up with an example that demonstrates most of the features we’ve covered and that’s a useful class in itself. Although this section delves a bit deeper into generics, there’s nothing really hard about it. There’s plenty to remember, but all the features serve a purpose, and you’ll be grateful for them when you need them. Let’s get started. 3.3.1 Type constraints So far, all the type parameters we’ve looked at can be applied to any type at all— they’re unconstrained. You can have a List, a Dictionary, anything. That’s fine when you’re dealing with collections that don’t have to interact with what they store, but not all uses of generics are like that. Often you want to call methods on instances of the type parameter, or create new instances, or make sure you only accept reference types (or only accept value types). In other words, you want to specify rules that say which type arguments are considered valid for your generic type or method. In C# 2, you do this with constraints. Four kinds of constraints are available, and the general syntax is the same for all of them. Constraints come at the end of the declaration of a generic method or type and are introduced by the contextual keyword where. They can be combined together in sensible ways, as you’ll see later. First, though, we’ll explore each kind of constraint in turn. REFERENCE TYPE CONSTRAINTS The first kind of constraint ensures that the type argument used is a reference type. It’s expressed as T : class and must be the first constraint specified for that type parameter. The type argument can be any class, interface, array, delegate, or another type parameter that’s already known to be a reference type. For example, consider the following declaration: struct RefSample where T: class Valid closed types using this declaration include  RefSample  RefSample  RefSample 3 Well, aside from any C# 4 code that explicitly uses dynamic typing, anyway. 72 CHAPTER 3 Parameterized typing with generics Invalid closed types include  RefSample  RefSample I deliberately made RefSample a struct (and therefore a value type) to emphasize the difference between the constrained type parameter and the type itself. RefSample is still a value type with value semantics everywhere—it just happens to use the string type wherever T is specified in the code. When a type parameter is constrained this way, you can compare references (including null) with == and !=, but be aware that unless there are any other con- straints, only references will be compared, even if the type in question overloads those operators (as string does, for example). With a conversion type constraint (described shortly), you can end up with compiler guaranteed overloads of == and !=, in which case those overloads are used—but that’s relatively rare. VALUE TYPE CONSTRAINTS The value type constraint, expressed as T : struct, ensures that the type argument used is a value type, including enums. It excludes nullable types (as described in chap- ter 4), though. Let’s look at an example declaration: class ValSample where T: struct Valid closed types include  ValSample  ValSample Invalid closed types include  ValSample  ValSample This time ValSample is a reference type, despite T being constrained to be a value type. Note that System.Enum and System.ValueType are both reference types in themselves, so they aren’t allowed as valid type arguments for ValSample. When a type parameter is constrained to be a value type, comparisons using == and != are prohibited. I rarely find myself using value or reference type constraints, although you’ll see in the next chapter that nullable value types rely on value type constraints. The remain- ing two constraints are likely to prove more useful when you’re writing your own generic types. CONSTRUCTOR TYPE CONSTRAINTS The constructor type constraint is expressed as T : new() and must be the last constraint for any particular type parameter. It simply checks that the type argument used has a parameterless constructor that can be used to create an instance. This is the case for any value type; for any nonstatic, nonabstract class without any explicitly declared construc- tors; and for any nonabstract class with an explicit public parameterless constructor. 73Beyond the basics C# VERSUS CLI STANDARDS There’s a discrepancy between the C# and CLI standards when it comes to value types and constructors. The C# specification states that all value types have a default parameterless constructor, and the language uses the same syntax to call both explicitly declared constructors and the parameterless one, relying on the compiler to do the right thing underneath. The CLI specification has no such requirement but provides a special instruction to create a default value without specifying any parame- ters. You can see this discrepancy at work when you use reflection to find the constructors of a value type—you won’t see a parameterless one. Again, let’s look at a quick example, this time for a method. Just to show how it’s use- ful, I’ll give the implementation of the method too: public T CreateInstance() where T: new() { return new T(); } This method returns a new instance of whatever type you specify, provided that it has a parameterless constructor. That means calls to CreateInstance() and Create- Instance() are okay, but CreateInstance() isn’t, because string doesn’t have a parameterless constructor. There’s no way of constraining type parameters to force other constructor signa- tures. For instance, you can’t specify that there has to be a constructor taking a single string parameter. It can be frustrating, but that’s unfortunately just the way it is. We’ll look at this issue in more detail when we consider the various restrictions of .NET generics in section 3.5. Constructor type constraints can be useful when you need to use factory-like pat- terns, where one object will create another one as and when it needs to. Factories often need to produce objects that are compatible with a certain interface, of course, and that’s where our last type of constraint comes in. CONVERSION TYPE CONSTRAINTS The final (and most complicated) kind of constraint lets you specify another type that the type argument must be implicitly convertible to via an identity, reference, or box- ing conversion. You can specify that one type argument be convertible to another type argument, too—this is called a type parameter constraint. These constraints make it harder to understand the declaration, but they can be handy every so often. Table 3.2 shows some examples of generic type declarations with conversion type constraints, along with valid and invalid examples of corresponding constructed types. The third constraint in table 3.2, T : IComparable, is just one example of using a generic type as the constraint. Other variations, such as T : List (where U is another type parameter) and T : IList, are also fine. 74 CHAPTER 3 Parameterized typing with generics You can specify multiple interfaces, but only one class. For instance, this is fine (if hard to satisfy): class Sample where T: Stream, IEnumerable, IComparable But this isn’t: class Sample where T: Stream, ArrayList, IComparable No type can derive directly from more than one class anyway, so such a constraint would usually either be impossible (like the preceding one) or part of it would be redundant (specifying that the type had to derive from both Stream and Memory- Stream, for example). There’s one more set of restrictions: the type you specify can’t be a value type, a sealed class (such as string), or any of the following “special” types:  System.Object  System.Enum  System.ValueType  System.Delegate WORKING AROUND THE LACK OF ENUM AND DELEGATE CONSTRAINTS The inabil- ity to specify the preceding types in conversion type constraints sounds like it’s due to a CLR restriction—but it’s not. That may have been true historically (at some point when generics were still being designed), but if you construct the appropriate code in IL, it works fine. The CLI specification even lists enum and delegate constraints as examples and explains what would be valid and what wouldn’t. This is frustrating, and there are plenty of generic meth- ods that would be useful when restricted to delegates or enums. I have an Table 3.2 Examples of conversion type constraints Declaration Constructed type examples class Sample where T : Stream Valid: Sample (identity conversion) Invalid: Sample struct Sample where T : IDisposable Valid: Sample (reference conversion) Invalid: Sample class Sample where T : IComparable Valid: Sample (boxing conversion) Invalid: Sample class Sample where T : U Valid: Sample (reference conversion) Invalid: Sample 75Beyond the basics open source project called Unconstrained Melody (http://code.google.com/ p/unconstrained-melody/), which performs some hackery to build a class library that does have these constraints on various utility methods. Although the C# compiler won’t let you declare such constraints, it’s happy to apply them when you call the methods in the library. Perhaps the prohibition will be lifted in a future version of C#. Conversion type constraints are probably the most useful kind, as they mean you can use members of the specified type on instances of the type parameter. One particu- larly handy example of this is T : IComparable, which enables you to compare two instances of T meaningfully and directly. We’ll look at an example of this (and discuss other forms of comparison) in section 3.3.3. COMBINING CONSTRAINTS I’ve mentioned the possibility of having multiple constraints, and you’ve seen them in action for conversion type constraints, but I haven’t shown the different kinds being combined together. Obviously no type can be both a reference type and a value type, so that combination is forbidden. Likewise, every value type has a parameterless con- structor, so you can’t specify the construction constraint when you already have a value type constraint (although you can still use new T() within methods if T is constrained to be a value type). If you have multiple conversion type constraints and one of them is a class, that has to come before the interfaces—and you can’t specify the same inter- face more than once. Different type parameters can have different constraints, and they’re each introduced with a separate where. Let’s look at some valid and invalid examples: Valid: class Sample where T: class, IDisposable, new() class Sample where T: struct, IDisposable class Sample where T: class where U: struct, T class Sample where T: Stream where U: IDisposable Invalid: class Sample where T: class, struct class Sample where T: Stream, class class Sample where T: new(), Stream class Sample where T: IDisposable, Stream class Sample where T: XmlReader, IComparable, IComparable class Sample where T: struct where U: class, T class Sample where T: Stream, U: IDisposable I included the last example in each list because it’s so easy to try the invalid one instead of the valid version, and the compiler error isn’t at all helpful. Just remember that each list of type parameter constraints needs its own introductory where. The third valid example is interesting—if U is a value type, how can it derive from T, which is a reference type? The answer is that T could be an object or an interface that U implements. It’s a pretty nasty constraint, though. 76 CHAPTER 3 Parameterized typing with generics SPECIFICATION TERMINOLOGY The specification categorizes constraints slightly differently—into primary constraints, secondary constraints, and con- structor constraints. A primary constraint is a reference type constraint, a value type constraint, or a conversion type constraint using a class. A secondary con- straint is a conversion type constraint using an interface or another type parameter. I don’t find these particularly useful categories, but they make it easier to define the grammar of constraints: the primary constraint is optional but you can only have one; you can have as many secondary constraints as you like; the constructor constraint is optional (unless you have a value type con- straint, in which case it’s forbidden). Now that you know all you need to read generic type declarations, let’s look at the type argument inference that I mentioned earlier. In listing 3.2 you explicitly stated the type arguments to List.ConvertAll, and you did the same in listing 3.3 for the MakeList method—now let’s ask the compiler to work them out when it can, mak- ing it simpler to call generic methods. 3.3.2 Type inference for type arguments of generic methods Specifying type arguments when you’re calling a generic method can often seem pretty redundant. Usually it’s obvious what the type arguments should be, based on the method arguments themselves. To make life easier, from C# 2 onward, the com- piler is allowed to be smart in tightly defined ways, so you can call the method without explicitly stating the type arguments. But before we go any further, I should stress that this is only true for generic methods. It doesn’t apply to generic types. Let’s look at the relevant lines from listing 3.3 and see how things can be simpli- fied. Here are the lines declaring and invoking the method: static List MakeList(T first, T second) ... List list = MakeList("Line 1", "Line 2"); Look at the arguments—they’re both strings. Each of the parameters in the method is declared to be of type T. Even if you didn’t have the part of the method invo- cation expression, it would be fairly obvious that you meant to call the method using string as the type argument for T. The compiler allows you to omit it, leaving this: List list = MakeList("Line 1", "Line 2"); That’s a bit neater, isn’t it? At least, it’s shorter. That doesn’t always mean it’s more readable, of course. In some cases it’ll be harder for the reader to work out what type arguments you’re trying to use, even if the compiler can do it easily. I recommend that you judge each case on its merits. My personal preference is to let the compiler infer the type arguments in most cases where it works. Note how the compiler definitely knows that you’re using string as the type argu- ment, because the assignment to list works too, and that still does specify the type argument (and has to). The assignment has no influence on the type argument inference process, though. It just means that if the compiler works out what type 77Beyond the basics arguments it thinks you want to use but gets it wrong, you’re still likely to get a compile-time error. How could the compiler get it wrong? Suppose you actually want to use object as the type argument. The method parameters are still valid, but the compiler thinks you meant to use string, as they’re both strings. Changing one of the parameters to explicitly be cast to object makes type inference fail, as one of the method arguments would suggest that T should be string, and the other suggests that T should be object. The compiler could look at this and say that setting T to object would satisfy everything but setting T to string wouldn’t, but the specification only has a limited number of steps to follow. This subject is fairly complicated in C# 2, and C# 3 takes things even further. I won’t try to cover all the nuts and bolts of the C# 2 rules here, but the basic steps are as follows: 1 For each method argument (the bits in normal parentheses, not angle brack- ets), try to infer some of the type arguments of the generic method, using some fairly simple techniques. 2 Check that all the results from the first step are consistent. In other words, if one argument implied one type argument for a particular type parameter, and another implied a different type argument for the same type parameter, then inference fails for the method call. 3 Check that all the type parameters needed for the generic method have been inferred. You can’t let the compiler infer some while you specify others explic- itly—it’s all or nothing. To avoid learning all the rules (and I wouldn’t recommend it unless you’re particu- larly interested in the fine details), there’s one simple thing to do: try it and see what happens. If you think the compiler might be able to infer all the type arguments, try calling the method without specifying any. If it fails, stick the type arguments in explic- itly. You lose nothing more than the time it takes to compile the code once, and you don’t need to have all the extra language-lawyer garbage in your head. To make it easier to use generic types, type inference can be combined with the idea of overloading type names based on the number of type parameters. We’ll look at an example of this in a while, when we put everything together. 3.3.3 Implementing generics You’re likely to spend more time using generic types and methods than writing them yourself. Even when you’re providing the implementation, you can usually just pre- tend that T (or whatever your type parameter is called) is the name of a type and get on with writing code as if you weren’t using generics at all. But there are a few extra things you should know. DEFAULT VALUE EXPRESSIONS When you know exactly what type you’re working with, you know its default value— the value an otherwise uninitialized field would have, for instance. When you don’t 78 CHAPTER 3 Parameterized typing with generics know what type you’re referring to, though, you can’t specify that default value directly. You can’t use null because it might not be a reference type. You can’t use 0 because it might not be a numeric type. It’s fairly rare to need the default value, but it can be useful on occasion. Dictionary is a good example—it has a TryGetValue method that works a bit like the TryParse methods on the numeric types: it uses an output parameter for the value you’re trying to fetch and a Boolean return value to indicate whether it succeeded. This means that the method has to have some value of type TValue to populate the output parameter with. (Remember that output parameters must be assigned before the method returns normally.) THE TRYXXX PATTERN A few patterns in .NET are easily identifiable by the names of the methods involved—BeginXXX and EndXXX suggest an asynchro- nous operation, for example. The TryXXX pattern is one that has had its use expanded from .NET 1.1 to 2.0. It’s designed for situations that might nor- mally be considered to be errors (in that the method can’t perform its pri- mary duty), but where failure could well occur without indicating a serious issue, and shouldn’t be deemed exceptional. For instance, users often fail to type in numbers correctly, so being able to try to parse some text without hav- ing to catch an exception and swallow it is useful. Not only does it improve performance in the failure case, but more importantly, it saves exceptions for genuine error cases where something is wrong in the system (however widely you wish to interpret that). It’s a useful pattern to have up your sleeve as a library designer, when applied appropriately. C# 2 provides the default value expression to care for just this need. The specification doesn’t refer to it as an operator, but you can think of it as being similar to the typeof operator, just returning a different value. The following listing shows this in a generic method, and also gives an example of type inference and a conversion type constraint in action. static int CompareToDefault(T value) where T: IComparable { return value.CompareTo(default(T)); } ... Console.WriteLine(CompareToDefault("x")); Console.WriteLine(CompareToDefault(10)); Console.WriteLine(CompareToDefault(0)); Console.WriteLine(CompareToDefault(-10)); Console.WriteLine(CompareToDefault(DateTime.MinValue)); Listing 3.4 shows a generic method being used with three different types: string, int,and DateTime. The CompareToDefault method dictates that it can only be used with types implementing the IComparable interface, which allows you to call Listing 3.4 Comparing a given value to the default in a generic way 79Beyond the basics CompareTo(T) on the value passed in. The other value you use for the comparison is the default value for the type. As string is a reference type, the default value is null, and the documentation for CompareTo states that for reference types, everything should be greater than null, so the first result is 1. The next three lines show compar- isons with the default value of int, demonstrating that the default value is 0. The out- put of the last line is 0, showing that DateTime.MinValue is the default value for DateTime. Of course, the method in listing 3.4 will fail if you pass it null as the argument— the line calling CompareTo will throw NullReferenceException in the normal way. Don’t worry about that for the moment—there’s an alternative using IComparer, as you’ll see soon. DIRECT COMPARISONS Although listing 3.4 showed how a comparison is possible, you won’t always want to con- strain your types to implement IComparable or its sister interface, IEquatable, which provides a strongly typed Equals(T) method to complement the Equals (object) method that all types have. Without the extra information these interfaces give you access to, there’s little you can do in terms of comparisons, other than calling Equals(object), which will result in boxing the value you want to compare with when it’s a value type. (There are a couple of types to help you in some situations—we’ll come to them in a minute.) When a type parameter is unconstrained (no constraints are applied to it), you can use the == and != operators, but only to compare a value of that type with null; you can’t compare two values of type T with each other. When the type argument is a refer- ence type, the normal reference comparison will be used. In the case where the type argument provided for T is a non-nullable value type, a comparison with null will always decide that they’re unequal (so the comparison can be removed by the JIT compiler). When the type argument is a nullable value type, the comparison will behave in the natural way, making the comparison against the null value of the type.4 (Don’t worry if this last bit doesn’t make sense yet—it will when you’ve read the next chapter. Some features are too intertwined to allow me to describe either of them completely without referring to the other, unfortunately.) When a type parameter is constrained to be a value type, == and != can’t be used with it at all. When it’s constrained to be a reference type, the kind of comparison performed depends on how the type parameter is constrained. If the only constraint is that it’s a reference type, simple reference comparisons are performed. If it’s fur- ther constrained to derive from a particular type that overloads the == and != opera- tors, those overloads are used. Beware, though—extra overloads that happen to be made available by the type argument specified by the caller are not used. The next 4 At the time of this writing (testing with .NET 4.5 and earlier), the code generated by the JIT compiler for com- paring unconstrained type parameter values against null is extremely slow for nullable value types. If you constrain a type parameter T to be non-nullable and then compare a value of type T? against null, that com- parison is much faster. This shows some scope for future JIT optimization. 80 CHAPTER 3 Parameterized typing with generics listing demonstrates this with a simple reference type constraint and a type argument of string. static bool AreReferencesEqual(T first, T second) where T: class { return first == second; } ... string name = "Jon"; string intro1 = "My name is " + name; string intro2 = "My name is " + name; Console.WriteLine(intro1 == intro2); Console.WriteLine(AreReferencesEqual(intro1, intro2)); Even though string overloads == (as demonstrated by the comparison at C printing True), this overload isn’t used by the comparison at B. Basically, when AreReferences- Equal is compiled, the compiler doesn’t know what overloads will be available—it’s as if the parameters passed in were of type object. This isn’t specific to operators—on encountering a generic type, the compiler resolves all the method overloads when compiling the unbound generic type, rather than reconsidering each possible method call for more specific overloads at execution time. For instance, a statement of Console.WriteLine(default(T)); will always resolve to a call to Console.WriteLine(object value)—it doesn’t call Console .WriteLine(string value) when T happens to be string. This is similar to the nor- mal situation of overloads being chosen at compile time rather than execution time, but readers familiar with templates in C++ may be surprised nonetheless.5 Two classes that are extremely useful when it comes to comparing values are EqualityComparer and Comparer, both in the System.Collections.Generic namespace. They implement IEqualityComparer and IComparer, respectively, and the Default property returns an implementation that generally does the right thing for the appropriate type. THE GENERIC COMPARISON INTERFACES There are four main generic inter- faces for comparisons. Two of them—IComparer and IComparable— are about comparing values for ordering (is one value less than, equal to, or greater than the other?), and the other two—IEqualityComparer and IEquatable—are for comparing two items for equality according to some criteria and for finding the hash of an item (in a manner compatible with the same notion of equality). Splitting the four another way, IComparer and IEqualityComparer are implemented by types that are capable of comparing two different values, whereas an instance of IComparable or IEquatable is capable of com- paring itself with another value. Listing 3.5 Comparisons using == and != performing reference comparisons 5 You’ll see in chapter 14 that dynamic typing provides the ability to resolve overloads at execution time. Compares references B Compares using string overload C 81Beyond the basics See the documentation for more details, and consider using these (and similar types such as StringComparer) when performing comparisons. We’ll use Equality- Comparer in the next example. FULL COMPARISON EXAMPLE: REPRESENTING A PAIR OF VALUES To finish off our section on implementing generics, here’s a complete example. It implements a useful generic type—a Pair that holds two values together, like a key/value pair, but with no expectations as to the relationship between the two values. .NET 4 AND TUPLES .NET 4 provides a lot of this functionality out of the box— and for many different numbers of type parameters, too. Look for Tuple, Tuple, and so on in the System namespace. In addition to providing properties to access the values themselves, you’ll override Equals and GetHashCode to allow instances of your type to play nicely when used as keys in a dictionary. The following listing gives the complete code. using System; using System.Collections.Generic; public sealed class Pair : IEquatable> { private static readonly IEqualityComparer FirstComparer = EqualityComparer.Default; private static readonly IEqualityComparer SecondComparer = EqualityComparer.Default; private readonly T1 first; private readonly T2 second; public Pair(T1 first, T2 second) { this.first = first; this.second = second; } public T1 First { get { return first; }} public T2 Second { get { return second; }} public bool Equals(Pair other) { return other != null && FirstComparer.Equals(this.First, other.First) && SecondComparer.Equals(this.Second, other.Second); } public override bool Equals(object o) { return Equals(o as Pair); } Listing 3.6 Generic class representing a pair of values 82 CHAPTER 3 Parameterized typing with generics public override int GetHashCode() { return FirstComparer.GetHashCode(first) * 37 + SecondComparer.GetHashCode(second); } } Listing 3.6 is straightforward. The constituent values are stored in appropriately typed member variables, and access is provided by simple read-only properties. You imple- ment IEquatable> to give a strongly typed API that’ll avoid unnecessary execution-time checks. The equality and hash-code computations both use the default equality comparer for the two type parameters—these handle nulls automatically, which makes the code somewhat simpler. The static variables used to store the equal- ity comparers for T1 and T2 are mostly there for the sake of formatting the code for the printed page, but they’ll also be useful as a reference point in the next section. CALCULATING HASH CODES The formula used for calculating the hash code based on the two “part” results comes from Effective Java, 2nd edition (Addi- son-Wesley, 2008), by Joshua Bloch. It certainly doesn’t guarantee a good dis- tribution of hash codes, but in my opinion it’s better than using a bitwise exclusive OR. See Effective Java for more details, and for many other useful tips and design insights. Now that you have your Pair class, how do you construct an instance of it? At the moment, you’d need to use something like this: Pair pair = new Pair(10, "value"); That’s not terribly nice. It would be good to use type inference, but that only works for generic methods, and you don’t have any of those. If you put a generic method in the generic type, you’d still need to specify the type arguments for the type before you could call a method on it, which would defeat the purpose. The solution is to use a nongeneric helper class with a generic method in it, as shown in the following listing. public static class Pair { public static Pair Of(T1 first, T2 second) { return new Pair(first, second); } } If this is your first time reading this book, ignore the fact that the class is declared to be static—we’ll come to that in chapter 7. The important point is that you have a non- generic class with a generic method. That means you can turn the previous example into this far-more-pleasant version: Pair pair = Pair.Of(10, "value"); Listing 3.7 Using a nongeneric type with a generic method to enable type inference 83Advanced generics In C# 3 you could even dispense with the explicit typing of the pair variable, but let’s not get ahead of ourselves. This use of nongeneric helper classes (or partially generic helper classes, if you have two or more type parameters and want to infer some of them but leave others explicit) is a handy trick. We’ve finished looking at the intermediate features now. I realize it can all seem complicated at first, but don’t be put off; the benefits of generics far outweigh the added complexity. Over time, they become second nature. Now that you have the Pair class as an example, it might be worth looking over your own code base to see whether there are some patterns that you keep reimplementing solely to use differ- ent types. In any large topic there’s always more to learn. The next section will take you through the most important advanced topics in generics. If you’re feeling over- whelmed at this point, you might want to skip to the relative comfort of section 3.5, where we’ll explore some of the limitations of generics. It’s worth understanding the topics in the next section eventually, but if everything so far has been new to you, it won’t hurt to skip it for the moment. 3.4 Advanced generics You may expect me to claim that in the rest of this chapter we’ll cover every aspect of generics that we haven’t looked at so far. But there are so many little nooks and cran- nies involving generics that it’s simply not possible—and I certainly wouldn’t want to read about all the details, let alone write about them. Fortunately, the nice people at Microsoft and ECMA have written down all the details in the language specification, so if you ever want to check some obscure situation that isn’t covered here, that should be your next port of call. Unfortunately I can’t point to one particular area of the specification that covers generics: they pop up almost everywhere. Arguably, if your code ends up in a corner case so complicated that you need to consult the specifica- tion to work out what it should do, you should refactor it into a more obvious form anyway; you don’t want each maintenance engineer from now until eternity to have to read the gory details. My aim with this section is to cover everything you’re likely to want to know about generics. I’ll talk more about the CLR and the framework side of things than the par- ticular syntax of the C# 2 language, although it’s all relevant when developing in C#. We’ll start by considering static members of generic types, including type initializa- tion. From there, it’s a natural step to wonder how all this is implemented under the covers, but we’ll keep it fairly light on detail, concentrating on the important effects of the implementation decisions. We’ll look at what happens when you enumerate a generic collection using foreach in C# 2, and round off the section by seeing how reflection in the .NET Framework is affected by generics. 84 CHAPTER 3 Parameterized typing with generics 3.4.1 Static fields and static constructors Just as instance fields belong to an instance, static fields belong to the type they’re declared in. If you declare a static field x in class SomeClass, there’s exactly one Some- Class.x field, no matter how many instances of SomeClass you create, and no matter how many types derive from SomeClass.6 That’s the familiar scenario from C# 1—so how does it map across to generics? The answer is that each closed type has its own set of static fields. You saw this in list- ing 3.6 when you stored the default equality comparers for T1 and T2 in static fields, but let’s look at it in more detail with another example. The following listing creates a generic type including a static field. You set the field’s value for different closed types and then print out the values to show that they’re separate. class TypeWithField { public static string field; public static void PrintField() { Console.WriteLine(field + ": " + typeof(T).Name); } } ... TypeWithField.field = "First"; TypeWithField.field = "Second"; TypeWithField.field = "Third"; TypeWithField.PrintField(); TypeWithField.PrintField(); TypeWithField.PrintField(); You set the value of each field to a different value and then print out each field along with the name of the type argument used for that closed type. Here’s the output from listing 3.8: First: Int32 Second: String Third: DateTime The basic rule is one static field per closed type. The same applies for static initializers and static constructors. But it’s possible to have one generic type nested within another, and types with multiple generic parameters. This sounds a lot more compli- cated, but it works as you probably think it should. The following listing shows this in action, this time using static constructors to show just how many types there are. 6 Well, there’s one per application domain. For the purposes of this section, we’ll assume we’re only dealing with one application domain. The concepts for different application domains work the same with generics as with nongeneric types. Variables decorated with [ThreadStatic] violate this rule, too. Listing 3.8 Proof that different closed types have different static fields 85Advanced generics public class Outer { public class Inner { static Inner() { Console.WriteLine("Outer<{0}>.Inner<{1},{2}>", typeof(T).Name, typeof(U).Name, typeof(V).Name); } public static void DummyMethod() {} } } ... Outer.Inner.DummyMethod(); Outer.Inner.DummyMethod(); Outer.Inner.DummyMethod(); Outer.Inner.DummyMethod(); Outer.Inner.DummyMethod(); Outer.Inner.DummyMethod(); The first call to DummyMethod() for any type will cause the type to be initialized, at which point the static constructor prints out some diagnostics. Each different list of type arguments counts as a different closed type, so the output of listing 3.9 looks like this: Outer.Inner Outer.Inner Outer.Inner Outer.Inner Outer.Inner Just as with nongeneric types, the static constructor for any closed type is only exe- cuted once, which is why the last line of listing 3.9 doesn’t create a sixth line of out- put—the static constructor for Outer.Inner executed earlier, producing the second line of output. To clear up any doubts, if you had a nongeneric PlainInner class inside Outer, there still would’ve been one possible Outer.PlainInner type per closed Outer type, so Outer.PlainInner would be separate from Outer.PlainInner, with a separate set of static fields, as seen earlier. Now that you’ve seen what constitutes a different type, we should look at what the effects of that might be in terms of the amount of native code generated. And no, it’s not as bad as you might think… 3.4.2 How the JIT compiler handles generics Given that we have all of these different closed types, the JIT’s job is to convert the IL of the generic type into native code so it can actually be run. In some ways, you Listing 3.9 Static constructors with nested generic types 86 CHAPTER 3 Parameterized typing with generics shouldn’t care exactly how it does that—beyond keeping a close eye on memory and CPU time, you wouldn’t see much difference if the JIT took the simplest possible approach and generated native code for each closed type separately, as if each one had nothing to do with any other type. But the JIT authors are clever enough that it’s worth looking at what they’ve done. Let’s start with a simple situation first, with a single type parameter—we’ll use List for the sake of convenience. The JIT creates different code for each closed type with a type argument that’s a value type—int, long, Guid, and the like. But it shares the native code generated for all the closed types that use a reference type as the type argument, such as string, Stream, and StringBuilder. It can do this because all references are the same size (the size varies between a 32-bit CLR and a 64-bit CLR, but within any one CLR all references are the same size). An array of references will always be the same size, whatever the references happen to be. The space required on the stack for a reference will always be the same. The JIT can use the same optimizations to store references in registers regardless of the type—the List goes on. Each of the types still has its own static fields, as described in section 3.4.1, but the executable code itself is reused. Of course, the JIT does all of this lazily—it won’t gen- erate the code for List before it needs to, and it’ll cache that code for all future uses of List. In theory, it’s possible to share code for at least some value types. The JIT would have to be careful, not just due to size, but also for garbage collection reasons—it would have to be able to quickly identify areas of a struct value that are live references. But value types that are the same size and have the same in-memory footprint as far as the garbage collector is concerned could share code. At the time of this writing, that’s been of sufficiently low priority that it hasn’t been implemented, and it may well stay that way. This level of detail is primarily of academic interest, but it does have a slight perfor- mance impact in terms of more code being JIT compiled. The performance benefits of generics can be huge, though, and again that comes down to having the opportunity to compile to different code for different types. Consider a List, for instance. In .NET 1.1, adding individual bytes to an ArrayList would’ve meant boxing each one of them and storing a reference to each boxed value. Using List has no such impact—List has a member of type T[] to replace the object[] within ArrayList, and that array is of the appropriate type, taking the appropriate space. List has a straight byte[] within it used to store the elements of the array. (In many ways, this makes a List behave like a MemoryStream.) Figure 3.3 shows an ArrayList and a List, each with the same six values. The arrays themselves have more than six elements, to allow for growth. Both List and ArrayList have a buffer, and they create a larger buffer when they need to. The difference in efficiency here is incredible. Let’s look at the ArrayList first, con- sidering a 32-bit CLR.7 Each of the boxed bytes will take up 8 bytes of object overhead, 7 When running on a 64-bit CLR, the overheads are bigger. 87Advanced generics plus 4 bytes (1 byte, rounded up to a word boundary) for the data itself. On top of that, you have all the references themselves, each of which takes up 4 bytes. So for each byte of useful data, you’re paying at least 16 bytes—and then there’s the extra unused space for references in the buffer. Compare this with the List. Each byte in the list takes up a single byte within the elements array. There’s still wasted space in the buffer, waiting to be used by new items, but at least you’re only wasting a single byte per unused element there. You don’t just gain space, you gain execution speed, too. You save the time taken to allocate the box, to perform the type checking involved in unboxing the bytes in order to get at them, and to garbage collect the boxes when they’re no longer referenced. You don’t have to go down to the CLR level to find things happening transparently on your behalf, though. C# has always made life easier with syntactic shortcuts, and the next section looks at a familiar example but with a generic twist: iterating with foreach. 3.4.3 Generic iteration One of the most common operations you’ll want to perform on a collection is to iter- ate through all its elements. Usually, the simplest way of doing that is to use the foreach statement. In C# 1, this relied on the collection either implementing the System.Collections.IEnumerable interface or having a similar GetEnumerator() method that returned a type with a suitable MoveNext() method and a Current prop- erty. The Current property didn’t have to be of type object, and that was the whole 5 3 15 10 12 13 ... List ref 6 byte[]ArrayList ref 6 ref ref ref ref ref ref ... object[] 5 3 15 10 12 13 Boxed bytes count count elements elements Figure 3.3 Visual demonstration of why List takes up a lot less space than ArrayList when storing value types 88 CHAPTER 3 Parameterized typing with generics point of having these extra rules, which look odd at first sight. Yes, even in C# 1 you could avoid boxing and unboxing during iteration if you had a custom iteration type. C# 2 makes this somewhat easier, as the rules for the foreach statement have been extended to also use the System.Collections.Generic.IEnumerable interface along with its partner, IEnumerator. These are simply the generic equivalents of the old iteration interfaces, and they’re used in preference to the nongeneric ver- sions. This means that if you iterate through a generic collection of value type ele- ments—List, for example—then no boxing is performed at all. If the old interface had been used instead, you wouldn’t have incurred the boxing cost while storing the elements of the list, but you’d still have ended up boxing them when you retrieved them using foreach. All of this is done for you under the covers—all you need to do is use the foreach statement in the normal way, using an appropriate type for the iteration variable, and all will be well. That’s not the end of the story, though. In the relatively rare situation where you need to implement iteration over one of your own types, you’ll find that IEnumerable extends the old IEnumerable interface, which means you have to implement two different methods: IEnumerator GetEnumerator(); IEnumerator GetEnumerator(); Can you see the problem? The methods differ only in return type, and the overloading rules of C# prevent you from writing two such methods normally. Back in section 2.2.2, you saw a similar situation, and you can use the same workaround here. If you imple- ment IEnumerable using explicit interface implementation, you can implement IEnumerable with a “normal” method. Fortunately, because IEnumerator extends IEnumerator, you can use the same return value for both methods and imple- ment the nongeneric method by just calling the generic version. Of course, now you need to implement IEnumerator and you quickly run into similar problems, this time with the Current property. The following listing gives a full example, implementing an enumerable class that always enumerates the integers 0 to 9. class CountingEnumerable: IEnumerable { public IEnumerator GetEnumerator() { return new CountingEnumerator(); } IEnumerator IEnumerable.GetEnumerator() { return GetEnumerator(); } } Listing 3.10 A full generic iterator—of the numbers 0 to 9 Implements IEnumerable implicitlyB Implements IEnumerable explicitlyC 89Advanced generics class CountingEnumerator : IEnumerator { int current = -1; public bool MoveNext() { current++; return current < 10; } public int Current { get { return current; }} object IEnumerator.Current { get { return Current; }} public void Reset() { current = -1; } public void Dispose() {} } ... CountingEnumerable counter = new CountingEnumerable(); foreach (int x in counter) { Console.WriteLine(x); } Clearly these results aren’t particularly useful, but the code shows the little hoops you have to go through in order to implement generic iteration appropriately—at least if you’re doing it all longhand. (And that’s without making an effort to throw excep- tions if Current is accessed at an inappropriate time.) If you think that listing 3.10 looks like a lot of work just to print out the numbers 0 to 9, I can’t help but agree with you, and there’d be even more code if you wanted to iterate over anything useful. For- tunately, you’ll see in chapter 6 that C# 2 takes a large amount of the work away from iterators in many cases. I’ve shown the full version here so you can appreciate the slight wrinkles that have been introduced by the design decision for IEnumerable to extend IEnumerable. I’m not suggesting it was the wrong decision, though; it allows you to pass any IEnumerable into a method written in C# 1 with an IEnumerable parameter. That’s not as important now as it was back in 2005, but it’s still a useful transition path. You only need the trick of using explicit interface implementation twice—once for IEnumerable.GetEnumerator C and once for IEnumerator.Current E. Both of these call their generic equivalents (B and D, respectively). Another addition to IEnumerator is that it extends IDisposable, so you have to provide a Dispose method. The foreach statement in C# 1 already called Dispose on an iterator if it implemented IDisposable, but in C# 2 no execution-time testing is required—if the compiler finds that you’ve implemented IEnumerable, it creates an unconditional call to Dispose at the end of the loop (in a finally block). Many iterators won’t actu- ally need to dispose of anything, but it’s nice to know that when it is required, the most common way of working through an iterator (the foreach statement F) handles the Implements IEnumerator.Current implicitly D Implements IEnumerator.Current explicitly E Proves that enumerable type works F 90 CHAPTER 3 Parameterized typing with generics calling side automatically. This is most commonly used to release resources when you’ve finished iterating. For example, you might have an iterator that reads lines from a file and needs to close the file handle when the calling code has finished looping. We’ll now go from compile-time efficiency to execution-time flexibility: our final advanced topic is reflection. Even in .NET 1.0/1.1, reflection could be tricky, but generic types and methods introduce an extra level of complexity. The framework provides everything you need (with a bit of helpful syntax from C# 2 as a language), and although the additional considerations can be daunting, it’s not too bad if you take it one step at a time. 3.4.4 Reflection and generics Reflection is used by different people for all sorts of things. You might use it for execution-time introspection of objects to perform a simple form of data binding. You might use it to inspect a directory full of assemblies to find implementations of a plug- in interface. You might write a file for an inversion of control framework (see www.martinfowler.com/articles/injection.html) to load and dynamically configure your application’s components. As the uses of reflection are so diverse, I won’t focus on any particular one but will instead give you more general guidance on performing common tasks. We’ll start by looking at the extensions to the typeof operator. USING TYPEOF WITH GENERIC TYPES Reflection is all about examining objects and their types. As such, one of the most important things you need to be able to do is obtain a reference to a particular System.Type object, which allows access to all the information about that type. C# uses the typeof operator to obtain such a reference for types known at compile time, and this has been extended to encompass generic types. There are two ways of using typeof with generic types—one retrieves the generic type definition (in other words, the unbound generic type) and one retrieves a particu- lar constructed type. To obtain the generic type definition—the type with none of the type arguments specified—you simply take the name of the type as it would’ve been declared and remove the type parameter names, keeping any commas. To retrieve constructed types, you specify the type arguments in the same way as you would to declare a variable of the generic type. The next listing gives an example of both uses. It uses a generic method so we can revisit how typeof can be used with a type parame- ter, which we previously saw in 3.8. static void DemonstrateTypeof() { Console.WriteLine(typeof(X)); Console.WriteLine(typeof(List<>)); Console.WriteLine(typeof(Dictionary<,>)); Console.WriteLine(typeof(List)); Listing 3.11 Using the typeof operator with type parameters Displays method’s type parameter Displays generic types Displays closed types (despite using type parameter) B 91Advanced generics Console.WriteLine(typeof(Dictionary)); Console.WriteLine(typeof(List)); Console.WriteLine(typeof(Dictionary)); } ... DemonstrateTypeof(); Most of listing 3.11 works as you might naturally expect, but it’s worth pointing out two things. First, look at the syntax for obtaining the generic type definition of Dic- tionary . The comma in the angle brackets is required to tell the compiler to look for the type with two type parameters; remember that there can be several generic types with the same name, as long as they vary by the number of type parameters they have. Similarly, you’d retrieve the generic type definition for MyClass using typeof(MyClass<,,,>). The number of type parameters is specified in IL (and in full type names as far as the framework is concerned) by put- ting a back tick after the first part of the type name and then the number. The type parameters are then indicated in square brackets instead of the angle brackets we’re used to. For instance, the second line printed ends with List`1[T], showing that there’s one type parameter, and the third line includes Dictionary`2[TKey,TValue]. Second, note that wherever the method’s type parameter (X) is used, the actual value of the type argument is used at execution time. So this line B prints List`1[System. Int32] rather than List`1[X], which you might have expected.8 In other words, a type that’s open at compile time may be closed at execution time. This is very confusing. You should be aware of it in case you don’t get the results you expect, but other- wise, don’t worry. To retrieve a truly open constructed type at execution time, you need to work a bit harder. See the MSDN documentation for Type.IsGenericType for a suit- ably convoluted example (http://mng.bz/9W6O). For reference, here’s the output of listing 3.11: System.Int32 System.Collections.Generic.List`1[T] System.Collections.Generic.Dictionary`2[TKey,TValue] System.Collections.Generic.List`1[System.Int32] System.Collections.Generic.Dictionary`2[System.String,System.Int32] System.Collections.Generic.List`1[System.Int64] System.Collections.Generic.Dictionary`2[System.Int64,System.Guid] Having retrieved an object representing a generic type, there are many next steps you can take. All the previously available ones (finding the members of the type, creating an instance, and so on) are still present—although some aren’t applicable for generic type definitions—and there are new ones as well that let you inquire about the generic nature of the type. 8 I deliberately bucked the convention of using a type parameter named T, precisely so that we could tell the difference between the T in the List declaration and the X in our method declaration. Displays closed types 92 CHAPTER 3 Parameterized typing with generics METHODS AND PROPERTIES OF SYSTEM.TYPE There are far too many new methods and properties to look at them all in detail, but there are two particularly important ones: GetGenericTypeDefinition and Make- GenericType. They’re effectively opposites—the first acts on a constructed type, retrieving the generic type definition; the second acts on a generic type definition and returns a constructed type. Arguably it would’ve been clearer if this method had been called ConstructType, MakeConstructedType, or some other name with construct or constructed in it, but we’re stuck with what we’ve got. Just like normal types, there’s only one Type object for any particular type—so call- ing MakeGenericType twice with the same types as arguments will return the same ref- erence twice. Similarly, calling GetGenericTypeDefinition on two types constructed from the same generic type definition will give the same result for both calls, even if the constructed types are different (such as List and List). Two other methods worth exploring—this time methods that already existed in .NET 1.1—are Type.GetType(string) and its related Assembly.GetType(string) method, both of which provide a dynamic equivalent to typeof. You might expect to be able to feed each line of the output of listing 3.11 to the GetType method called on an appropriate assembly, but unfortunately life isn’t quite that straightforward. It’s fine for closed constructed types—the type arguments just go in square brackets. For generic type definitions, though, you need to remove the square brackets entirely— otherwise GetType thinks you mean an array type. The following listing shows all of these methods in action. string listTypeName = "System.Collections.Generic.List`1"; Type defByName = Type.GetType(listTypeName); Type closedByName = Type.GetType(listTypeName + "[System.String]"); Type closedByMethod = defByName.MakeGenericType(typeof(string)); Type closedByTypeof = typeof(List); Console.WriteLine(closedByMethod == closedByName); Console.WriteLine(closedByName == closedByTypeof); Type defByTypeof = typeof(List<>); Type defByMethod = closedByName.GetGenericTypeDefinition(); Console.WriteLine(defByMethod == defByName); Console.WriteLine(defByName == defByTypeof); The output of listing 3.12 is just True four times, validating that however you obtain a reference to a particular type object, only one such object is involved. As I mentioned earlier, there are many new methods and properties on Type, such as GetGenericArguments, IsGenericTypeDefinition, and IsGenericType. Again, the documentation for IsGenericType is probably the best starting point for further exploration. Listing 3.12 Various ways of retrieving generic and constructed Type objects 93Advanced generics REFLECTING GENERIC METHODS Generic methods have a similar (though smaller) set of additional properties and methods. The following listing gives a brief demonstration of this, calling a generic method by reflection. public static void PrintTypeParameter() { Console.WriteLine(typeof(T)); } ... Type type = typeof(Snippet); MethodInfo definition = type.GetMethod("PrintTypeParameter"); MethodInfo constructed = definition.MakeGenericMethod(typeof(string)); constructed.Invoke(null, null); First you retrieve the generic method definition, and then you make a constructed generic method using MakeGenericMethod. As with types, you could go the other way if you wanted to, but unlike Type.GetType, there’s no way of specifying a constructed method in the GetMethod call. The framework also has a problem if methods are over- loaded purely by number of type parameters—there are no methods in Type that allow you to specify the number of type parameters, so instead you’d have to call Type.GetMethods and find the right one by looking through all the methods. After retrieving the constructed method, you invoke it. The arguments in this example are both null, as you’re invoking a static method that doesn’t have any nor- mal parameters. The output is System.String, as you’d expect. Note that the meth- ods retrieved from generic type definitions can’t be invoked directly—instead, you must get the methods from a constructed type. This applies to both generic and non- generic methods. SAVED BY C# 4 If all of this looks messy to you, I agree. Fortunately, in many cases C#’s dynamic typing can come to the rescue, taking a lot of the work out of generic reflection. It doesn’t help in all situations, so it’s worth being aware of the general flow of the preceding code, but where it does apply it’s great. We’ll look at dynamic typing in detail in chapter 14. Again, more methods and properties are available on MethodInfo, and IsGeneric- Method is a good starting point in MSDN (http://mng.bz/P36u). Hopefully the infor- mation in this section will have been enough to get you going, and to point out some of the added complexities you might not have otherwise anticipated when first start- ing to access generic types and methods with reflection. That’s all we’ll cover in the way of advanced features. Just to reiterate, this chapter isn’t meant to be a complete guide to generics by any means, but most developers are unlikely to need to know the more obscure details. I hope for your sake that you fall into this camp, as specifications tend to get harder to read the deeper you go into them. Remember that unless you’re developing alone and just for yourself, you’re Listing 3.13 Retrieving and invoking a generic method with reflection 94 CHAPTER 3 Parameterized typing with generics unlikely to be the only one to work on your code. If you need features that are more complex than the ones demonstrated here, you should assume that anyone reading your code will need help to understand it. On the other hand, if you find that your co- workers don’t know about some of the topics we’ve covered so far, please feel free to direct them to the nearest bookshop… Our final main section of the chapter looks at some of the limitations of generics in C# and considers similar features in other languages. 3.5 Limitations of generics in C# and other languages There’s no doubt that generics contribute a great deal to C# in terms of expressive- ness, type safety, and performance. The feature has been carefully designed to cope with most of the tasks that C++ programmers typically used templates for, but without some of the accompanying disadvantages. But this isn’t to say limitations don’t exist. There are some problems that C++ templates solve with ease but that C# generics can’t help with. Similarly, though generics in Java are generally less powerful than in C#, there are some concepts that can be expressed in Java but that don’t have a C# equivalent. This section will take you through some of the most commonly encoun- tered weaknesses, and I’ll briefly compare the C#/.NET implementation of generics with C++ templates and Java generics. It’s important to stress that pointing out these snags doesn’t imply that they should’ve been avoided in the first place. In particular, I’m in no way saying that I could’ve done a better job! The language and platform designers have had to balance power with complexity (and the small matter of achieving both design and implemen- tation within a reasonable time scale). Most likely, you won’t encounter problems, and if you do, you’ll be able to work around them with the guidance given here. We’ll start with the answer to a question that almost everyone raises sooner or later: Why can’t I convert a List to a List? 3.5.1 Lack of generic variance In section 2.2.2, we looked at the covariance of arrays—the fact that an array of a refer- ence type can be viewed as an array of its base type, or an array of any of the interfaces it implements. There are actually two forms of this idea, called covariance and contra- variance, or collectively just variance. Generics don’t support this—they’re invariant. This is for the sake of type safety, as you’ll see, but it can be annoying. One thing I’d like to make clear to start with: C# 4 improves the generic variance situation somewhat. Many of the restrictions listed here do still apply though, and this section serves as a useful introduction to the idea of variance. We’ll see how C# 4 helps in chapter 13, but many of the clearest examples of generic variance rely on other new features from C# 3, including LINQ. Variance is also quite a complicated topic in itself, so it’s worth waiting until you’re comfortable with the rest of C# 2 and 3 before you tackle it. For the sake of readability, I won’t point out every place in this section that’s slightly different in C# 4…it’ll all become clear in chapter 13. 95Limitations of generics in C# and other languages WHY DON’T GENERICS SUPPORT COVARIANCE? Suppose you have two classes, Turtle and Cat, both of which derive from an abstract Animal class. In the code that follows, the array code (first block) is valid C# 2; the generic code (second block) isn’t. The compiler has no problem with the second line in either case, but the first line under Invalid causes the following error: error CS0029: Cannot implicitly convert type 'System.Collections.Generic.List' to 'System.Collections.Generic.List' This was a deliberate choice on the part of the framework and language designers. The obvious question to ask is why this is prohibited, and the answer lies in the second line. There’s nothing about the second line that should raise any suspicion. After all, List effectively has a method with the signature void Add(Animal value)— you should be able to put a Turtle into any list of animals, for instance. But the actual object referred to by animals is a Cat[] (in the code under Valid) or a List (under Invalid), both of which require that only references to instances of Cat (or fur- ther subclasses) are stored in them. Although the array version will compile, it’ll fail at execution time. This was deemed by the designers of generics to be worse than failing at compile time, which is reasonable—the whole point of static typing is to find out about errors before the code ever gets run. WHY ARE ARRAYS COVARIANT? Having answered the question about why generics are invariant, the next obvious step is to question why arrays are cova- riant. According to the Common Language Infrastructure Annotated Standard (Miller and Ragsdale, Addison-Wesley Professional, 2003), for the first version of .NET the designers wanted to reach as broad an audience as possible, which included being able to run code compiled from Java source. In other words, .NET has covariant arrays because Java has covariant arrays—despite this being a known wart in Java. So, that’s why things are the way they are—but why should you care, and how can you get around the restriction? WHERE COVARIANCE WOULD BE USEFUL The example I’ve given with a list is clearly problematic. You can add items to the list, which is where you lose the type safety in this case, and an add operation is an exam- ple of a value being used as an input into the API: the caller is supplying the value. What would happen if you limited yourself to getting values out? Valid (at compile time) Invalid Animal[] animals = new Cat[5]; animals[0] = new Turtle(); List animals = new List(); animals.Add(new Turtle()); 96 CHAPTER 3 Parameterized typing with generics The obvious examples of this are IEnumerator and (by association) IEnumerable. In fact, these are almost the canonical examples for generic covari- ance. Together they describe a sequence of values—all you know about the values you see is that each one will be compatible with T, such that you can always write T currentValue = iterator.Current; This uses the normal idea of compatibility—it would be fine for an IEnumerator to yield references to instances of Cat or Turtle, for example. There’s no way you can push values that are inappropriate for the actual sequence type, so you’d like to be able to treat an IEnumerator as an IEnumerator. Let’s con- sider an example of where that might be useful. Suppose you take the customary shape example for inheritance, but using an inter- face (IShape). Now consider another interface, IDrawing, that represents a drawing made up of shapes. You’ll have two concrete types of drawing—a MondrianDrawing (made of rectangles) and a SeuratDrawing (made of circles).9 Figure 3.4 shows the class hierarchies involved. Both drawing types need to implement the IDrawing interface, so they need to expose a property with this signature: IEnumerable Shapes { get; } But each drawing type would probably find it easier to maintain a more strongly typed list internally. For example, a Seurat drawing may include a field of type List. It’s useful for it to have this rather than a List so that if it needs to manipulate the circles in a circle-specific way, it can do so without casting. If you had a List, you could either return it directly or at least wrap it in a ReadOnlyCollection to prevent callers from messing with it via casting—the property implementation would be cheap and simple either way. But you can’t do that when your types don’t match up. You can’t convert from an IEnumerable to an IEnumerable. So what can you do? 9 If these names mean nothing to you, check out the artists’ Wikipedia entries (http://en.wikipedia.org/wiki/ Piet_Mondrian and http://en.wikipedia.org/wiki/Georges-Pierre_Seurat). They have special meanings to me for different reasons: Mondrian is also the name of a code review tool we used at Google, and Seurat is the eponymous George of Sunday in the Park with George—a wonderful musical by Stephen Sondheim. <> IShape Circle Rectangle <> IDrawing +Shapes: IEnumerable MondrianDrawing +rectangles: List SeuratDrawing +circles: List Figure 3.4 Interfaces for shapes and drawings, and two implementations of each 97Limitations of generics in C# and other languages There are a few options here:  Change the field type to List and just live with the casts. This isn’t pleasant, and it pretty much defeats the point of using generics.  Use the new features provided by C# 2 for implementing iterators, as you’ll see in chapter 6. This is a reasonable solution for this particular case, but only this case (where you’re dealing with IEnumerable).  Make each Shapes property implementation create a new copy of the list, possi- bly using List.ConvertAll for simplicity. Creating an independent copy of a collection is often the right thing to do in an API anyway, but it causes a lot of copying, which can be unnecessarily inefficient in many cases.  Make IDrawing generic, indicating the type of shapes in the drawing. Thus, MondrianDrawing would implement IDrawing, and SeuratDrawing would implement IDrawing. This is only viable when you own the interface.  Create a helper class to adapt one kind of IEnumerable into another: class EnumerableWrapper : IEnumerable where TOriginal : TWrapper Again, as this particular situation (IEnumerable) is special, you could get away with just a utility method. In fact, .NET 3.5 ships with two useful methods like this: Enumerable.Cast and Enumerable.OfType. They’re part of LINQ, and we’ll look at them in chapter 11. Although this is a special case, it’s probably the most common form of generic covariance you’ll come across. When you run into covariance issues, you may need to consider all of these options and anything else you can think of. It depends heavily on the exact nature of the situ- ation. Unfortunately, covariance isn’t the only problem you have to deal with. There’s also the matter of contravariance, which is like covariance in reverse. WHERE CONTRAVARIANCE WOULD BE USEFUL Contravariance feels slightly less intuitive than covariance, but it does make sense. With covariance, you were trying to convert from SomeType to SomeType (using IEnumerable for SomeType in the previous example). Contravari- ance is about converting the other way—from SomeType to SomeType . How can that be safe? Well, covariance is safe when SomeType only describes operations that return the type parameter—and contravariance is safe when SomeType only describes operations that accept the type parameter.10 The simplest example of a type that only uses its type parameter in an input posi- tion is IComparer, which is commonly used to sort collections. Let’s expand the IShape interface (which has been empty so far) to include an Area property. It’s now easy to write an implementation of IComparer that compares any two shapes by area. You’d then like to be able to write the following code: 10 You’ll see in chapter 13 that there’s slightly more to it than that, but that’s the general principle. 98 CHAPTER 3 Parameterized typing with generics IComparer areaComparer = new AreaComparer(); List circles = new List(); circles.Add(new Circle(Point.Empty, 20)); circles.Add(new Circle(Point.Empty, 10)); circles.Sort(areaComparer); That won’t work, though, because the Sort method on List effectively takes an IComparer. The fact that AreaComparer can compare any shape rather than just circles doesn’t impress the compiler at all. It considers IComparer and IComparer to be completely different types. Maddening, isn’t it? It would be nice if the Sort method had this signature instead: void Sort(IComparer comparer) where T:S Unfortunately, not only is that not the signature of Sort, but it can’t be—the constraint is invalid, because it’s a constraint on T instead of S. You want a conversion type con- straint but in the other direction, constraining the S to be somewhere up the inheri- tance tree of T instead of down. Given that this isn’t possible, what can you do? There are fewer options this time. First, you could revisit the idea of creating a generic helper class, as follows. class ComparisonHelper : IComparer where TDerived : TBase { private readonly IComparer comparer; public ComparisonHelper(IComparer comparer) { this.comparer = comparer; } public int Compare(TDerived x, TDerived y) { return comparer.Compare(x, y); } } This is an example of the adapter pattern at work, although instead of adapting one interface to a completely different one, you’re just adapting from IComparer to IComparer. You just remember the original comparer providing the real logic to compare items of the base type C and then call it when you’re asked to compare items of the derived type D. The fact that no casts are involved (not even hidden ones) should give you some confidence: this helper is completely type-safe. You’re able to call the base comparer due to an implicit conversion being available from TDerived to TBase, which you required with a type constraint B. The second option is to make the area-comparison class generic with a conversion type constraint, so it can compare any two values of the same type, as long as that type implements IShape. For the sake of simplicity in the situation where you really don’t Listing 3.14 Working around the lack of contravariance with a helper INVALID Constrains type parameter appropriately B Uses implicit conversions to call comparerC Uses implicit conversions to call comparer D 99Limitations of generics in C# and other languages need this functionality, you could keep the nongeneric class by just making it derive from the generic one: class AreaComparer : IComparer where T: IShape class AreaComparer : AreaComparer Of course, you can only do this when you’re able to change the comparison class. This can be an effective solution, but it still feels unnatural—why should you have to con- struct the comparer in various ways for different types when it’s not going to behave any differently? Why should you have to derive from the class to simplify things when you’re not actually specializing the behavior? Note that the various options for both covariance and contravariance use more generics and constraints to express the interface in a more general manner, or to pro- vide generic helper classes. I know that adding a constraint makes it sound less gen- eral, but the generality is added by first making the type or method generic. When you run into a problem like this, adding a level of genericity somewhere with an appropri- ate constraint should be the first option to consider. Generic methods (rather than generic types) are often helpful here, as type inference can make the lack of variance invisible to the naked eye. This is particularly true in C# 3, which has stronger type inference capabilities than C# 2. This limitation is a very common cause of questions on C# discussion sites. The remaining issues are either relatively academic or affect only a moderate subset of the development community. The next one mostly affects those who do a lot of calcula- tions (usually scientific or financial) in their work. 3.5.2 Lack of operator constraints or a “numeric” constraint C# isn’t without its downsides when it comes to heavily mathematical code. The need to explicitly use the Math class for every operation beyond the simplest arithmetic and the lack of C-style typedefs to allow the data representation used throughout a pro- gram to be easily changed have always been raised by the scientific community as bar- riers to C#’s adoption. Generics weren’t likely to fully solve either of those issues, but there’s a common problem that stops generics from helping as much as they could have. Consider this (illegal) generic method: public T FindMean(IEnumerable data) { T sum = default(T); int count = 0; foreach (T datum in data) { sum += datum; count++; } return sum / count; } INVALID 100 CHAPTER 3 Parameterized typing with generics Obviously that could never work for all types of data—what could it mean to add one Exception to another, for instance? Clearly a constraint of some kind is called for… something that can express what you need to be able to do: add two instances of T together, and divide a T by an integer. If that were available, even if it were limited to built-in types, you could write generic algorithms that wouldn’t care whether they were working on an int, a long, a double, a decimal, and so forth. Limiting it to the built-in types would’ve been disappointing, but better than noth- ing. The ideal solution would have to also allow user-defined types to act in a numeric capacity, so you could define a Complex type to handle complex numbers, for instance.11 That complex number could then store each of its components in a generic way as well, so you could have a Complex, a Complex, and so on. Two related (but hypothetical) solutions present themselves. One would be to allow constraints on operators, so you could write a set of constraints such as these (currently invalid) ones: where T:T operator+ (T, T), T operator/ (T, int) This would require that T have the operations you need in the earlier code. The other solution would be to define a few operators and perhaps conversions that must be sup- ported in order for a type to meet the extra constraint—you could make it the “numeric constraint” written where T : numeric. One problem with both of these options is that they can’t be expressed as normal interfaces, because operator overloading is performed with static members, which can’t be used to implement interfaces. I find the idea of static interfaces appealing: interfaces that only declare static members, including methods, operators, and con- structors. Such static interfaces would only be useful within type constraints, but they’d present a type-safe generic way of accessing static members. This is just blue sky thinking, though (see my blog post on the topic for more details: http://mng.bz/ 3Rk3). I don’t know of any plans to include this in a future version of C#. The two neatest workarounds for this problem to date require later versions of .NET: one designed by Marc Gravell (http://mng.bz/9m8i) uses expression trees (which you’ll meet in chapter 9) to build dynamic methods; the other uses the dynamic features of C# 4. You’ll see an example of the latter in chapter 14. But, as you can tell by the descriptions, both of these are dynamic—you have to wait until execu- tion time to see whether your code will work with a particular type. There are a few workarounds that still use static typing, but they have other disadvantages (surpris- ingly enough, they can sometimes be slower than the dynamic code). The two limitations we’ve looked at so far have been quite practical—they’ve been issues you may well run into during actual development. But if you’re generally curi- ous like I am, you may also be asking yourself about other limitations that don’t neces- sarily slow down development but are intellectual curiosities. In particular, why are generics limited to types and methods? 11 This is assuming you’re not using .NET 4 or higher, of course, because then you could use System .Numerics.Complex. 101Limitations of generics in C# and other languages 3.5.3 Lack of generic properties, indexers, and other member types We’ve looked at generic types (classes, structs, delegates, and interfaces) and generic methods. There are plenty of other members that could be parameterized, but there are no generic properties, indexers, operators, constructors, finalizers, or events. First, let’s be precise about what we mean here: clearly an indexer can have a return type that’s a type parameter—List is an obvious example. KeyValue- Pair provides similar examples for properties. What you can’t have is an indexer or property (or any of the other members in that list) with extra type parameters. Leaving the possible syntax of declaration aside for the minute, let’s look at how these members might have to be called: SomeClass instance = new SomeClass("x"); int x = instance.SomeProperty; byte y = instance.SomeIndexer["key"]; instance.Click += ByteHandler; instance = instance + instance; I hope you’ll agree that all of those look somewhat silly. Finalizers can’t even be called explicitly from C# code, which is why there isn’t a line for them. The fact that you can’t do any of these isn’t going to cause significant problems anywhere, as far as I can see—it’s just worth being aware of this as an academic limitation. The member where this restriction is most irritating is probably the constructor. A static generic method in the class is a good workaround for this, though, and the sam- ple generic constructor syntax shown previously with two lists of type arguments is horrific. These are by no means the only limitations of C# generics, but I believe they’re the ones that you’re most likely to run up against, either in your daily work, in community conversations, or when idly considering the feature as a whole. In the next two sec- tions, we’ll look at how some aspects of these aren’t issues in the two other languages whose features are most commonly compared with C#’s generics: C++ (with tem- plates) and Java (with generics, as of Java 5). We’ll tackle C++ first. 3.5.4 Comparison with C++ templates C++ templates are a bit like macros taken to an extreme level. They’re incredibly pow- erful, but there are costs associated with them both in terms of code bloat and ease of understanding. When a template is used in C++, the code is compiled for that particular set of tem- plate arguments, as if the template arguments were in the source code. This means that there’s not as much need for constraints, because the compiler will check what you’re allowed to do with the type while it’s compiling the code for this particular set of template arguments. The C++ standards committee has recognized that constraints are still useful, though. Constraints were included and then removed from C++11 (the latest version of C++) but they may yet see the light of day, under the name of concepts. INVALID 102 CHAPTER 3 Parameterized typing with generics The C++ compiler is smart enough to compile the code only once for any given set of template arguments, but it isn’t able to share code in the way that the CLR does with reference types. That lack of sharing does have its benefits, though—it allows type- specific optimizations, such as inlining method calls for some type parameters but not others, from the same template. It also means that overload resolution can be per- formed separately for each set of type parameters, rather than just once based solely on the limited knowledge the C# compiler has due to any constraints present. Don’t forget that with normal C++ there’s only one compilation involved, rather than the “compile to IL” and then “JIT compile to native code” model of .NET. A program using a standard template in 10 different ways will include the code 10 times in a C++ program. A similar program in C# using a generic type from the framework in 10 different ways won’t include the code for the generic type at all—it’ll refer to it, and the JIT will compile as many different versions as required (as described in section 3.4.2) at execution time. One significant feature that C++ templates have over C# generics is that the tem- plate arguments don’t have to be type names. Variable names, function names, and constant expressions can be used as well. A common example of this is a buffer type that has the size of the buffer as one of the template arguments—a buffer will always be a buffer of 20 integers, and a buffer will always be a buffer of 35 doubles. This ability is crucial to template metaprogramming (see the Wikipedia article, http://en.wikipedia.org/wiki/Template_metaprogramming), which is an advanced C++ technique, the very idea of which scares me but that can be powerful in the hands of experts. C++ templates are more flexible in other ways too. They don’t suffer from the lack of operator constraints described in section 3.5.2, and there are a few other restrictions that don’t exist in C++: you can derive a class from one of its type parameters, and you can specialize a template for a particular set of type arguments. The latter ability allows the template author to write general code to be used when there’s no more knowledge available, and specific (often highly optimized) code for particular types. The same variance issues of .NET generics exist in C++ templates as well. An exam- ple given by Bjarne Stroustrup (the inventor of C++) is that there are no implicit con- versions between vector and vector with similar reasoning—in this case, it might allow you to put a square peg in a round hole. For further details on C++ templates, I recommend Stroustrup’s The C++ Program- ming Language, 3rd edition (Addison-Wesley Professional, 1997). It’s not always the easiest book to follow, but the templates chapter is fairly clear (once you get your mind around C++ terminology and syntax). For more comparisons with .NET gener- ics, look at the blog post by the Visual C++ team on this topic (http://mng.bz/En13). The other obvious language to compare with C# in terms of generics is Java, which introduced the feature into the mainstream language for the 1.5 release12 several years after other projects had created Java-like languages that supported generics. 12 Or 5.0, depending on which numbering system you use. Don’t get me started. 103Limitations of generics in C# and other languages 3.5.5 Comparison with Java generics Where C++ includes more of the template in the generated code than C# does, Java includes less. In fact, the Java runtime doesn’t know about generics at all. The Java byte- code (roughly equivalent to IL) for a generic type includes some extra metadata to say that it’s generic, but after compilation the calling code doesn’t have much to indicate that generics were involved at all, and an instance of a generic type only knows about the nongeneric side of itself. For example, an instance of HashSet doesn’t know whether it was created as a HashSet or a HashSet. The compiler effectively adds casts where necessary and performs more sanity checking. Here’s an example—first the generic Java code: ArrayList strings = new ArrayList(); strings.add("hello"); String entry = strings.get(0); strings.add(new Object()); And here’s the equivalent nongeneric code: ArrayList strings = new ArrayList(); strings.add("hello"); String entry = (String) strings.get(0); strings.add(new Object()); They would generate the same Java bytecode, except for the last line, which is valid in the nongeneric case but will be caught by the compiler as an error in the generic version. You can use a generic type as a raw type, which is similar to using java.lang.Object for each of the type arguments. This rewriting—and loss of infor- mation—is called type erasure. Java doesn’t have user-defined value types, but you can’t even use the built-in ones as type arguments. Instead, you have to use the boxed ver- sions—ArrayList for a list of integers, for example. You’ll be forgiven for thinking this is all a bit disappointing compared with gener- ics in C#, but there are some nice features of Java generics too:  The virtual machine doesn’t know anything about generics, so you can use code compiled using generics on an older version, as long as you don’t use any classes or methods that aren’t present on the old version. Versioning in .NET is much stricter in general—for each assembly you reference, you can specify whether the version number has to match exactly. In addition, code built to run on the 2.0 CLR won’t run on .NET 1.1.  You don’t need to learn a new set of classes to use Java generics—where a non- generic developer would use ArrayList, a generic developer just uses Array- List. Existing classes can be upgraded to generic versions reasonably easily.  The previous feature has been utilized quite effectively with the reflection sys- tem—java.lang.Class (the equivalent of System.Type) is generic, which allows compile-time type safety to be extended to cover many situations involv- ing reflection. In some other situations it’s a pain, though. 104 CHAPTER 3 Parameterized typing with generics  Java has support for generic variance using wildcards. For instance, ArrayList can be read as “this is an ArrayList of some type that derives from Base, but we don’t know which exact type.” When we discuss C# 4’s support for generic variance in chapter 13, we’ll revisit this with a short example. My personal opinion is that .NET generics are superior in almost every respect, although when I run into covariance/contravariance issues, I often wish I had wild- cards. C# 4’s limited generic variance improves this somewhat, but there are still times when the variance Java model works better. Java with generics is still much better than Java without generics, but there are no performance benefits and the safety only applies at compile time. 3.6 Summary Phew! It’s a good thing generics are simpler to use in reality than they are to describe. Although they can get complicated, they’re widely regarded as the most important addition to C# 2 and they’re incredibly useful. The worst thing about writing code using generics is that if you ever have to go back to C# 1, you’ll miss them terribly. (Fortunately that’s becoming increasingly unlikely, of course.) In this chapter, I haven’t tried to cover every detail of what is and isn’t allowed when using generics—that’s the job of the language specification, and it makes for dry reading. Instead, I’ve aimed for a practical approach, providing the information you’ll need in everyday use, with a smattering of theory for the sake of academic interest. We’ve looked at the three main benefits of generics: compile-time type safety, per- formance, and code expressiveness. Being able to get the IDE and compiler to validate your code early is certainly a good thing, but it’s arguable that more is to be gained from tools providing intelligent options based on the types involved than from the actual safety aspect. Performance is improved most radically when it comes to value types, which no longer need to be boxed and unboxed when they’re used in strongly typed generic APIs, particularly the generic collection types provided in .NET 2.0. Performance with reference types is usually improved but only slightly. Your code is able to express its intention more clearly using generics—instead of a comment or a long variable name being required to describe exactly what types are involved, the details of the type itself can do the work. Comments and variable names can often become inaccurate over time, as they can be forgotten when the code is changed, but the type information is correct by definition. Generics aren’t capable of doing everything you might sometimes like them to do, and I’ve covered some of their limitations in the chapter, but if you truly embrace C# 2 and the generic types within the .NET 2.0 Framework, you’ll come across good uses for them incredibly frequently in your code. This topic will come up time and time again in future chapters, as other new fea- tures build on this key one. Indeed, the subject of the next chapter would be very dif- ferent without generics—we’ll look at nullable types, as implemented by Nullable. 105 Saying nothing with nullable types Nullity is a concept that has provoked debate over the years. Is a null reference a value, or the absence of a value? Is “nothing” a “something”? Should languages sup- port the concept of nullity at all, or should it be represented in other patterns? In this chapter, I’ll try to stay more practical than philosophical. First we’ll look at why there’s a problem at all—why you can’t set a value type variable to null in C# 1 and what the traditional alternatives have been. After that, I’ll introduce you to our knight in shining armor—System.Nullable—and then we’ll look at how C# 2 makes working with nullable types simple and compact. Like generics, nul- lable types sometimes have uses beyond what you might expect, and we’ll look at a few examples of these at the end of the chapter. So, when is a value not a value? Let’s find out. This chapter covers  Reasons for using null values  Framework and runtime support for nullable types  Language support in C# 2 for nullable types  Patterns using nullable types 106 CHAPTER 4 Saying nothing with nullable types 4.1 What do you do when you just don’t have a value? The C# and .NET designers don’t add features just for kicks. There has to be a real, significant problem that needs fixing before they’ll go as far as changing C# as a lan- guage or .NET at the platform level. In this case, the problem is best summed up in one of the most frequently asked questions in C# and .NET discussion groups: I need to set my DateTime variable to null, but the compiler won’t let me. What should I do?1 It’s a question that comes up fairly naturally—an example might be in an e-commerce application where users are looking at their account history. If an order has been placed but not delivered, there may be a purchase date but no dispatch date, so how would you represent that in a type that’s meant to provide the order details? Before C# 2, the answer to the question usually came in two parts: an explanation of why you couldn’t use null in the first place, and a list of which options were avail- able. Nowadays the answer would usually explain nullable types instead, but it’s worth looking at the C# 1 options to understand where the problem comes from. 4.1.1 Why value type variables can’t be null As you saw in chapter 2, the value of a reference type variable is a reference, and the value of a value type variable is the real data itself. A non-null reference is a way of get- ting at an object, but null acts as a special value that means I don’t refer to any object. If you want to think of references as being like URLs, null is (very roughly speak- ing) the reference equivalent of about:blank. It’s represented as all zeroes in memory (which is why it’s the default value for all reference types—clearing a whole block of memory is cheap, so that’s the way objects are initialized), but it’s still basically stored in the same way as other references. There’s no extra bit hidden somewhere for each reference type variable. That means you can’t use the “all zeroes” value for a real refer- ence, but that’s okay—your memory is going to run out long before you have that many live objects anyway. This is the key to why null isn’t a valid value type value. Let’s consider the byte type as a familiar one that’s easy to think about. The value of a variable of type byte is stored in a single byte—it may be padded for alignment purposes, but the value itself is conceptually only made up of one byte. You’ve got to be able to store the values 0–255 in that variable; otherwise it’s useless for reading arbitrary binary data. With the 256 normal values and one null value, you’d have to cope with a total of 257 values, and there’s no way of squeezing that many values into a single byte. The designers could’ve decided that every value type would have an extra flag bit somewhere determining whether a value was null or contained real data, but the memory usage implications are horrible, not to mention the fact that you’d have to check the flag every time you wanted to use the value. In a nutshell, with value types you often care about having the whole range of possible bit patterns available as real 1 It’s almost always DateTime rather than any other value type. I’m not entirely sure why—it’s as if developers inherently understand why a byte shouldn’t be null, but feel that dates are more inherently nullable. 107What do you do when you just don’t have a value? values, whereas with reference types it’s okay to lose one potential value in order to gain the benefits of making the null reference available. That’s the usual situation. Now why would you want to be able to represent null for a value type anyway? The most common reason is simply because databases typi- cally support NULL as a value for every type (unless you specifically make the field non- nullable), so you can have nullable character data, nullable integers, nullable Bool- eans—the whole works. When you fetch data from a database, it’s generally not a good idea to lose information, so you want to be able to represent the nullity of whatever you read, somehow. That just moves the question one step further on, though. Why do databases allow null values for dates, integers, and the like? Null values are typically used for unknown or missing values, such as the dispatch date in the earlier e-commerce example. Nullity represents an absence of definite information, which can be impor- tant in many situations. Indeed, there doesn’t have to be a database involved for nul- lable value types to be useful; that’s just the scenario where developers typically encounter the problem first. That brings us to options for representing null values in C# 1. 4.1.2 Patterns for representing null values in C# 1 There are three basic patterns commonly used to get around the lack of nullable value types in C# 1. Each has its pros and cons—mostly cons—and all of them are fairly unsatisfying. But they’re worth knowing, partly to more fully appreciate the ben- efits of the integrated solution in C# 2. PATTERN 1: THE MAGIC VALUE The first pattern is to sacrifice one value to represent a null value. This tends to be used as the solution for DateTime; few people expect their databases to actually con- tain dates in AD 1, so DateTime.MinValue can be used as a convenient magic value without losing any useful data. In other words, it goes against the line of reasoning I gave earlier, which assumes that every possible value needs to be available for normal purposes. The semantic meaning of such a null value will vary from application to application—it may mean that the user hasn’t entered the value into a form yet, or that it’s not required for that record, for example. The good news is that using a magic value doesn’t waste any memory or require any new types. But it does rely on you picking an appropriate value that you’ll never want to use for real data. Also, it’s basically inelegant. It just doesn’t feel right. If you ever find yourself needing to go down this path, you should at least use a constant (or static read-only value for types that can’t be expressed as constants) to represent the magic value—comparisons with DateTime.MinValue everywhere, for instance, don’t express the meaning of the magic value. Additionally, it’s easy to accidentally use the magic value as if it were a normal, meaningful value—neither the compiler nor the runtime will help you spot the error. In contrast, most of the other solutions presented here 108 CHAPTER 4 Saying nothing with nullable types (including the one in C# 2) would result in either a compilation error or an exception at execution time, depending on the exact situation. The magic value pattern is deeply embedded in computing in the form of IEEE-754 binary floating-point types such as float and double. These go further than the idea of a single value representing this isn’t really a number—there are many bit patterns that are classified as not-a-number (NaN), as well as values for positive and negative infin- ity. I suspect few programmers (myself included) are as cautious around these values as we should be, which is another indication of the pattern’s shortcomings. ADO.NET has a variation on this pattern where the same magic value— DBNull.Value—is used for all null values, regardless of the type. In this case, an extra value and indeed an extra type have been introduced to indicate when a database has returned null. But it’s only applicable where compile-time type safety isn’t important (in other words, when you’re happy to use object and cast after testing for nullity), and again it doesn’t feel quite right. In fact, it’s a mixture of the magic value pattern and the reference type wrapper pattern, which we’ll look at next. PATTERN 2: A REFERENCE TYPE WRAPPER The second solution can take two forms. The simpler one is to use object as the vari- able type, boxing and unboxing values as necessary. The more complex (and more appealing) form is to have a reference type for each value type you need in a nullable form, containing a single instance variable of that value type, and with implicit conver- sion operators to and from the value type. With generics, you could do this in one generic type, but if you’re using C# 2 anyway, you might as well use the nullable types described in this chapter instead. If you’re stuck in C# 1, you have to create extra source code for each type you want to wrap. This isn’t hard to put in the form of a template for automatic code generation, but it’s still a burden that’s best avoided if possible. Both of these forms have the problem that though they allow you to use null directly, they require objects to be created on the heap, which can lead to garbage col- lection pressure if you need to use this approach frequently and adds memory use due to the overhead associated with objects. For the more complex solution, you could make the reference type mutable, which may reduce the number of instances you need to create, but it could also make for some unintuitive code. PATTERN 3: AN EXTRA BOOLEAN FLAG The final pattern involves a normal value type value and another value—a Boolean flag—indicating whether the value is “real” or whether it should be disregarded. Again, there are two ways of implementing this solution. Either you could maintain two separate variables in the code that uses the value, or you could encapsulate the value-plus-flag into another value type. This latter solution is quite similar to the more complicated reference type idea described earlier, except that you avoid the garbage collection issue by using a value type and indicate nullity within the encapsulated value rather than with a null refer- ence. The downside of having to create a new one of these types for every value type 109System.Nullable and System.Nullable you wish to handle is the same, though. Also, if the value is ever boxed for some rea- son, it’ll be boxed in the normal way whether it’s considered to be null or not. The last pattern (in the more encapsulated form) is effectively how nullable types work in C# 2, although the new features of the framework, CLR, and language all com- bine to provide a solution that’s significantly neater than anything that was possible in C# 1. The next section deals with the support provided by the framework and the CLR in .NET 2: if C# 2 only supported generics, most of section 4.2 would still be relevant and the feature would still work and be useful. But C# 2 provides extra syntactic sugar to make it even better—that’s the subject of section 4.3. 4.2 System.Nullable and System.Nullable The core structure at the heart of nullable types is the System.Nullable struct. In addition, the System.Nullable static class provides utility methods that occasionally make nullable types easier to work with. (From now on I’ll leave out the namespace, to make life simpler.) We’ll look at both of these types in turn, and for this section I’ll avoid any extra features provided by the language, so you’ll be able to understand what’s going on in the IL code when we do look at the shorthand provided by C# 2. 4.2.1 Introducing Nullable As you can tell by its name, Nullable is a generic type. The type parameter T has a value type constraint, so you can’t use Nullable, for example. As I men- tioned in section 3.3.1 this also means you can’t use another nullable type as the argument, so Nullable> is forbidden, even though Nullable is a value type in every other way. The type of T for any particular nullable type is called the underlying type of that nullable type. For example, the underlying type of Nullable is int. The most important parts of Nullable are its properties, HasValue and Value. They do the obvious: Value represents the non-nullable value (the real one, if you will), when there is one, and throws an InvalidOperationException if (conceptually) there’s no real value. HasValue is a Boolean property indicating whether there’s a real value or whether the instance should be regarded as null. For now, I’ll talk about an “instance with a value” or an “instance without a value,” which mean an instance where the HasValue property returns true or false, respectively. These properties are backed by simple fields in the obvious way. Figure 4.1 shows instances of Nullable representing (from left to right) no value, 0, and 5. Remember that Nullable is still a value type, so if you have a variable of type Nullable, the variable’s value will directly contain a bool and an int—it won’t be a reference to a separate object. false hasValue 0 value true hasValue 0 value true hasValue 5 value No value (null) Value of 0 Value of 5 Figure 4.1 Sample values of Nullable 110 CHAPTER 4 Saying nothing with nullable types Now that you know what the properties should achieve, let’s look at how you can cre- ate an instance of the type. Nullable has two constructors: the default one (creat- ing an instance without a value) and one taking an instance of T as the value. Once an instance has been constructed, it’s immutable. Nullable introduces a single new method, GetValueOrDefault, which has two overloads. Both return the value of the instance if there is one, or a default value oth- erwise. One overload doesn’t have any parameters (in which case the default value of the underlying type is used), and the other allows you to specify the default value to return if necessary. The other methods implemented by Nullable all override existing methods: GetHashCode, ToString, and Equals. GetHashCode returns 0 if the instance doesn’t have a value, or the result of calling GetHashCode on the value if there is one. ToString returns an empty string if there isn’t a value, or the result of calling ToString on the value if there is. Equals is slightly more complicated—we’ll come back to it when we’ve discussed boxing. Finally, two conversions are provided by the framework. First, there’s an implicit conversion from T to Nullable. This always results in an instance where HasValue returns true. Likewise, there’s an explicit conversion from Nullable to T, which behaves exactly the same as the Value property, including throwing an exception when there’s no real value to return. WRAPPING AND UNWRAPPING The C# specification names the process of con- verting an instance of T to an instance of Nullable wrapping, with the obvi- ous opposite process being called unwrapping. The specification defines these terms with reference to the constructor taking a parameter and the Value property, respectively. Indeed, these calls are generated by the C# code even when it otherwise looks as if you’re using the conversions provided by the Value types and mutability A type is said to be immutable if it’s designed so that an instance can’t be changed after it’s been constructed. Immutable types often lead to a cleaner design than you’d get if you had to keep track of what might be changing shared values—partic- ularly among different threads. Immutability is particularly important for value types; they should almost always be immutable. Most value types in the framework are immutable, but there are some commonly used exceptions—in particular, the Point structures for both Windows Forms and Windows Presentation Foundation are mutable. If you need a way of basing one value on another, follow the lead of DateTime and TimeSpan—provide methods and operators that return a new value rather than mod- ifying an existing one. This avoids all kinds of subtle bugs, including situations where you may appear to be changing something, but you’re actually changing a copy. Just say No to mutable value types. 111System.Nullable and System.Nullable framework. The results are the same either way, though. For the rest of this chapter, I won’t distinguish between the two implementations available. Before we go any further, let’s see all this in action. The following listing shows every- thing you can do with Nullable directly, leaving Equals aside for the moment. static void Display(Nullable x) { Console.WriteLine("HasValue: {0}", x.HasValue); if (x.HasValue) { Console.WriteLine("Value: {0}", x.Value); Console.WriteLine("Explicit conversion: {0}", (int)x); } Console.WriteLine("GetValueOrDefault(): {0}", x.GetValueOrDefault()); Console.WriteLine("GetValueOrDefault(10): {0}", x.GetValueOrDefault(10)); Console.WriteLine("ToString(): \"{0}\"", x.ToString()); Console.WriteLine("GetHashCode(): {0}", x.GetHashCode()); Console.WriteLine(); } ... Nullable x = 5; x = new Nullable(5); Console.WriteLine("Instance with value:"); Display(x); x = new Nullable(); Console.WriteLine("Instance without value:"); Display(x); In listing 4.1, you first use the two different ways (in terms of C# source code) of wrap- ping a value of the underlying type C, and then you use various different members on the instance B. Next, you create an instance that doesn’t have a value D and use the same members in the same order, just omitting the Value property and the explicit conversion to int, because these would throw exceptions. The output of listing 4.1 is as follows: Instance with value: HasValue: True Value: 5 Explicit conversion: 5 GetValueOrDefault(): 5 GetValueOrDefault(10): 5 ToString(): "5" GetHashCode(): 5 Instance without value: HasValue: False GetValueOrDefault(): 0 GetValueOrDefault(10): 10 Listing 4.1 Using various members of Nullable Displays diagnosticsB Wraps value of 5 C Constructs instance without valueD 112 CHAPTER 4 Saying nothing with nullable types ToString(): "" GetHashCode(): 0 So far, you could probably have predicted all of the results by looking at the members provided by Nullable. When it comes to boxing and unboxing, though, there’s special behavior to make nullable types behave how you’d really like them to behave, rather than how they’d behave if you slavishly followed the normal boxing rules. 4.2.2 Boxing Nullable and unboxing It’s important to remember that Nullable is a struct—a value type. This means that if you want to convert it to a reference type (object is the most obvious example), you’ll need to box it. It’s only with respect to boxing and unboxing that the CLR has any special behavior regarding nullable types—the rest is standard generics, conver- sions, method calls, and so forth. In fact, the behavior was only changed shortly before the release of .NET 2.0, as the result of community requests. In the preview releases, nullable value types were boxed just like any other value types. An instance of Nullable is boxed to either a null reference (if it doesn’t have a value) or a boxed value of T (if it does), as shown in figure 4.2. It never boxes to a “boxed nullable int”—there’s no such type. You can unbox from a boxed value either to its normal type or to the corre- sponding nullable type. Unboxing a null reference will throw a NullReference- Exception if you unbox to the normal type, but will unbox to an instance without a value if you unbox to the appropriate nullable type. This behavior is shown in the following listing. Nullable nullable = 5; object boxed = nullable; Console.WriteLine(boxed.GetType()); int normal = (int)boxed; Console.WriteLine(normal); nullable = (Nullable)boxed; Console.WriteLine(nullable); nullable = new Nullable(); boxed = nullable; Listing 4.2 Boxing and unboxing behavior of nullable types Boxes nullable with value Unboxes to non- nullable variable Unboxes to nullable variable Boxes nullable without value false hasValue 0 false 0 value null reference true hasValue 5 value Boxing Nullable 5 Boxed int reference Boxing Nullable Figure 4.2 Results of boxing an instance without a value (top) and with a value (bottom) 113System.Nullable and System.Nullable Console.WriteLine(boxed == null); nullable = (Nullable)boxed; Console.WriteLine(nullable.HasValue); The output of listing 4.2 shows the type of the boxed value as System.Int32 (not System.Nullable). This confirms that you can retrieve the value by unboxing to either int or to Nullable. Finally, the output demonstrates that you can box from a nullable instance without a value to a null reference and success- fully unbox again to another valueless nullable instance. If you’d tried unboxing the last value of boxed to a non-nullable int, the program would’ve blown up with a Null- ReferenceException. Now that you understand the behavior of boxing and unboxing, we can begin to tackle the behavior of Nullable.Equals. 4.2.3 Equality of Nullable instances Nullable overrides object.Equals(object) but doesn’t introduce any equality operators or provide an Equals(Nullable) method. Because the framework has supplied the basic building blocks, languages can add extra functionality on top, including making existing operators work as you’d expect them to. You’ll see the details of that in section 4.3.3, but the basic equality, as defined by the vanilla Equals method, follows these rules for a call to first.Equals(second):  If first has no value and second is null, they’re equal.  If first has no value and second isn’t null, they aren’t equal.  If first has a value and second is null, they aren’t equal.  Otherwise, they’re equal if first’s value is equal to second. Note that you don’t have to consider the case where second is another Nullable because the rules of boxing prohibit that situation. The type of second is object, so in order to be a Nullable, it would have to be boxed, and as you’ve just seen, boxing a nullable instance creates a box of the non-nullable type or returns a null reference. Initially, the first rule may appear to be breaking the contract for object .Equals(object), which insists that x.Equals(null) returns false—but that’s only when x is a non-null reference. Again, due to the boxing behavior, Nullable’s implementation will never be called via a reference. The rules are mostly consistent with the rules of equality elsewhere in .NET, so you can use nullable instances as keys for dictionaries and any other situations where you need equality. Just don’t expect equality to differentiate between a non-nullable instance and a nullable instance with a value—it’s been carefully set up so that those two cases are treated the same way as each other. That covers the Nullable structure itself, but it has a shadowy partner: the Nullable class. Unboxes to nullable variable 114 CHAPTER 4 Saying nothing with nullable types 4.2.4 Support from the nongeneric Nullable class The System.Nullable struct does almost everything you want it to, but it gets help from the System.Nullable class. This is a static class—it only contains static methods, and you can’t create an instance of it.2 In fact, everything it does could’ve been done equally well by other types, and if Microsoft had shown more foresight, the Nullable class might not have even existed—which would’ve saved some confusion over what the two types are there for. But this accident of history has three methods to its name, and they’re still useful. The first two are comparison methods: public static int Compare(Nullable n1, Nullable n2) public static bool Equals(Nullable n1, Nullable n2) Compare uses Comparer.Default to compare the two underlying values (if they exist), and Equals uses EqualityComparer.Default. When presented with instances with no values, the results returned from each method comply with the .NET conventions of nulls comparing equal to each other and less than anything else. Both of these methods could happily be part of Nullable as static but nonge- neric methods. The one small advantage of having them as generic methods in a non- generic type is that generic type inference can be applied, so you’ll rarely need to explicitly specify the type parameter. The final method of System.Nullable isn’t generic—it couldn’t be. Its signature is as follows: public static Type GetUnderlyingType(Type nullableType) If the parameter is a nullable type, the method returns its underlying type; otherwise it returns null. The reason this couldn’t be a generic method is that if you knew the underlying type to start with, you wouldn’t have to call it. You’ve now seen what the framework and the CLR provide to support nullable types—but C# 2 adds language features to make life a lot more pleasant. 4.3 C# 2’s syntactic sugar for nullable types So far you’ve seen nullable types doing their jobs, but the examples haven’t been par- ticularly pretty to look at. Admittedly it’s clear that you’re using nullable types when you have to type Nullable<> around the name of the type you’re interested in, but that makes the nullability more prominent than the underlying type, which is usually not a good idea. In addition, the very name nullable suggests that you should be able to assign null to a variable of a nullable type, and you haven’t seen that—you’ve always used the default constructor of the type. In this section, we’ll look at how C# 2 deals with these issues and others. 2 You’ll learn more about static classes in chapter 7. 115C# 2’s syntactic sugar for nullable types Before we get into the details of what C# 2 provides as a language, there’s one def- inition I can finally introduce. The null value of a nullable value type is the value where HasValue returns false—or an “instance without a value,” as I referred to it in section 4.2. I didn’t use the term before because it’s specific to C#. The CLI specifica- tion doesn’t mention it, and the documentation for Nullable itself doesn’t men- tion it. I’ve honored that difference by waiting until we’re specifically talking about C# 2 before introducing the term. The term also applies to reference types: the null value of a reference type is simply the null reference you’re familiar with from C# 1. NULLABLE TYPE VERSUS NULLABLE VALUE TYPE In the C# language specifica- tion, nullable type is used to mean any type with a null value—so any reference type, or any Nullable. You may have noticed that I’ve been using this term as if it were synonymous with nullable value type (which obviously doesn’t include reference types). Although I’m usually a huge pedant when it comes to terminology, if I’d used “nullable value type” everywhere in this chapter, it would’ve been horrible to read. You should also expect “nullable type” to be used ambiguously in the real world: it’s probably more common to use it when describing Nullable than in the sense described in the specification. With that out of the way, let’s see what features C# 2 gives us, starting by reducing the clutter in our code. 4.3.1 The ? modifier There are some elements of syntax that may be unfamiliar at first but that have an appropriate feel to them. The conditional operator (a ? b : c) is one of them for me— it asks a question and then has two corresponding answers. In the same way, the ? modifier for nullable types just feels right. The ? modifier is a shorthand way of specifying a nullable type, so instead of using Nullable , you can use byte? throughout your code. The two are interchange- able and compile to exactly the same IL, so you can mix and match them if you want to—but on behalf of whoever reads your code next, I urge you to pick one way or the other and use it consistently. The following listing is exactly equivalent to listing 4.2 but uses the ? modifier, as shown in bold. int? nullable = 5; object boxed = nullable; Console.WriteLine(boxed.GetType()); int normal = (int)boxed; Console.WriteLine(normal); nullable = (int?)boxed; Console.WriteLine(nullable); nullable = new int?(); Listing 4.3 The same code as 4.2 but using the ? modifier 116 CHAPTER 4 Saying nothing with nullable types boxed = nullable; Console.WriteLine(boxed == null); nullable = (int?)boxed; Console.WriteLine(nullable.HasValue); I won’t go through what the code does or how it does it, because the result is exactly the same as in listing 4.2. The two listings compile down to the same IL—they simply use different syntax, just as int is interchangeable with System.Int32. You can use the shorthand version everywhere, including in method signatures, typeof expressions, casts, and the like. The reason I feel the modifier is well chosen is that it adds an air of uncertainty to the nature of the variable. Does the variable nullable in listing 4.3 have an integer value? Well, at any particular time it might, or it might be the null value. From now on, I’ll use the ? modifier in all the examples—it’s neater, and it’s argu- ably the idiomatic way to use nullable types in C#. But you may feel that it’s too easy to miss when reading the code, in which case there’s nothing to stop you from using the longer syntax. You might want to compare the listings in this and the previous section to see which you find more clear. Given that the C# 2 specification defines the null value, it would be odd if we couldn’t use the null literal that’s already in the language to represent it. Fortunately we can… 4.3.2 Assigning and comparing with null A concise author could cover this whole section in a single sentence: “The C# com- piler allows the use of null to represent the null value of a nullable type in both com- parisons and assignments.” I prefer to show you what it means in real code and to consider why the language has been given this feature. You may have felt uncomfortable every time you used the default constructor of Nullable. It achieves the desired behavior, but it doesn’t express the reason why you want to do it—it doesn’t leave the right impression with the reader. It should ide- ally give the same sort of feeling that using null does with reference types. If it seems odd to you that I’ve talked about feelings in both this section and the previous one, just think about who writes code and who reads it. Sure, the compiler has to understand the code, and it couldn’t care less about the subtle nuances of style, but few pieces of code used in production systems are written and then never read again. Anything you can do to get the reader into the mental process you were going through when you originally wrote the code is good, and using the familiar null lit- eral helps to achieve that. With that in mind, we’ll switch from using an example that just shows syntax and behavior to one that gives an impression of how nullable types might be used. We’ll model a Person class where you need to know a person’s name, date of birth, and date of death. We’ll only keep track of people who have definitely been born, but some of those people may still be alive—in which case the date of death will be represented by 117C# 2’s syntactic sugar for nullable types null. The following listing shows some of the possible code. A real class would have more operations available—we’ll just look at the calculation of age for this example. class Person { DateTime birth; DateTime? death; string name; public TimeSpan Age { get { if (death == null) { return DateTime.Now - birth; } else { return death.Value - birth; } } } public Person(string name, DateTime birth, DateTime? death) { this.birth = birth; this.death = death; this.name = name; } } ... Person turing = new Person("Alan Turing ", new DateTime(1912, 6, 23), new DateTime(1954, 6, 7)); Person knuth = new Person("Donald Knuth ", new DateTime(1938, 1, 10), null); Listing 4.4 doesn’t produce any output, but the fact that it compiles might have sur- prised you before reading this chapter. Apart from the use of the ? modifier causing confusion, you might have found it odd that you could compare a DateTime? with null or pass null as the argument for a DateTime? parameter. Hopefully by now the meaning is intuitive—when you compare the death variable with null, you’re asking whether its value is the null value or not. Likewise when you use null as a DateTime? instance, you’re really creating the null value for the type by calling the default constructor. Indeed, you can see in the generated IL that the code the com- piler spits out for listing 4.4 really does just call the death.HasValue property B and create a new instance of DateTime? E using the default constructor (represented in IL Listing 4.4 Part of a Person class including calculation of age Checks HasValueB Unwraps for calculationC Wraps DateTime as nullable D Specifies null date of death E 118 CHAPTER 4 Saying nothing with nullable types as the initobj instruction). The date of Alan Turing’s death D is created by calling the normal DateTime constructor and then passing the result into the Nullable constructor that takes a parameter. I mention looking at the IL because that can be a useful way of finding out what your code is actually doing, particularly if something compiles when you don’t expect it to. You can use the ildasm tool that comes with the .NET SDK, or one of the many decompilers now available, such as .NET Reflector, ILSpy, dotPeek, or JustDecompile. (Whenever I refer to Reflector in this book, it’s solely because that’s the tool I use out of habit. The others are perfectly fine too, I’m sure.) You’ve seen how C# provides shorthand syntax for the concept of a null value, making the code more expressive once nullable types are understood in the first place. But one part of listing 4.4 took a bit more work than you might have hoped— the subtraction at C. Why did you have to unwrap the value? Why couldn’t you just return death - birth directly? What would you want that expression to mean if death had been null (excluded in this code by the earlier test)? These questions—and more—are answered in the next section. 4.3.3 Nullable conversions and operators You’ve seen that you can compare instances of nullable types with null, but there are other comparisons that can be made and other operators that can be used in some cases. Likewise you’ve seen wrapping and unwrapping, but other conversions can be used with some types. This section explains what’s available. I’m afraid it’s pretty much impossible to make this kind of topic genuinely exciting, but carefully designed fea- tures like these are what make C# a pleasant language to work with in the long run. Don’t worry if not all of it sinks in the first time through: just remember that the details are here if you need to refer to them in the middle of a coding session. The executive summary is that if there’s an operator or conversion available on a non-nullable value type, and that operator or conversion only involves other non- nullable value types, then the nullable value type also has the same operator or con- version available, usually converting the non-nullable value types into their nullable equivalents. To give a more concrete example, there’s an implicit conversion from int to long, and that means there’s also an implicit conversion from int? to long? that behaves in the obvious manner. Unfortunately, although that broad description gives the right general idea, the exact rules are slightly more complicated. Each rule is simple, but there are quite a few of them. It’s worth knowing about them because otherwise you might end up star- ing at a compiler error or warning for a while, wondering why it believes you’re trying to make a conversion that you never intended in the first place. We’ll start with the conversions and then look at the operators. 119C# 2’s syntactic sugar for nullable types CONVERSIONS INVOLVING NULLABLE TYPES For completeness, let’s start with the conversions you already know about:  An implicit conversion from the null literal to T?  An implicit conversion from T to T?  An explicit conversion from T? to T Now consider the predefined and user-defined conversions available on types. For instance, there’s a predefined conversion from int to long. For any conversion like this, from one non-nullable value type (S) to another (T), the following conversions are also available:  S? to T? (explicit or implicit depending on original conversion)  S to T? (explicit or implicit depending on original conversion)  S? to T (always explicit) To carry the example forward, this means that you can convert implicitly from int? to long? and from int to long? as well as explicitly from int? to long. The conversions behave in the natural way, with null values of S? converting to null values of T?, and non-null values using the original conversion. As before, the explicit conversion from S? to T will throw an InvalidOperationException when converting from a null value of S?. For user-defined conversions, these extra conversions involving nullable types are known as lifted conversions. So far, so relatively simple. Now let’s consider the operators, where things are slightly more tricky. OPERATORS INVOLVING NULLABLE TYPES C# allows the following operators to be overloaded:3  Unary: + ++ - -- ! ~ truefalse  Binary: + - * / % & | ^ << >>  Equality: == !=  Relational: < > <= >= When these operators are overloaded for a non-nullable value type T, the nullable type T? has the same operators, with slightly different operand and result types. These are called lifted operators, whether they’re predefined operators such as addition on numeric types or user-defined operators such as adding a TimeSpan to a DateTime. There are a few restrictions as to when they apply:  The true and false operators are never lifted. They’re incredibly rare in the first place, though, so it’s no great loss.  Only operators with non-nullable value types for the operands are lifted. 3 The equality and relational operators are also binary operators, but they behave slightly differently than the others; hence their separation in this list. 120 CHAPTER 4 Saying nothing with nullable types  For the unary and binary operators (other than equality and relational opera- tors), the return type has to be a non-nullable value type.  For the equality and relational operators, the return type has to be bool.  The & and | operators on bool? have separately defined behavior, which you’ll see in section 4.3.4. For all the operators, the operand types become their nullable equivalents. For the unary and binary operators, the return type also becomes nullable, and a null value is returned if any of the operands is a null value. The equality and relational operators keep their non-nullable Boolean return types. For equality, two null values are consid- ered equal, and a null value and any non-null value are considered different, which is consistent with the behavior you saw in section 4.2.3. The relational operators always return false if either operand is a null value. When none of the operands is a null value, the operator of the non-nullable type is invoked in the obvious way. All these rules sound more complicated than they really are—for the most part, everything works as you probably expect it to. It’s easiest to see what happens with a few examples, and as int has so many predefined operators (and integers can be so easily expressed), it’s the natural demonstration type. Table 4.1 shows a number of expressions, the lifted operator signature, and the result. It’s assumed that there are variables four, five, and nullInt, each with type int? and with the obvious values. Possibly the most surprising line of the table is the last one—that a null value isn’t deemed less than or equal to another null value, even though they are deemed to be equal to each other (as per the fifth row)! Very odd, but unlikely to cause problems in real life, in my experience. One aspect of lifted operators and nullable conversion that has caused some con- fusion is unintended comparisons with null when using a non-nullable value type. The code that follows is legal, but not useful: Table 4.1 Examples of lifted operators applied to nullable integers Expression Lifted operator Result -nullInt -five five + nullInt five + five nullInt == nullInt five == five five == nullInt five == four four < five nullInt < five five < nullInt nullInt < nullInt nullInt <= nullInt int? –(int? x) int? –(int? x) int? +(int? x, int? y) int? +(int? x, int? y) bool ==(int? x, int? y) bool ==(int? x, int? y) bool ==(int? x, int? y) bool ==(int? x, int? y) bool <(int? x, int? y) bool <(int? x, int? y) bool <(int? x, int? y) bool <(int? x, int? y) bool <=(int? x, int? y) null -5 null 10 true true false false true false false false false 121C# 2’s syntactic sugar for nullable types int i = 5; if (i == null) { Console.WriteLine ("Never going to happen"); } The C# compiler raises warnings on this code, but you may consider it surprising that it’s allowed at all. What’s happening is that the compiler sees the int expression on the left side of the ==, sees null on the right side, and knows that there’s an implicit conversion to int? from each of them. Because a comparison between two int? val- ues is perfectly valid, the code doesn’t generate an error—just the warning. As a fur- ther complication, this isn’t allowed in the case where, instead of int, you’re dealing with a generic type parameter that has been constrained to be a value type—the rules on generics prohibit the comparison with null in that situation. Either way, there’ll be an error or a warning, so as long as you look closely at warn- ings, you shouldn’t end up with deficient code due to this quirk, and hopefully my pointing it out to you now will save you from getting a headache trying to work out exactly what’s going on. Now you can answer the question at the end of the previous section—why we used death.Value - birth in listing 4.4 instead of just death - birth. Applying the previ- ous rules, you could have used the latter expression, but the result would’ve been a TimeSpan? instead of a TimeSpan. This would’ve left you with the options of casting the result to TimeSpan using its Value property, or changing the Age property to return a TimeSpan?, which just pushes the issue onto the caller. It’s still a bit ugly, but you’ll see a nicer implementation of the Age property in section 4.3.6. In the list of restrictions regarding operator lifting, I mentioned that bool? works slightly differently than the other types. The next section explains this and pulls the lens back to see the bigger picture of why all these operators work the way they do. 4.3.4 Nullable logic I vividly remember my early electronics lessons at school. They always seemed to revolve around either working out the voltage across different parts of a circuit using the V=IxR formula, or applying truth tables—the reference charts for explaining the difference between NAND gates and NOR gates and so on. The idea is simple—a truth table maps out every possible combination of inputs into whatever piece of logic you’re interested in and tells you the output. The truth tables we drew for simple, two-input logic gates always had four rows— each input had two possible values, which means there were four possible combina- tions. Boolean logic is simple like that, but what happens when you have a tristate log- ical type? Well, bool? is just such a type—the value can be true, false, or null. That means that your truth tables now need nine rows for binary operators as there are nine combinations. The specification only highlights the logical AND and inclusive OR operators (& and |, respectively) because the other operators—unary logical negation (!) and exclusive OR (^)—follow the same rules as other lifted operators. There are 122 CHAPTER 4 Saying nothing with nullable types no conditional logical operators (the short-circuiting && and || operators) defined for bool?, which makes life simpler. For the sake of completeness, table 4.2 gives the truth table for all four valid bool? logical operators. If you find reasoning about rules easier to understand than looking up values in tables, the idea is that a null bool? value is in some senses a “maybe.” If you imagine that each null entry in the input side of the table is a variable instead, you’ll always get a null value on the output side of the table if the result depends on the value of that variable. For instance, looking at the third line of the table, the expression true & y will only be true if y is true, but the expression true | y will always be true whatever the value of y is, so the nullable results are null and true, respectively. When considering the lifted operators and particularly how nullable logic works, the language designers had two slightly contradictory sets of existing behavior—C# 1 null references and SQL NULL values. In many cases, these don’t conflict at all—C# 1 had no concept of applying logical operators to null references, so there was no prob- lem in using the SQL-like results given earlier. The definitions you’ve seen may sur- prise some SQL developers, though, when it comes to comparisons. In standard SQL, the result of comparing two values (in terms of equality or greater than/less than) is always unknown if either value is NULL. The result in C# 2 is never null, and in particu- lar two null values are considered to be equal to each other. REMINDER: THIS IS C# SPECIFIC! It’s worth remembering that the lifted opera- tors and conversions, along with the bool? logic described in this section, are all provided by the C# compiler and not by the CLR or the framework itself. If you use ildasm on code that evaluates any of these nullable operators, you’ll find that the compiler has created all the appropriate IL to test for null values and dealt with them accordingly. This means that different languages can behave differently on these matters—definitely something to look out for if you need to port code between different .NET-based languages. For example, VB treats lifted operators far more like SQL, so the result of x < y is Nothing if x or y is Nothing. Table 4.2 Truth table for the logical operators AND, inclusive OR, exclusive OR, and logical negation, applied to the bool? type x y x & y x | y x ^ y !x true true true false false false null null null true false null true false null true false null true false null false false false null false null true true true true false null true null null false true null true false null null null null false false false true true true null null null 123C# 2’s syntactic sugar for nullable types Another familiar operator is now available with nullable value types, and it behaves exactly as you’d expect it to if you consider your existing knowledge of null references and just tweak it to be in terms of null values. 4.3.5 Using the as operator with nullable types Prior to C# 2, the as operator was only available for reference types. As of C# 2, it can now be applied to nullable value types as well. The result is a value of that nullable type—either the null value if the original reference was the wrong type or null, or a meaningful value otherwise. Here’s a short example: static void PrintValueAsInt32(object o) { int? nullable = o as int?; Console.WriteLine(nullable.HasValue ? nullable.Value.ToString() :"null"); } ... PrintValueAsInt32(5); PrintValueAsInt32("some string"); This allows you to safely convert from an arbitrary reference to a value in a single step—although you’d normally check. In C# 1, you’d have had to use the is operator followed by a cast, which is inelegant: it’s essentially asking the CLR to perform the same type check twice. SURPRISING PERFORMANCE TRAP I’d always assumed that doing one check would be faster than two, but it appears that’s not the case—at least with the versions of .NET I’ve tested with (up to and including .NET 4.5). When writing a quick benchmark that summed all the integers within an array of type object[], where only a third of the values were actually boxed integers, using is and then a cast ended up being 20 times faster than using the as operator. The details are beyond the scope of this book, and as always you should test performance with your actual code and data before deciding on the best course of action for your specific situation, but it’s worth being aware of. You now know enough to use nullable types and predict how they’ll behave, but C# 2 has a sort of “bonus track” when it comes to syntax enhancements: the null coalescing operator. 4.3.6 The null coalescing operator Aside from the ? modifier, all of the rest of the C# compiler’s tricks relating to nul- lable types so far have worked with the existing syntax. But C# 2 introduces a new operator that can occasionally make code shorter and sweeter. It’s called the null coalescing operator and appears in code as ?? between its two operands. It’s like the con- ditional operator but specially tweaked for nulls. It’s a binary operator that evaluates first ?? second by going through the follow- ing steps (roughly speaking): Yields 5 Yields null 124 CHAPTER 4 Saying nothing with nullable types 1 Evaluate first. 2 If the result is non-null, that’s the result of the whole expression. 3 Otherwise, evaluate second; the result then becomes the result of the whole expression. I say “roughly speaking” because the formal rules in the specification have to deal with situations involving conversions between the types of first and second. As ever, these aren’t important in most uses of the operator, and I don’t intend to go through them—consult section 7.13 of the specification (“The Null Coalescing Operator”) if you need the details. Importantly, if the type of the second operand is the underlying type of the first operand (and therefore non-nullable), the overall result is that underlying type. For example, this code is perfectly valid: int? a = 5; int b = 10; int c = a ?? b; Note that you’re assigning directly to c even though its type is the non-nullable int type. You can only do this because b is non-nullable, so you know that you’ll get a non- nullable result eventually. Obviously that’s a pretty simplistic example; let’s find a more practical use for this operator by revisiting the Age property from listing 4.4. As a reminder, here’s how it was implemented back then, along with the relevant variable declarations: DateTime birth; DateTime? death; public TimeSpan Age { get { if (death == null) { return DateTime.Now - birth; } else { return death.Value - birth; } } } Note how both branches of the if statement subtract the value of birth from some non-null DateTime value. The value you’re interested in is the latest time the person was alive—the time of the person’s death if they have already died, or now otherwise. To make progress in little steps, let’s try using the normal conditional operator first: DateTime lastAlive = (death == null ? DateTime.Now : death.Value); return lastAlive - birth; 125C# 2’s syntactic sugar for nullable types That’s progress of a sort, but arguably the conditional operator has made it harder to read rather than easier, even though the new code is shorter. The conditional opera- tor is often like that—how much you use it is a matter of personal preference, although it’s worth consulting the rest of your team before using it extensively. Let’s see how the null coalescing operator improves things. You want to use the value of death if it’s non-null, and DateTime.Now otherwise. You can change the implementa- tion to the following: DateTime lastAlive = death ?? DateTime.Now; return lastAlive - birth; Note how the type of the result is DateTime rather than DateTime? because you’ve used DateTime.Now as the second operand. You could shorten the whole thing to one expression: return (death ?? DateTime.Now) - birth; But this is more obscure—in particular, in the two-line version the name of the last- Alive variable helps the reader to see why you’re applying the null coalescing opera- tor. I hope you agree that the two-line version is simpler and more readable than either the original version using the if statement or the version using the normal con- ditional operator from C# 1. Of course, it relies on the reader understanding what the null coalescing operator does. In my experience, this is one of the least-known aspects of C# 2, but it’s useful enough to make it worth trying to enlighten your co-workers rather than avoiding it. There are two further aspects that increase the operator’s usefulness. First, it doesn’t just apply to nullable value types—it works with reference types too; you just can’t use a non-nullable value type for the first operand, as that would be pointless. Also, it’s right associative, which means an expression of the form first ?? second ?? third is evaluated as first ?? (second ?? third)—and so it continues for more oper- ands. You can have any number of expressions, and they’ll be evaluated in order, stop- ping with the first non-null result. If all of the expressions evaluate to null, the result will be null too. As a concrete example of this, suppose you have an online ordering system with the concepts of a billing address, contact address, and shipping address. The business rules declare that any user must have a billing address, but the contact address is optional. The shipping address for a particular order is also optional, defaulting to the billing address. These optional addresses are easily represented as null references in the code. To determine who should be contacted in the case of a problem with a shipment, the code in C# 1 might look something like this: Address contact = user.ContactAddress; if (contact == null) { contact = order.ShippingAddress; if (contact == null) { 126 CHAPTER 4 Saying nothing with nullable types contact = user.BillingAddress; } } Using the conditional operator in this case is even more horrible. But using the null coalescing operator makes the code very straightforward: Address contact = user.ContactAddress ?? order.ShippingAddress ?? user.BillingAddress; If the business rules changed to use the shipping address by default instead of the user’s contact address, the change here would be extremely obvious. It wouldn’t be particularly taxing with the if/else version, but I know I’d have to stop and think twice, and verify the code mentally. I’d also be relying on unit tests, so there’d be little chance of actually getting it wrong, but I’d prefer not to think about things like this unless I absolutely have to. EVERYTHING IN MODERATION Just in case you’re thinking that my code is lit- tered with uses of the null coalescing operator, it’s really not. I tend to con- sider it when I see defaulting mechanisms involving nulls and possibly the conditional operator, but it doesn’t come up often. When its use is natural, though, it can be a powerful tool in the battle for readability. You’ve seen how nullable types can be used for ordinary properties of objects—cases where you naturally might not have a value for some particular aspect that’s still best expressed with a value type. Those are the more obvious uses for nullable types and indeed the most common ones. A few other patterns aren’t as obvious, but can still be powerful when you’re used to them. We’ll explore two of these patterns in the next section. This is more for the sake of interest than as part of learning about the behav- ior of nullable types themselves—you now have all the tools you need to use them in your own code. If you’re interested in quirky ideas and perhaps trying something new, read on… 4.4 Novel uses of nullable types Before nullable types became a reality, I saw lots of people effectively asking for them, usually in relation to database access. That’s not the only use they can be put to, though. The patterns presented in this section are unconventional but can make code simpler. If you always stick to normal idioms of C#, that’s fine—this section might not be for you, and I have a lot of sympathy for that point of view. I usually prefer simple code over code that’s clever, but if a whole pattern provides benefits, that sometimes makes the pattern worth learning. Whether you use these techniques is entirely up to you—you may find that they suggest other ideas to use elsewhere in your code. Without further ado, let’s start with an alternative to the TryXXX pattern men- tioned in section 3.3.3. 127Novel uses of nullable types 4.4.1 Trying an operation without using output parameters The pattern of using a return value to say whether an operation worked and using an output parameter to return the real result is becoming increasingly common in the .NET Framework. I have no issues with the aims—the idea that some methods are likely to fail to perform their primary purpose in non-exceptional circumstances is common sense. My one problem with it is that I’m not a huge fan of output parameters. There’s something slightly clumsy about the syntax of declaring a variable on one line, and then immediately using it as an output parameter. Methods returning reference types have often used a pattern of returning null on failure and non-null on success, but that doesn’t work so well when null is a valid return value in the success case. Hashtable is an example of both of these statements, in a slightly ambivalent way: null is a theoretically valid value in a Hashtable, but in my experience most uses of Hashtable never use null values, which makes it perfectly acceptable to have code that assumes that a null value means a missing key. One common scenario is to have each value of the Hashtable as a list: the first time an item is added for a particular key, a new list is created and the item is added to it. Thereafter, adding another item for the same key involves adding the item to the existing list. Here’s the code in C# 1: ArrayList list = hash[key]; if (list == null) { list = new ArrayList(); hash[key] = list; } list.Add(newItem); Hopefully you’d use variable names more specific to your situation, but I’m sure you get the idea and you may well have used the pattern yourself.4 With nullable types, this pattern can be extended to value types, and it’s safer with value types, because if the natural result type is a value type, then a null value could only be returned as a result of failure. Nullable types add that extra Boolean piece of information in a nice gen- eral way with language support, so why not use them? To demonstrate this pattern in practice and in a context other than dictionary lookups, I’ll use the classic example of the TryXXX pattern—parsing an integer. The implementation of the TryParse method in the following listing shows the version of the pattern using an output parameter, but then you see the version using nullable types in the main part at the bottom. 4 Wouldn’t it be great if Hashtable and Dictionary could take a delegate to call whenever a new value was required due to looking up a missing key? Situations like this would be a lot simpler. 128 CHAPTER 4 Saying nothing with nullable types static int? TryParse(string text) { int ret; if (int.TryParse(text, out ret)) { return ret; } else { return null; } } ... int? parsed = TryParse("Not valid"); if (parsed != null) { Console.WriteLine ("Parsed to {0}", parsed.Value); } else { Console.WriteLine ("Couldn't parse"); } You may think there’s little to distinguish the two versions here—they’re the same number of lines, after all. But I believe there’s a difference in emphasis. The nullable version encapsulates the natural return value and the success or failure into a single variable. It also separates the doing from the testing, which puts the emphasis in the right place, in my opinion. Usually, if I call a method in the condition part of an if statement, that method’s primary purpose is to return a Boolean value. Here, the return value is in some ways less important than the output parameter. When you’re reading code, it’s easy to miss an output parameter in a method call and be left won- dering what’s actually doing all the work and magically giving the answer. With the nullable version, this is more explicit—the result of the method has all the informa- tion you’re interested in. I’ve used this technique in a number of places (often with more method parameters, at which point output parameters become even harder to spot), and I believe it has improved the general feel of the code. Of course, this only works for value types. Another advantage of this pattern is that it can be used in conjunction with the null coalescing operator—you can try to understand several pieces of input, stopping at the first valid one. The normal TryXXX pattern allows this using the short-circuiting operators, but the meaning isn’t nearly as clear when you use the same variable for two different output parameters in the same statement. ALTERNATIVELY, USE A TUPLE… Another alternative to using a nullable result is to use a return type with two very clearly separate members, one of which is responsible for indicating success or failure and another of which is responsi- ble for indicating the value on success. Nullable is convenient because it Listing 4.5 An alternative implementation of the TryXXX pattern Classic call with output parameter Nullable call 129Novel uses of nullable types gives you a Boolean property and another of type T, but the meaning of the return value could perhaps be more explicit. .NET 4 includes the Tuple family of types: arguably a Tuple might be cleaner than int? here. Even cleaner would be a custom type to represent the result of a parse operation: ParseResult, for example. In this case, you could hand the value around to other code without any fear that its meaning will be confused, and you can add extra information such as the cause of any parsing failure. The next pattern is an answer to a specific pain point—the irritation and fluff that can be present when writing multitiered comparisons. 4.4.2 Painless comparisons with the null coalescing operator I suspect you dislike writing the same code over and over again as much as I do. Refac- toring can often get rid of duplication, but some cases resist refactoring surprisingly effectively. Code for Equals and Compare often falls firmly into this category. Suppose you’re writing an e-commerce site and have a list of products. You might want to sort them by popularity (descending), then price, then name—so that the five-star-rated products come first, but the cheapest five-star products come before the more expensive ones. If there are multiple products with the same price, products beginning with A are listed before products beginning with B. This isn’t a problem specific to e-commerce sites—sorting data by multiple criteria is a fairly common requirement in computing. Assuming you have a suitable Product type, you could write the comparison with code like this in C# 1: public int Compare(Product first, Product second) { // Reverse comparison of popularity to sort descending int ret = second.Popularity.CompareTo(first.Popularity); if (ret != 0) { return ret; } ret = first.Price.CompareTo(second.Price); if (ret != 0) { return ret; } return first.Name.CompareTo(second.Name); } This assumes that you won’t be asked to compare null references and that all of the properties will return non-null references too. You could use some up-front null com- parisons and Comparer.Default to handle those cases, but that would make the code even longer and more involved. The code could be shorter (and avoid returning from the middle of the method) if you rearranged it slightly, but the fundamental “compare, check, compare, check” pattern would still be present, and it wouldn’t be as obvious that once you have a nonzero answer you’re done. 130 CHAPTER 4 Saying nothing with nullable types Ah…that last sentence is reminiscent of something else: the null coalescing opera- tor. As you saw in section 4.3, if you have a lot of expressions separated by ??, then the operator will be repeatedly applied until it hits a non-null expression. Now all you have to do is work out a way of returning null instead of zero from a comparison. This is easy to do in a separate method that can also encapsulate the use of the default com- parer. You can even have an overload to use a specific comparer if you want. You can also deal with the case where either of the Product references you’re passed is null. First, let’s look at the class implementing the helper methods, as shown in the fol- lowing listing. public static class PartialComparer { public static int? Compare(T first, T second) { return Compare(Comparer.Default, first, second); } public static int? Compare(IComparer comparer, T first, T second) { int ret = comparer.Compare(first, second); return ret == 0 ? new int?() : ret; } public static int? ReferenceCompare(T first, T second) where T: class { return first == second ? 0 : first == null ?-1 : second == null ? 1 : new int?(); } } The Compare methods in listing 4.6 are almost pathetically simple—when a comparer isn’t specified, the default comparer for the type is used, and all that happens to the comparison’s return value is that zero is translated to the null value. NULL VALUES AND THE CONDITIONAL OPERATOR You may have been surprised to see me use new int?() rather than null to return the null value in the sec- ond Compare method. The conditional operator requires that its second and third operands either be of the same type, or that there be an implicit conver- sion from one to the other, and that wouldn’t be the case with null, because the compiler wouldn’t know what type the value was meant to be. The lan- guage rules don’t take the overall aim of the statement (returning from a method with a return type of int?) into account when examining subexpres- sions. Other options include casting either operand to int? explicitly or using default(int?) for the null value. Basically, the important thing is to make sure that one of the operands is known to be an int? value. Listing 4.6 Helper class for providing partial comparisons 131Summary The ReferenceCompare method uses another conditional operator—three of them, in fact. You may find this less readable than the (rather longer) equivalent code using if/else blocks—it depends on how comfortable you are with the conditional opera- tor. I like it because it makes the order of the comparisons clear. Also, this could easily have been a nongeneric method with two object parameters, but this form prevents you from accidentally using the method to compare value types via boxing. The method really is only useful with reference types, which is indicated by the type param- eter constraint. Even though this class is simple, it’s remarkably useful. You can now replace the previous product comparison with a neater implementation: public int Compare(Product first, Product second) { return PC.ReferenceCompare(first, second) ?? // Reverse comparison of popularity to sort descending PC.Compare(second.Popularity, first.Popularity) ?? PC.Compare(first.Price, second.Price) ?? PC.Compare(first.Name, second.Name) ?? 0; } As you may have noticed, I’ve used PC rather than PartialComparer—this is solely for the sake of being able to fit the lines on the printed page. In real code, I’d use the full type name and still have one comparison per line. Of course, if you wanted short lines for some reason, you could specify a using directive to make PC an alias for Partial- Comparer—I just wouldn’t recommend it. The final 0 indicates that if all of the earlier comparisons have passed, the two Product instances are equal. You could just use Comparer.Default.Compare (first.Name, second.Name) as the final comparison, but that would hurt the symme- try of the method. This comparison plays nicely with nulls, it’s easy to modify, it forms an easy pattern to use for other comparisons, and it only compares as far as it needs to—if the prices are different, the names won’t be compared. You may be wondering whether the same technique could be applied to equality tests, which often have similar patterns. There’s much less point in the case of equal- ity, because after the nullity and reference equality tests, you can just use && to provide the desired short-circuiting functionality for Booleans. A method returning a bool? can be used to obtain an initial definitely equal, definitely not equal, or unknown result based on the references, though. The complete code of PartialComparer on this book’s website contains the appropriate utility method and examples of its use. 4.5 Summary When faced with a problem, developers tend to take the easiest short-term solution, even if it’s not particularly elegant. That’s often the right decision—you don’t want to be guilty of overengineering, after all. But it’s always nice when a good solution is also the easiest solution. 132 CHAPTER 4 Saying nothing with nullable types Nullable types solve a specific problem that only had somewhat ugly solutions before C# 2. The features provided are a better-supported version of a solution that was feasible but time consuming in C# 1. The combination of generics (to avoid code duplication), CLR support (to provide suitable boxing and unboxing behavior), and language support (to provide concise syntax along with convenient conversions and operators) makes the solution far more compelling than it was previously. It just so happens that in providing nullable types, the C# and framework designers have made some other patterns available that weren’t worth the effort before. We’ve looked at some of them in this chapter, and I wouldn’t be surprised to see more of them appearing over time. So far generics and nullable types have addressed areas where in C# 1 you occa- sionally had to hold your nose due to unpleasant code smells. This pattern continues in the next chapter, where we’ll discuss the enhancements to delegates. These form an important part of the subtle change of direction of both the C# language and the .NET Framework toward a slightly more functional viewpoint. This emphasis is made even clearer in C# 3, so although we’re not looking at those features quite yet, the del- egate enhancements in C# 2 act as a bridge between the familiarity of C# 1 and the style of idiomatic C# 3, which can often be radically different from earlier versions. 133 Fast-tracked delegates The journey of delegates in C# and .NET has been an interesting one, showing remarkable foresight (or really good luck) on the part of the designers. The con- ventions suggested for event handlers in .NET 1.0/1.1 didn’t make a lot of sense— until C# 2 showed up. Likewise, the effort put into delegates for C# 2 seems in some ways out of proportion to how widely used they are—until you see how pervasive they are in idiomatic C# 3 code. In other words, it’s as if the language and platform designers had a vision of at least the rough direction they’d be taking, years before the destination itself became clear. Of course, C# 3 isn’t a final destination in itself—generic delegates get a bit more flexibility in C# 4, C# 5 makes it easy to write asynchronous delegates, and we may see even more advances in the future—but the differences between C# 1 and This chapter covers  Long-winded C# 1 syntax  Simplified delegate construction  Covariance and contravariance  Anonymous methods  Captured variables 134 CHAPTER 5 Fast-tracked delegates C# 3 in this area are the most startling ones. (The primary change in C# 3 supporting delegates is in lambda expressions, which you’ll meet in chapter 9.) C# 2 is a sort of stepping stone in terms of delegates. Its new features pave the way for the dramatic changes of C# 3, keeping developers reasonably comfortable while still providing useful benefits. I’m reliably informed that language designers were aware that the combined feature set of C# 2 would open up whole new ways of looking at code, but they didn’t necessarily know where those paths would lead. So far, their instincts have proved remarkably beneficial in the area of delegates. Delegates play a more prominent part in .NET 2.0 than in earlier versions, although they’re not as common as they are in .NET 3.5. In chapter 3 you saw how they can be used to convert from one type of list to another, and way back in chapter 1 you sorted a list of products using the Comparison delegate instead of the IComparer interface. Although the framework and C# keep a respectful distance from each other where possible, I believe that the language and platform drove each other in this case: the inclusion of more delegate-based API calls supports the improved syntax available in C# 2, and vice versa. In this chapter, we’ll look at how C# 2 includes two small changes that make life easier when creating delegate instances from normal methods, and then we’ll look at the biggest change: anonymous methods, which allow you to specify a delegate instance’s action inline at the point of its creation. The largest section of the chapter is devoted to the most complicated part of anonymous methods—captured variables— which provide delegate instances with a richer environment to play in. We’ll cover the topic in significant detail due to its importance and complexity. Once you’ve come to grips with anonymous methods, lambda expressions are easy to understand. First, though, let’s review the pain points of C# 1’s delegate facilities. 5.1 Saying goodbye to awkward delegate syntax The syntax for delegates in C# 1 doesn’t sound too bad—the language already has syn- tactic sugar around Delegate.Combine, Delegate.Remove, and the invocation of dele- gate instances. It makes sense to specify the delegate type when creating a delegate instance; after all, it’s the same syntax used to create instances of other types. This is all true, but for some reason it also sucks. It’s hard to say exactly why the del- egate creation expressions of C# 1 raise hackles, but they do—at least for me. When hooking up a bunch of event handlers, it just looks ugly to have to write new Event- Handler (or whatever is required) all over the place, when the event itself has speci- fied which delegate type it’ll use. Beauty is in the eye of the beholder, of course, and you could argue that there’s less call for guesswork when reading event handler wiring code in the C# 1 style, but the extra text just gets in the way and distracts from the important part of the code: the method you want to handle the event. Life becomes more black and white when you consider covariance and contravari- ance as applied to delegates. Suppose you have an event handling method that saves the current document, or logs that it’s been called, or performs any number of other 135Saying goodbye to awkward delegate syntax actions that may not need to know details of the event. The event itself shouldn’t mind that your method is capable of working with only the information provided by the EventHandler signature, even though the event is declared to pass in mouse event details. Unfortunately, in C# 1 you need to have a different method for each different event handler signature. Likewise it’s undeniably ugly to write methods that are so simple that their imple- mentation is shorter than their signature, solely because delegates need to have an action to execute in the form of a method. It adds an extra layer of indirection between the code creating the delegate instance and the code that should execute when it’s invoked. Extra layers of indirection are often welcome—that option hasn’t been removed in C# 2—but at the same time it frequently makes the code harder to read and pollutes the class with a bunch of methods that are only used for delegates. Unsurprisingly, all of these issues are improved greatly in C# 2. The syntax can still be wordier than you might like (until you get lambda expressions in C# 3), but the dif- ference is significant. To illustrate the pain, we’ll start with some code in C# 1 and improve it in the next couple of sections. The following listing builds a (very) simple form with a button and then subscribes to three of the button’s events. static void LogPlainEvent(object sender, EventArgs e) { Console.WriteLine("LogPlain"); } static void LogKeyEvent(object sender, KeyPressEventArgs e) { Console.WriteLine("LogKey"); } static void LogMouseEvent(object sender, MouseEventArgs e) { Console.WriteLine("LogMouse"); } ... Button button = new Button(); button.Text = "Click me"; button.Click += new EventHandler(LogPlainEvent); button.KeyPress += new KeyPressEventHandler(LogKeyEvent); button.MouseClick += new MouseEventHandler(LogMouseEvent); Form form = new Form(); form.AutoSize = true; form.Controls.Add(button); Application.Run(form); The output lines in the three event handling methods are there to prove that the code is working: if you press the spacebar with the button highlighted, you’ll see that the Click and KeyPress events are both raised. Pressing Enter just raises the Click event; Listing 5.1 Subscribing to three of a button’s events 136 CHAPTER 5 Fast-tracked delegates clicking on the button raises the Click and MouseClick events. In the following sec- tions, we’ll improve this code using some of the C# 2 features. Let’s start by asking the compiler to make a pretty obvious deduction—which dele- gate type you want to use when subscribing to an event. 5.2 Method group conversions In C# 1, if you want to create a delegate instance, you need to specify both the dele- gate type and the action. Chapter 2 defined the action as the method to call and (for instance methods) the target as the object it’s called on. For example, in listing 5.1, this expression was used to create a KeyPressEvent- Handler: new KeyPressEventHandler(LogKeyEvent) As a standalone expression, it doesn’t look too bad. Even used in a simple event sub- scription it’s tolerable. It becomes uglier when used as part of a longer expression, though. A common example of this is starting a new thread: Thread t = new Thread(new ThreadStart(MyMethod)); What you want to do is start a new thread that’ll execute MyMethod. As ever, you want to express yourself as simply as possible, and C# 2 allows you to do this by means of an implicit conversion from a method group to a compatible delegate type. A method group is simply the name of a method, optionally with a target—exactly the same kind of expression you used in C# 1 to create delegate instances. (Indeed, the expression was called a method group back then—it’s just that the conversion wasn’t available.) If the method is generic, the method group may also specify type arguments—although this is rarely used, in my experience. The new implicit conversion allows you to turn your event subscription into button.KeyPress += LogKeyEvent; Likewise, the thread-creation code becomes simply Thread t = new Thread(MyMethod); The readability differences between the original and the streamlined versions aren’t huge for a single line, but in the context of a significant amount of code, they can reduce the clutter considerably. To make it seem less like magic, let’s briefly look at what this conversion is doing. First, let’s consider the expressions LogKeyEvent and MyMethod as they appear in the examples. The reason they’re classified as method groups is because more than one method may be available, due to overloading. The implicit conversions available will convert a method group to any delegate type with a compatible signature. So, if you had two method signatures like these, void MyMethod() void MyMethod(object sender, EventArgs e) 137Covariance and contravariance you could use MyMethod as the method group in an assignment to either a Thread- Start or an EventHandler, as follows: ThreadStart x = MyMethod; EventHandler y = MyMethod; But you couldn’t use it as the parameter to a method that itself was overloaded to take either a ThreadStart or an EventHandler—the compiler would complain that the call was ambiguous. Likewise, you unfortunately can’t use an implicit method group conversion to convert to the plain System.Delegate type, because the compiler doesn’t know which specific delegate type to create an instance of. This is a pain, but you can still be slightly briefer than in C# 1 by making the conversion explicit. Here’s an example: Delegate invalid = SomeMethod; Delegate valid = (ThreadStart)SomeMethod; For local variables this usually isn’t a problem, but it’s somewhat more annoying when you’re using an API that has a parameter of type Delegate, such as Control.Invoke. There are a few solutions here: using a helper method, casting, or using an intermedi- ate variable. Here’s an example using the MethodInvoker delegate type, which takes no parameters and doesn’t return anything: static void SimpleInvoke(Control control, MethodInvoker invoker) { control.Invoke(invoker); } ... SimpleInvoke(form, UpdateUI); form.Invoke((MethodInvoker)UpdateUI); MethodInvoker invoker = UpdateUI; form.Invoke(invoker); Different situations will encourage different solutions; none of these is particularly appealing, but they’re not awful either.1 As with generics, the precise rules of conversion validity are slightly complicated, and the just-try-it approach works well; if the compiler complains that it doesn’t have enough information, just tell it what conversion to use, and all should be well. If it doesn’t complain, you should be fine. For the exact details, consult the language spec- ification, section 6.6 (“Method group conversions”). Speaking of possible conversions, there may be more than you expect, as you’ll see in the next section. 5.3 Covariance and contravariance We’ve already talked a lot about the concepts of covariance and contravariance in dif- ferent contexts, usually bemoaning their absence, but delegate construction is the one area in which they’re available in C# prior to version 4. If you want to refresh 1 Extension methods (discussed in chapter 10) make the helper method approach somewhat more appealing if you’re using C# 3. Invokes with a helper method Invokes with a cast Invokes with a local variable 138 CHAPTER 5 Fast-tracked delegates yourself about the meaning of the terms at a relatively detailed level, refer back to sec- tion 2.2.2, but the gist of the topic with respect to delegates is that if it would be valid (in a static typing sense) to call a method and use its return value everywhere that you could invoke an instance of a particular delegate type and use its return value, then that method can be used to create an instance of that delegate type. That’s wordy—it’s a lot simpler with examples. DIFFERENT TYPES OF VARIANCE IN DIFFERENT VERSIONS You may already be aware that C# 4 offers generic covariance and contravariance for delegates and interfaces. This is entirely different from the variance we’re looking at here— we’re only dealing with creating new instances of delegates at the moment. The generic variance in C# 4 uses reference conversions, which don’t create new objects—they just view the existing object as a different type. We’ll look at contravariance first, and then covariance. 5.3.1 Contravariance for delegate parameters Let’s consider the event handlers in the little Windows Forms application in listing 5.1. The signatures of the three delegate types are as follows:2 void EventHandler(object sender, EventArgs e) void KeyPressEventHandler(object sender, KeyPressEventArgs e) void MouseEventHandler(object sender, MouseEventArgs e) Consider that KeyPressEventArgs and MouseEventArgs both derive from EventArgs (as do a lot of other types—MSDN lists 403 types that derive directly from EventArgs in .NET 4). If you have a method with an EventArgs parameter, you could always call it with a KeyPressEventArgs argument instead. It therefore makes sense to be able to use a method with the same signature as EventHandler to create an instance of Key- PressEventHandler, and that’s exactly what C# 2 does. This is an example of contra- variance of parameter types. To see that in action, think back to listing 5.1 and suppose that you don’t need to know which event was firing—you just want to write out the fact that an event has hap- pened. Using method group conversions and contravariance, the code becomes a lot simpler, as shown in the following listing. static void LogPlainEvent(object sender, EventArgs e) { Console.WriteLine("An event occurred"); } ... Button button = new Button(); button.Text = "Click me"; button.Click += LogPlainEvent; 2 I’ve removed the public delegate part for reasons of space. Listing 5.2 Demonstration of method group conversions and delegate contravariance Handles all eventsB Uses method group conversion C 139Covariance and contravariance button.KeyPress += LogPlainEvent; button.MouseClick += LogPlainEvent; Form form = new Form(); form.AutoSize = true; form.Controls.Add(button); Application.Run(form); The two handler methods that dealt specifically with key and mouse events have been completely removed, and you’re now using one event handling method for everything B. Of course, this isn’t terribly useful if you want to do different things for different types of events, but sometimes all you need to know is that an event occurred and, potentially, the source of the event. The subscription to the Click event C only uses the implicit conversion we discussed in the previous section because it has a sim- ple EventArgs parameter, but the other event subscriptions D involve the conversion and contravariance due to their different parameter types. I mentioned earlier that the .NET 1.0/1.1 event handler convention didn’t make much sense when it was first introduced. This example shows exactly why the guide- lines are more useful with C# 2. The convention dictates that event handlers should have a signature with two parameters, the first of which is of type object and is the ori- gin of the event, and the second of which carries any extra information about the event in a type deriving from EventArgs. Before contravariance became available, this wasn’t useful—there was no benefit to making the informational parameter derive from EventArgs, and sometimes there wasn’t much use for the origin of the event. It was often more sensible to pass the relevant information directly in the form of nor- mal parameters with appropriate types, just like any other method. Now you can use a method with the EventHandler signature as the action for any delegate type that hon- ors the convention. So far we’ve looked at the values entering a method or delegate—what about the value coming out? 5.3.2 Covariance of delegate return types Demonstrating covariance is harder, as relatively few of the delegates available in .NET 2.0 are declared with a nonvoid return type, and those that are tend to return value types. There are some available, but it’s easier to declare your own delegate type that uses Stream as its return type. For simplicity, we’ll make it parameterless:3 delegate Stream StreamFactory(); You can now use this with a method that’s declared to return a specific type of stream, as shown in the following listing. You declare a method that always returns a Memory- Stream with some sequential data (bytes 0, 1, 2, and so on up to 15), and then use that method as the action for a StreamFactory delegate instance. 3 Return type covariance and parameter type contravariance can be used at the same time, but you’re unlikely to come across situations where that would be useful. Uses conversion and contravariance D 140 CHAPTER 5 Fast-tracked delegates delegate Stream StreamFactory(); static MemoryStream GenerateSampleData() { byte[] buffer = new byte[16]; for (int i = 0; i < buffer.Length; i++) { buffer[i] = (byte) i; } return new MemoryStream(buffer); } ... StreamFactory factory = GenerateSampleData; using (Stream stream = factory()) { int data; while ((data = stream.ReadByte()) != -1) { Console.WriteLine(data); } } The generation and display of the data in listing 5.3 is only present to give the code something to do. The important points are the annotated lines. You declare that the delegate type has a return type of Stream B, but the GenerateSampleData method has a return type of MemoryStream C. The line creating the delegate instance D per- forms the conversion you saw earlier and uses covariance of return types to allow GenerateSampleData to be used as the action for StreamFactory. By the time you invoke the delegate instance E, the compiler no longer knows that a MemoryStream will be returned—if you changed the type of the stream variable to MemoryStream, you’d get a compilation error. Covariance and contravariance can also be used to construct one delegate instance from another. For instance, consider these two lines of code (which assume an appro- priate HandleEvent method): EventHandler general = new EventHandler(HandleEvent); KeyPressEventHandler key = new KeyPressEventHandler(general); The first line is valid in C# 1, but the second isn’t—in order to construct one delegate from another in C# 1, the signatures of the two delegate types involved have to match. For instance, you could create a MethodInvoker from a ThreadStart, but you couldn’t create a KeyPressEventHandler from an EventHandler as shown in the second line. You’re using contravariance to create a new delegate instance from an existing one with a compatible delegate type signature, where compatibility is defined in a less restrictive manner in C# 2 than in C# 1. All of this is positive, except for one small fly in the ointment. Listing 5.3 Demonstration of covariance of return types for delegates Declares delegate type returning StreamB Declares method returning MemoryStreamC Converts method group with covariance D Invokes delegate to obtain streamE 141Covariance and contravariance 5.3.3 A small risk of incompatibility This new flexibility in C# 2 creates one of the few cases where existing valid C# 1 code may produce different results when compiled under C# 2. Suppose a derived class overloads a method declared in its base class, and you try to create an instance of a delegate using a method group conversion. A conversion that previously only matched the base class method could match the derived class method due to covari- ance or contravariance in C# 2, in which case that derived class method would be cho- sen by the compiler. The following listing gives an example of this. delegate void SampleDelegate(string x); public void CandidateAction(string x) { Console.WriteLine("Snippet.CandidateAction"); } public class Derived : Snippet { public void CandidateAction(object o) { Console.WriteLine("Derived.CandidateAction"); } } ... Derived x = new Derived(); SampleDelegate factory = new SampleDelegate(x.CandidateAction); factory("test"); Remember that Snippy4 will be generating all of this code within a class called Snippet, which the nested type derives from. Under C# 1, listing 5.4 would print Snippet.CandidateAction because the method taking an object parameter wasn’t compatible with SampleDelegate. Under C# 2, the method is compatible, and it’s the method chosen due to being declared in a more derived type, so the result is that Derived.CandidateAction is printed. Fortunately, the C# 2 compiler knows that this is a breaking change and issues an appropriate warning. I’ve included this section because you ought to be aware of the possibility of such a problem, but I’m sure it’s rarely encountered in real life. Enough doom and gloom about potential breakage. We’ve still got to see the most important new feature regarding delegates: anonymous methods. They’re a bit more complicated than the topics we’ve covered so far, but they’re also very powerful—and a large step toward C# 3. Listing 5.4 Demonstration of breaking change between C# 1 and C# 2 4 In case you skipped the first chapter, Snippy is a tool I’ve built to create short but complete code samples. See section 1.8.1 for more details. 142 CHAPTER 5 Fast-tracked delegates 5.4 Inline delegate actions with anonymous methods Back in C# 1, it was common to implement a delegate with a particular signature, even though you already had a method with exactly the right behavior but a slightly different set of parameters. Likewise, you’d often want a delegate to do just one teeny, tiny thing—but that meant you needed a whole extra method. The new method would represent behavior that was only relevant within the original method, but it was now exposed to the whole class, creating noise in IntelliSense and generally getting in the way. All this was intensely frustrating. The covariance and contravariance features we’ve just talked about can sometimes help with the first problem, but often they don’t. Anonymous methods, which are also new in C# 2, can pretty much always help with these issues. Informally, anonymous methods allow you to specify the action for a delegate instance inline as part of the delegate instance creation expression. They also provide some far more powerful behavior in the form of closures, but we’ll come to those in section 5.5. For the moment, let’s stick with relatively simple stuff. First we’ll look at examples of anonymous methods that take parameters but don’t return any values; then we’ll explore the syntax involved in providing return values and a shortcut available when you don’t need to use the parameter values passed to you. 5.4.1 Starting simply: acting on a parameter .NET 2.0 introduced a generic delegate type called Action, which we’ll use for our examples. Its signature is simple (aside from the fact that it’s generic): public delegate void Action(T obj) In other words, an Action does something with a value of type T; for example, an Action could reverse the string and print it out, an Action could print out the square root of the number passed to it, and an Action> could find the average of all the numbers given to it and print that out. By complete coinci- dence, these examples are all implemented using anonymous methods in the follow- ing listing. Action printReverse = delegate(string text) { char[] chars = text.ToCharArray(); Array.Reverse(chars); Console.WriteLine(new string(chars)); }; Action printRoot = delegate(int number) { Console.WriteLine(Math.Sqrt(number)); }; Listing 5.5 Anonymous methods used with the Action delegate type Uses anonymous method to create Action B 143Inline delegate actions with anonymous methods Action> printMean = delegate(IList numbers) { double total = 0; foreach (double value in numbers) { total += value; } Console.WriteLine(total / numbers.Count); }; printReverse("Hello world"); printRoot(2); printMean(new double[] { 1.5, 2.5, 3, 4.5 }); Listing 5.5 shows a few of the different features of anonymous methods. First, there’s the syntax of anonymous methods: use the delegate keyword, followed by the param- eters (if there are any), followed by the code for the action of the delegate instance, in a block. The string-reversal code B shows that the block can contain local variable declarations, and the list-averaging code C demonstrates looping within the block. Basically, you can do (almost) anything in an anonymous method that you can do in a normal method body. Likewise, the result of an anonymous method is a delegate instance that can be used like any other one D. But be warned that contravariance doesn’t apply to anonymous methods; you have to specify the parameter types that match the delegate type exactly. A COUPLE OF RESTRICTIONS… One slight oddity is that if you’re writing an anonymous method in a value type, you can’t reference this from within it. There’s no such restriction within a reference type. Additionally, in the Microsoft C# 2 and 3 compiler implementations, accessing a base member within an anonymous method via the base keyword resulted in a warning that the resulting code was unverifiable. This has been fixed in the C# 4 compiler. In terms of implementation, you’re still creating a method in IL for each anonymous method in the source code. The compiler will generate a method within the existing class and use that as the action when it creates the delegate instance, just as if it were a normal method.5 The CLR neither knows nor cares that an anonymous method was used. You can see the extra methods within the compiled code using ildasm or Reflec- tor. (Reflector knows how to interpret the IL to display anonymous methods in the method that uses them, but the extra methods are still visible.) These methods have unspeakable names—ones that are valid in IL, but invalid in C#. This stops you from attempting to refer to them directly in your C# code and avoids the possibility of nam- ing collisions. Many of the features of C# 2 and later versions are implemented in a similar way; one easy way to spot them is that they usually contain angle brackets. For example, an anonymous method in a Main method might cause a method called
b__0 to be created. It’s entirely implementation-specific, though. Microsoft 5 You’ll see in section 5.5.4 that although there’s always a new method, it’s not always created where you might expect. Uses loop in anonymous method C Invokes delegates as normalD 144 CHAPTER 5 Fast-tracked delegates could change its private conventions in a future version, for example. This shouldn’t break anything, as nothing should be relying on these names. It’s worth pointing out at this stage that listing 5.5 is exploded compared with how anonymous methods normally look in real code. You’ll often see them used as argu- ments to another method (rather than assigned to a variable of the delegate type) and with few line breaks—compactness is part of the reason for using them, after all. To demonstrate this, we’ll use the List.ForEach method that takes an Action as a parameter and performs that action on each element. The following listing shows an extreme example, applying the same square-rooting action you used in listing 5.5, but in a compact form. List x = new List(); x.Add(5); x.Add(10); x.Add(15); x.Add(20); x.Add(25); x.ForEach(delegate(int n){Console.WriteLine(Math.Sqrt(n));}); That’s pretty horrendous—especially when the last six characters appear to be ordered almost at random. There’s a happy medium, of course. I tend to break my usual “braces on a line on their own” rule for anonymous methods (as I do for trivial properties), but I still allow a decent amount of whitespace. I might well write the last line of listing 5.6 in one of these two forms: x.ForEach(delegate(int n) { Console.WriteLine(Math.Sqrt(n)); } ); x.ForEach(delegate(int n) { Console.WriteLine(Math.Sqrt(n)); }); Even just adding spaces to listing 5.6 would’ve helped. In each of these formats, the parentheses and braces are now less confusing, and the what-it-does part stands out appropriately. Of course, how you space out your code is entirely your own business, but I encourage you to actively think about where you want to strike the balance, and talk about it with your teammates to try to achieve some consistency. Consistency doesn’t always lead to the most readable code, though—sometimes keeping every- thing on one line is the most straightforward format. So far the only interaction you’ve had with the calling code is through parameters. What about return values? Listing 5.6 Extreme example of code compactness. Warning: unreadable code ahead! 145Inline delegate actions with anonymous methods 5.4.2 Returning values from anonymous methods The Action delegate has a void return type, so you haven’t had to return anything from your anonymous methods yet. To demonstrate how you can do so when you need to, we’ll use the Predicate delegate type from .NET 2.0, which has this signature: public delegate bool Predicate(T obj) The following listing shows an anonymous method creating an instance of Predicate to return whether the argument passed in is odd or even. Predicates are usually used in filtering and matching—you could use the code in this listing to fil- ter a list for just the even elements, for instance. Predicate isEven = delegate(int x) { return x % 2 == 0; }; Console.WriteLine(isEven(1)); Console.WriteLine(isEven(4)); The new syntax is almost certainly what you’d have expected—you return the appro- priate value as if the anonymous method were a normal method. You may have expected to see a return type declared near the delegate keyword, but there’s no need. The compiler checks that all the possible return values are compatible with the declared return type of the delegate type it’s trying to convert the anonymous method into. JUST WHAT ARE YOU RETURNING FROM? When you return a value from an anonymous method, it really is only returning from the anonymous method—it’s not returning from the method creating the delegate instance. It’s easy to look down some code, see the return keyword, and think that it’s an exit point from the current method, so be careful. As I mentioned before, relatively few delegates in .NET 2.0 return values, although as you’ll see in part 3 of this book, .NET 3.5 uses this idea much more often, particularly with LINQ. There’s another reasonably common delegate type in .NET 2.0 though: Comparison, which can be used when sorting collections. It’s the delegate equiva- lent of the IComparer interface. Often you only need a particular sort order in one situation, so it makes sense to be able to specify that order inline, rather than expos- ing it as a method within the rest of the class. The following listing demonstrates this, printing out the files within the C:\ directory, ordering them first by name and then (separately) by size. static void SortAndShowFiles(string title, Comparison sortOrder) { FileInfo[] files = newDirectoryInfo(@"C:\").GetFiles(); Array.Sort(files, sortOrder); Listing 5.7 Returning a value from an anonymous method Listing 5.8 Using anonymous methods to sort files simply 146 CHAPTER 5 Fast-tracked delegates Console.WriteLine(title); foreach (FileInfo file in files) { Console.WriteLine (" {0} ({1} bytes)", file.Name, file.Length); } } ... SortAndShowFiles("Sorted by name:", delegate(FileInfo f1, FileInfo f2) { return f1.Name.CompareTo(f2.Name); } ); SortAndShowFiles("Sorted by length:", delegate(FileInfo f1, FileInfo f2) { return f1.Length.CompareTo(f2.Length); } ); If you weren’t using anonymous methods, you’d need a separate method for each sort order. Instead, listing 5.8 makes it clear what you’ll sort by in each case right where you call SortAndShowFiles. (Sometimes you’ll be calling Sort directly at the point where the anonymous method is called for. In listing 5.8, you’re performing the same fetch/sort/display sequence twice, just with different sort orders, so I encapsulated those steps in their own method.) One special syntactic shortcut is sometimes applicable. If you don’t care about the parameters of a delegate, you don’t have to declare them at all. Let’s see how that works. 5.4.3 Ignoring delegate parameters Occasionally, you want to implement a delegate that doesn’t depend on its parameter values. You might want to write an event handler whose behavior is only appropriate for one event and doesn’t depend on the event arguments—saving the user’s work, for instance. The event handlers from the example in listing 5.1 fit this criterion per- fectly. In this case, you can leave out the parameter list entirely, just using the delegate keyword and then a block of code as the action for the method. The follow- ing listing is equivalent to listing 5.1 but uses the shorter syntax. Button button = new Button(); button.Text = "Click me"; button.Click += delegate { Console.WriteLine("LogPlain"); }; button.KeyPress += delegate { Console.WriteLine("LogKey"); }; button.MouseClick += delegate { Console.WriteLine("LogMouse"); }; Form form = new Form(); form.AutoSize = true; form.Controls.Add(button); Application.Run(form); Normally you’d have to write each subscription as something like this: button.Click += delegate(object sender, EventArgs e) {... }; Listing 5.9 Subscribing to events with anonymous methods that ignore parameters 147Inline delegate actions with anonymous methods That wastes a lot of space for little reason—you don’t need the values of the parame- ters, so the compiler lets you get away with not specifying them at all. I’ve found this shortcut most useful when it comes to implementing my own events. For example, I get sick of having to perform a nullity check before raising an event. One way of getting around this is to make sure that the event starts off with a handler, which is then never removed. As long as the handler doesn’t do anything, all you lose is a tiny bit of performance. Before C# 2, you had to explicitly create a method with the right signature, which usually wasn’t worth the benefit, but now you can write code like this: public event EventHandler Click = delegate {}; From then on, you can just call Click without any nullity tests. You should be aware of one trap related to this parameter wildcarding feature—if the anonymous method could be converted to multiple delegate types (for example, to call different method overloads), the compiler needs more help. To show you what I mean, let’s take the same troublesome example we looked at with method group conversions: starting a new thread. There are four thread constructors in .NET 2.0: public Thread(ParameterizedThreadStart start) public Thread(ThreadStart start) public Thread(ParameterizedThreadStart start, int maxStackSize) public Thread(ThreadStart start, int maxStackSize) These are the two delegate types involved: public delegate void ThreadStart() public delegate void ParameterizedThreadStart(object obj) Now, consider the following three attempts to create a new thread: new Thread(delegate() { Console.WriteLine("t1"); }); new Thread(delegate(object o) { Console.WriteLine("t2"); }); new Thread(delegate { Console.WriteLine("t3"); }); The first and second lines contain parameter lists—the compiler knows that it can’t convert the anonymous method in the first line into a ParameterizedThreadStart or convert the anonymous method in the second line into a ThreadStart. Those lines compile because there’s only one applicable constructor overload in each case. The third line, though, is ambiguous—the anonymous method can be converted into either delegate type, so both of the single parameter constructor overloads are appli- cable. In this situation, the compiler throws its hands up and issues an error. You can solve this either by specifying the parameter list explicitly or casting the anonymous method to the right delegate type. Hopefully what you’ve seen of anonymous methods so far will have provoked some thought about your own code and made you consider where you could use these tech- niques to good effect. Indeed, even if anonymous methods could only do what you’ve already seen, they’d be very useful. But there’s more to anonymous methods than just avoiding the inclusion of an extra method in your code. Anonymous methods are 148 CHAPTER 5 Fast-tracked delegates C# 2’s implementation of a feature known elsewhere as closures by way of captured vari- ables. The next section explains both of these terms and shows how anonymous meth- ods can be extremely powerful—and confusing if you’re not careful. 5.5 Capturing variables in anonymous methods I don’t like having to give warnings, but I think it makes sense to include one here: if this topic is new to you, then don’t start this section until you’re feeling reasonably awake and have a bit of time to spend on it. I don’t want to alarm you unnecessarily, and you should feel confident that there’s nothing here so insanely complicated that you won’t be able to understand it with a little effort. It’s just that captured variables can be somewhat confusing to start with, partly because they overturn some of your existing knowledge and intuition. Stick with it, though! The payback can be massive in terms of code simplicity and readability. This topic will also be crucial when we look at lambda expressions and LINQ in C# 3, so it’s worth the investment. Let’s start with a few definitions. 5.5.1 Defining closures and different types of variables The concept of closures is an old one, first implemented in Scheme, but it’s been gain- ing more prominence in recent years as more mainstream languages have taken it on board. The basic idea is that a function6 is able to interact with an environment beyond the parameters provided to it. That’s all there is to it in abstract terms, but to understand how it applies to C# 2, we need a couple more terms:  An outer variable is a local variable or parameter (excluding ref and out param- eters) whose scope includes an anonymous method. The this reference also counts as an outer variable of any anonymous method within an instance mem- ber of a class.  A captured outer variable (usually shortened to captured variable) is an outer vari- able that’s used within an anonymous method. To go back to closures, the func- tion part is the anonymous method, and the environment it can interact with is the set of variables captured by it. That’s all very dry and may be hard to imagine, but the main thrust is that an anony- mous method can use local variables defined in the same method that declares it. This may not sound like a big deal, but in many situations it’s enormously handy—you can use contextual information that you have on hand rather than having to set up extra types just to store data you already know. We’ll look at some useful concrete examples soon, I promise—but first it’s worth looking at some code to clarify these definitions. Listing 5.10 provides an example with a number of local variables, and it’s a single method, so it can’t be run on its own. I’m not going to explain how it would work or 6 This is general computer science terminology, not C# terminology. 149Capturing variables in anonymous methods what it would do yet; I just want to discuss how the different variables are classified. Again, we’ll use the MethodInvoker delegate type for simplicity. void EnclosingMethod() { int outerVariable = 5; string capturedVariable = "captured"; if (DateTime.Now.Hour == 23) { int normalLocalVariable = DateTime.Now.Minute; Console.WriteLine(normalLocalVariable); } MethodInvoker x = delegate() { string anonLocal = "local to anonymous method"; Console.WriteLine(capturedVariable + anonLocal); }; x(); } Let’s go through all the variables from the simplest to the most complicated:  normalLocalVariable D isn’t an outer variable because there are no anony- mous methods within its scope. It behaves exactly the way that local variables always have.  anonLocal E isn’t an outer variable either, but it’s local to the anonymous method, not to EnclosingMethod. It’ll only exist (in terms of being present in an executing stack frame) when the delegate instance is invoked.  outerVariable B is an outer variable because the anonymous method is declared within its scope. But the anonymous method doesn’t refer to it, so it’s not captured.  capturedVariable C is an outer variable because the anonymous method is declared within its scope, and it’s captured by virtue of being used at F. Okay, you now understand the terminology, but we’re not a lot closer to seeing what captured variables do. I suspect you could guess the output if you ran the method from listing 5.10, but there are some other cases that would probably surprise you. We’ll start off with a simple example and build up to more complex ones. 5.5.2 Examining the behavior of captured variables When a variable is captured, it really is the variable that’s captured by the anonymous method, not its value at the time the delegate instance was created. You’ll see later that this has far-reaching consequences, but first you need to understand what that means for a relatively straightforward situation. Listing 5.10 Examples of variable kinds with respect to anonymous methods Outer variable (uncaptured) B Outer variable captured by anonymous methodC Local variable of normal methodD Local variable of anonymous method E Capture of outer variableF 150 CHAPTER 5 Fast-tracked delegates The following listing has a captured variable and an anonymous method that both prints out and changes the variable. You’ll see that changes to the variable from out- side the anonymous method are visible within the anonymous method, and vice versa. string captured = "before x is created"; MethodInvoker x = delegate { Console.WriteLine(captured); captured = "changed by x"; }; captured = "directly before x is invoked"; x(); Console.WriteLine(captured); captured = "before second invocation"; x(); The output of listing 5.11 is as follows: directly before x is invoked changed by x before second invocation Let’s look at how this happens. First, you declare the variable captured and set its value with a perfectly normal string literal. So far, there’s nothing special about the variable. You then declare x and set its value using an anonymous method that cap- tures captured. The delegate instance will always print out the current value of captured and then set it to “changed by x.” Don’t forget that creating this delegate instance doesn’t execute it. To make it absolutely clear that just creating the delegate instance doesn’t read the variable and stash its value away somewhere, you now change the value of captured to “directly before x is invoked.” You then invoke x for the first time. It reads the value of captured and prints it out—the first line of output. It sets the value of captured to “changed by x” and returns. When the delegate instance returns, the normal method continues in the usual way. It prints out the current value of captured, giving the sec- ond line of output. The normal method then changes the value of captured yet again (this time to “before second invocation”) and invokes x for the second time. The current value of captured is printed out, giving the last line of output. The delegate instance changes the value of captured to “changed by x” and returns, at which point the normal method has run out of code and it’s done. That’s a lot of detail about how a short piece of code works, but there’s really only one crucial idea in it: the captured variable is the same one that the rest of the method uses. For some people, that’s hard to grasp; for others it comes naturally. Don’t worry if it’s tricky to start with—it’ll get easier over time. Listing 5.11 Accessing a variable both inside and outside an anonymous method 151Capturing variables in anonymous methods Even if you’ve understood everything easily so far, you may be wondering why you’d want to do any of this. It’s about time we had an example that was actually useful. 5.5.3 What’s the point of captured variables? To put it simply, captured variables eliminate the need to write extra classes just to store the information a delegate needs to act on, beyond what it’s passed via parame- ters. Before ParameterizedThreadStart existed, if you wanted to start a new (non- threadpool) thread and give it some information—the URL of a page to fetch, for instance—you had to create an extra type to hold the URL and put the action of the ThreadStart delegate instance in that type. Even with ParameterizedThreadStart, your method had to accept a parameter of type object and cast it to the type you really wanted. It was an ugly way of achieving something that should’ve been simple. As another example, suppose you had a list of people and wanted to write a method that would return a second list containing all the people who were under a given age. List has a method called FindAll that returns another list of every- thing matching the specified predicate. Before anonymous methods and captured variables, it wouldn’t have made much sense for List.FindAll to exist, because of all the hoops you’d have to go through in order to create the right delegate to start with. It would’ve been simpler to do all the iteration and copying manually. With C# 2, though, you can do it all very easily: List FindAllYoungerThan(List people, int limit) { return people.FindAll(delegate (Person person) { return person.Age < limit; } ); } Here you’re capturing the limit parameter within the delegate instance—if you’d had anonymous methods but not captured variables, you could’ve performed a test against a hardcoded limit, but not one that was passed into the method as a parame- ter. I hope you’ll agree that this approach is neat: it expresses exactly what you want to do with much less fuss about exactly how it should happen than you’d have seen in a C# 1 version. (It’s even neater in C# 3, admittedly…)7 It’s relatively rare that you come across a situation where you need to write to a captured variable, but again that can have its uses. Still with me? Good. So far, you’ve only used the delegate instance within the method that creates it. That doesn’t raise many questions about the lifetime of the captured variables—but what would happen if the delegate instance escaped into the big bad world? How would it cope after the method that created it had finished? 7 In case you’re wondering: return people.Where(person => person.Age < limit); 152 CHAPTER 5 Fast-tracked delegates 5.5.4 The extended lifetime of captured variables The simplest way of tackling this topic is to state a rule, give an example, and then think about what would happen if the rule weren’t in place. Here we go: A captured variable lives for at least as long as any delegate instance referring to it. Don’t worry if it doesn’t make a lot of sense yet—that’s what the example is for. The following listing shows a method that returns a delegate instance. That delegate instance is created using an anonymous method that captures an outer variable. So, what’ll happen when the delegate is invoked after the method has returned? static MethodInvoker CreateDelegateInstance() { int counter = 5; MethodInvoker ret = delegate { Console.WriteLine(counter); counter++; }; ret(); return ret; } ... MethodInvoker x = CreateDelegateInstance(); x(); x(); The output of listing 5.12 consists of the numbers 5, 6, and 7 on separate lines. The first line of output comes from the invocation of the delegate instance within Create- DelegateInstance, so it makes sense that the value of counter is available at that point. But what about after the method has returned? Normally you’d consider counter to be on the stack, so when the stack frame for CreateDelegateInstance is destroyed, you’d expect counter to effectively vanish…and yet subsequent invocations of the returned delegate instance seem to keep using it. The secret is to challenge the assumption that counter is on the stack in the first place. It isn’t. The compiler has actually created an extra class to hold the variable. The CreateDelegateInstance method has a reference to an instance of that class so it can use counter, and the delegate has a reference to the same instance, which lives on the heap in the normal way. That instance isn’t eligible for garbage collection until the delegate is ready to be collected. Some aspects of anonymous methods are very compiler-specific (different compil- ers could achieve the same semantics in different ways), but it’s hard to see how the specified behavior could be achieved without using an extra class to hold the captured variable. Note that if you only capture this, no extra types are required—the com- piler just creates an instance method to act as the delegate’s action. As I mentioned before, you probably shouldn’t worry about the stack and heap details too much, but Listing 5.12 Demonstration of a captured variable having its lifetime extended 153Capturing variables in anonymous methods it’s worth knowing what the compiler is capable of doing, just in case you get confused as to how the specified behavior is even possible. Okay, so local variables can live on even after a method has returned. You may be wondering what I could possibly throw at you next—how about multiple delegates capturing different instances of the same variable? It sounds crazy, so it’s just the kind of thing you should be expecting by now. 5.5.5 Local variable instantiations On a good day, captured variables act exactly the way I expect them to at a glance. On a bad day, I’m still surprised when I’m not careful. When there are problems, it’s almost always due to my forgetting how many “instances” of local variables I’m actually creating. A local variable is said to be instantiated each time execution enters the scope where it’s declared. Here’s a simple example comparing two very similar bits of code: In the good old days, it was reasonable to say that pieces of code like this were seman- tically identical. Indeed, they’d usually compile to the same IL—and they still will, if there aren’t any anonymous methods involved. All the space for local variables is allo- cated on the stack at the start of the method, so there’s no cost to redeclaring the vari- able for each iteration of the loop.8 In our new terminology, the single variable will be instantiated only once, but the multiple variable will be instantiated 10 times—it’s as if there were 10 local variables, all called multiple, which were created one after another. I’m sure you can see where I’m going—when a variable is captured, it’s the rele- vant “instance” of the variable that’s captured. If you captured multiple inside the loop, the variable captured in the first iteration would be different from the variable captured the second time round, and so on. The following listing shows exactly this effect. List list = new List(); for (int index = 0; index < 5; index++) { int counter = index * 10; int single; for (int i = 0; i < 10; i++) { single = 5; Console.WriteLine(single + i); } for (int i = 0; i < 10; i++) { int multiple = 5; Console.WriteLine(multiple + i); } 8 In my view, it’s also cleaner to redeclare the variable unless you explicitly need to maintain its value between iterations. Listing 5.13 Capturing multiple variable instantiations with multiple delegates Instantiates counterB 154 CHAPTER 5 Fast-tracked delegates list.Add(delegate { Console.WriteLine(counter); counter++; }); } foreach (MethodInvoker t in list) { t(); } list[0](); list[0](); list[0](); list[1](); Listing 5.13 creates five different delegate instances C—one for each time you go around the loop. Invoking the delegate will print out the value of counter and then increment it. Because counter is declared inside the loop, it’s instantiated for each iteration B, and each delegate captures a different variable. When you go through and invoke each delegate D, you see the different values initially assigned to counter: 0, 10, 20, 30, 40. Just to hammer the point home, when you then go back to the first delegate instance and execute it three more times E, it keeps going from where that instance’s counter variable had left off: 1, 2, 3. Finally you execute the second dele- gate instance F, and that keeps going from where that instance’s counter variable had left off: 11. As you can see, each of the delegate instances has captured a different variable. Before we leave this example, I should point out what would’ve happened if you’d captured index—the variable declared by the for loop—instead of counter. In this case, all the delegates would have shared the same variable. The output would’ve been the numbers 5 to 13; 5 first because the last assignment to index before the loop ter- minates would’ve set it to 5, and then incrementing the same variable regardless of which delegate was involved. You’d see the same behavior with a foreach loop (in C# 2–4): the variable declared by the initial part of the loop is only instantiated once. It’s easy to get this wrong! If you want to capture the value of a loop variable for that particular iteration of the loop, introduce another variable within the loop, copy the loop variable’s value into it, and capture that new variable—effectively what you did in listing 5.13 with the counter variable. THIS CHANGES IN C# 5… Though the behavior in a for loop is reasonable— the variable does appear to be declared just once, after all—it’s more surpris- ing in the foreach case. In fact, it’s almost always wrong to capture a foreach iteration variable in an anonymous method that’s going to exist beyond the immediate iteration. (It’s fine if the delegate instance is only used within that iteration.) This has caused problems for so many developers that the C# team has changed the semantics of foreach for C# 5 to make it act more natu- rally—as if each iteration had its own separate variable. See section 16.1 for more details. Prints and increments captured variable C Executes all five delegate instances D Executes first one three more timesE Executes second one againF 155Capturing variables in anonymous methods For our final example, let’s look at something really nasty—sharing some captured variables but not others. 5.5.6 Mixtures of shared and distinct variables Let me say before I show you this next example that it’s not code I’d recommend. In fact, the whole point of presenting it is to show how if you try to use captured variables in too complicated a fashion, things can get tricky really fast. The following listing cre- ates two delegate instances that each capture “the same” two variables. But the story gets more convoluted when you look at what’s actually captured. MethodInvoker[] delegates = new MethodInvoker[2]; int outside = 0; for (int i = 0; i < 2; i++) { int inside = 0; delegates[i] = delegate { Console.WriteLine ("({0},{1})", outside, inside); outside++; inside++; }; } MethodInvoker first = delegates[0]; MethodInvoker second = delegates[1]; first(); first(); first(); second(); second(); How long would it take you to predict the output from listing 5.14 (even with the annotations)? Frankly, it would take me a while—longer than I like to spend under- standing code. Just as an exercise, though, let’s look at what happens. First consider the outside variable B. The scope it’s declared in is only entered once, so it’s a straightforward case—there’s only ever one of it, effectively. The inside variable C is a different matter—each loop iteration instantiates a new one. That means that when you create the delegate instance D, the outside variable is shared between the two delegate instances, but each of them has its own inside variable. After the loop has ended, you call the first delegate instance you created three times. Because it’s incrementing both of its captured variables each time, and both of them started off as 0, you see (0,0), then (1,1), and then (2,2). The difference between the two variables in terms of scope becomes apparent when you execute the second delegate instance. It has a different inside variable, so that still has its initial Listing 5.14 Capturing variables in different scopes. Warning: nasty code ahead! Instantiates variable onceB Instantiates variable multiple timesC Captures variables with anonymous methodD 156 CHAPTER 5 Fast-tracked delegates value of 0, but the outside variable is the one you’ve already incremented three times. The output from calling the second delegate twice is therefore (3,0), and then (4,1). Just for the sake of interest, let’s think about how this is implemented—at least with Microsoft’s C# 2 compiler. What happens is that one extra class is generated to hold the outside variable, and another one is generated to hold an inside variable and a reference to the first extra class. Essentially, each scope that contains a captured vari- able gets its own type, with a reference to the next scope out that contains a captured variable. In this example, there were two instances of the type holding inside, and they both referred to the same instance of the type holding outside. Other imple- mentations may vary, but this is the most obvious way of doing things. Figure 5.1 shows the values after listing 5.14 has executed. (The names in the figure aren’t the ones that the compiler would generate, but they’re close enough. Note that the delegate instances would also have other members in reality—only the target is interesting here, though.) Even after you understand this code fully, it’s still a good template for experiment- ing with other elements of captured variables. As I noted earlier, certain elements of variable capture are implementation-specific, and it’s often useful to refer to the spec- ification to see what’s guaranteed. But it’s also important to play with code to see what happens. It’s possible that there are situations where code like listing 5.14 would be the sim- plest and clearest way of expressing the desired behavior, but I’d have to see it to believe it, and I’d certainly want comments in the code to explain what would happen. So, when is it appropriate to use captured variables, and what do you need to look out for? 5.5.7 Captured variable guidelines and summary Hopefully this section has convinced you to be very careful with captured variables. They make good logical sense (and almost any change to make them simpler would <>_Generated1 inside <>_outside first <>_Generated1 inside <>_outside second outside <>_Generated2 3 2 5 first target ref second target ref ref ref Figure 5.1 Snapshot of multiple captured variable scopes in memory 157Capturing variables in anonymous methods probably make them either less useful or less logical), but they also make it easy to produce horribly complicated code. Don’t let that discourage you from using them sensibly, though—they can save you masses of tedious code, and when they’re used appropriately they can be the most readable way of getting the job done. But what counts as sensible? Here are some suggestions for using captured variables:  If code that doesn’t use captured variables is just as simple as code that does, don’t use them.  Before capturing a variable declared by a for or foreach statement, consider whether your delegate is going to live beyond the loop iteration, and whether you want it to see the subsequent values of that variable. If not, create another variable inside the loop that just copies the value you do want. (In C# 5 you don’t need to worry about foreach statements, but you still need to take care in for statements.)  If you create multiple delegate instances (whether in a loop or explicitly) that capture variables, put thought into whether you want them to capture the same variable.  If you capture a variable that doesn’t actually change (either in the anonymous method or the enclosing method body), you don’t need to worry as much.  If the delegate instances you create never escape from the method—in other words, they’re never stored anywhere else, or returned, or used for starting threads—life is a lot simpler.  Consider the extended lifetime of any captured variables in terms of garbage collection. This is normally not an issue, but if you capture an object that’s expensive in terms of memory, it may be significant. The first point is the golden rule. Simplicity is a good thing, so any time the use of a captured variable makes your code simpler after you’ve factored in the additional inherent complexity of forcing your code’s maintainers to understand what the cap- tured variable does, use it. You need to include that extra complexity in your consider- ations, that’s all—don’t just go for minimal line count. We’ve covered a lot of ground in this section, and I’m aware that it can be hard to take in. I’ve listed the most important things to remember next, so that if you need to come back to this section later, you can jog your memory without having to read through the whole thing again:  The variable is captured—not its value at the point of delegate instance creation.  Captured variables have lifetimes extended to at least that of the capturing delegate.  Multiple delegates can capture the same variable…  …but within loops, the same variable declaration can effectively refer to differ- ent variable “instances.” 158 CHAPTER 5 Fast-tracked delegates  for loop declarations create variables that live for the duration of the loop— they’re not instantiated on each iteration. The same is true for foreach state- ments before C# 5.  Extra types are created, where necessary, to hold captured variables.  Be careful! Simple is almost always better than clever. You’ll see more variables being captured when we look at C# 3 and its lambda expres- sions, but for now you may be relieved to hear that we’ve finished our rundown of the new C# 2 delegate features. 5.6 Summary C# 2 has radically changed the ways in which delegates can be created, and in doing so it’s opened up the framework to a more functional style of programming. There are more methods in .NET 2.0 that take delegates as parameters than there were in .NET 1.0/1.1, and this trend continues in .NET 3.5. The List type is the best exam- ple of this, and it’s a good test bed for checking your skills with using anonymous methods and captured variables. Programming in this way requires a slightly different mind-set—you must be able to take a step back and consider what the ultimate aim is, and whether it’s best expressed in the traditional C# manner, or whether a functional approach makes things clearer. All the changes to delegate handling are useful, but they add complexity to the language, particularly when it comes to captured variables. Closures are always tricky in terms of determining exactly how the available environment is shared, and C# is no different in this respect. The reason the concept has lasted so long, though, is that it can make code simpler to understand and more immediate. The balancing act between complexity and simplicity is always a difficult one, and it’s worth being cau- tious to start with. But over time you should expect to get better at working with cap- tured variables and understanding how they behave. LINQ encourages their use even further, and a great deal of modern, idiomatic C# code uses closures frequently. Anonymous methods aren’t the only change in C# 2 that involves the compiler cre- ating extra types behind the scenes and doing devious things with variables that appear to be local. You’ll see a lot more of this in the next chapter, where the com- piler effectively builds a whole state machine for you, in order to make it easier for you to implement iterators. 159 Implementing iterators the easy way The iterator pattern is an example of a behavioral pattern—a design pattern that sim- plifies communication between objects. It’s one of the simplest patterns to under- stand, and it’s incredibly easy to use. In essence, it allows you to access all the elements in a sequence of items without caring about what kind of sequence it is— an array, a list, a linked list, or none of the above. This can be effective for building a data pipeline, where an item of data enters the pipeline and goes through a num- ber of different transformations or filters before coming out the other end. Indeed, this is one of the core patterns of LINQ, as you’ll see in part 3 of the book. In .NET, the iterator pattern is encapsulated by the IEnumerator and IEnumerable interfaces and their generic equivalents. (The naming is unfortunate—the pattern is normally called iteration to avoid confusing it with other meanings of the word enu- meration. I’ve used iterator and iterable throughout this chapter.) If a type implements This chapter covers  Implementing iterators in C# 1  Iterator blocks in C# 2  Sample iterator usage  Iterators as coroutines 160 CHAPTER 6 Implementing iterators the easy way IEnumerable, that means it can be iterated over; calling the GetEnumerator method will return the IEnumerator implementation, which is the iterator itself. You can think of the iterator as being like a database cursor: a position within the sequence. The iterator can only move forward within the sequence, and there can be many iterators operating on the same sequence at the same time. As a language, C# 1 has built-in support for consuming iterators using the foreach statement. This makes it easy to iterate over collections—easier than using a straight for loop—and it’s nicely expressive. The foreach statement compiles down to calls to the GetEnumerator and MoveNext methods and the Current property, with support for disposing of the iterator afterward if IDisposable has been implemented. It’s a small but useful piece of syntactic sugar. In C# 1, though, implementing an iterator is a relatively difficult task. C# 2 makes this much simpler, which can sometimes lead to the iterator pattern being worth implementing in cases where previously it would’ve caused more work than it saved. In this chapter, we’ll look at what’s required to implement an iterator and at the support given by C# 2. After we’ve looked at the syntax in detail, we’ll examine a few examples from the real world, including an exciting (if slightly off-the-wall) use of the iteration syntax in a concurrency library from Microsoft. I’ve held off providing the examples until the end of the description, because there isn’t very much to learn, and the examples will be a lot clearer when you can understand what the code is doing. If you really want to read the examples first, they’re in sections 6.3 and 6.4. As in other chapters, let’s start off by looking at what life was like before C# 2. We’ll implement an iterator the hard way. 6.1 C# 1: The pain of handwritten iterators You’ve already seen one example of an iterator implementation in section 3.4.3, when we looked at what happens when you iterate over a generic collection. In some ways, that was harder than a real C# 1 iterator implementation would’ve been, because you implemented the generic interfaces as well—but it was also easier in other ways because it wasn’t actually iterating over anything useful. To put the C# 2 features into context, we’ll first implement an iterator that’s about as simple as it can be while still providing real functionality. Suppose you wanted a new type of collection based on a circular buffer. In this example, you’ll implement IEnumerable so that users of your new class can easily iterate over all the values in the collection. We’ll ignore the guts of the collection here and just concentrate on the iteration side. Your collection will store its values in an array (object[]—no generics here), and the collection will have the interesting feature that you can set its logical starting point—so if the array had five elements, you could set the start point to 2 and expect elements 2, 3, 4, 0, and then 1 to be returned. I won’t show the full circular buffer code here, but it’s in the downloadable code. To make the class easy to demonstrate, you’ll provide both the values and the start- ing point in the constructor, so you should be able to write code such as the following to iterate over the collection. 161C# 1: The pain of handwritten iterators object[] values = {"a", "b", "c", "d", "e"}; IterationSample collection = new IterationSample(values, 3); foreach (object x in collection) { Console.WriteLine (x); } Running listing 6.1 should (eventually) produce output of d, e, a, b, and finally c because you specified a starting point of 3. Now that you know what you need to achieve, let’s look at the skeleton of the class, as shown in the following listing. using System; using System.Collections; public class IterationSample : IEnumerable { object[] values; int startingPoint; public IterationSample(object[] values, int startingPoint) { this.values = values; this.startingPoint = startingPoint; } public IEnumerator GetEnumerator() { throw new NotImplementedException(); } } You haven’t implemented GetEnumerator yet, but the rest of the code is ready to go. And how do you go about writing the GetEnumerator code? The first thing to under- stand is that you need to store some state somewhere. One important aspect of the iterator pattern is that you don’t return all of the data in one go—the client asks for one element at a time. That means you need to keep track of how far you’ve already gone through your array. The stateful nature of iterators will be important when we look at what the C# 2 compiler does, so keep a close eye on the state required in this example. Where should this state live? Suppose you tried to put it in the IterationSample class, making that implement IEnumerator as well as IEnumerable. At first glance, this looks like a good plan—after all, the data is in the right place, including the starting point. Your GetEnumerator method could just return this. But there’s a big problem with this approach: if GetEnumerator is called several times, several independent itera- tors should be returned. For instance, you should be able to use two foreach state- ments, one inside another, to get all possible pairs of values. The two iterators need to be independent, which suggests you need to create a new object each time Listing 6.1 Code using the (as yet unimplemented) new collection type Listing 6.2 Skeleton of the new collection type, with no iterator implementation 162 CHAPTER 6 Implementing iterators the easy way GetEnumerator is called. You could still implement the functionality directly within IterationSample, but then you’d have a class that didn’t have a single clear responsi- bility—it would be pretty confusing. Instead, let’s create another class to implement the iterator itself. You can use the fact that in C# a nested type has access to its enclosing type’s private members, which means you can store a reference to the parent IterationSample, along with the state of how many iterations you’ve performed so far. This is shown in the following listing. class IterationSampleIterator : IEnumerator { IterationSample parent; int position; internal IterationSampleIterator(IterationSample parent) { this.parent = parent; position = -1; } public bool MoveNext() { if (position != parent.values.Length) { position++; } return position < parent.values.Length; } public object Current { get { if (position == -1 || position == parent.values.Length) { throw new InvalidOperationException(); } int index = position + parent.startingPoint; index = index % parent.values.Length; return parent.values[index]; } } public void Reset() { position = -1; } } What a lot of code to perform such a simple task! You remember the original collec- tion of values you’re iterating over B and keep track of where you’d be in a simple zero-based array C. To return an element, you offset that index by the starting Listing 6.3 Nested class implementing the collection’s iterator Collection you’re iterating over B How far you’ve iteratedC Starts before first elementD Increments position if still goingE Prevents access before first or after last element F Implements wraparound G Moves back to before first element H 163C# 2: Simple iterators with yield statements point G. In keeping with the interface, you consider your iterator to start logically before the first element D, so the client will have to call MoveNext before using the Current property for the first time. The conditional increment at E makes the test at F simple and correct even if MoveNext is called again after it’s first reported that no more data is available. To reset the iterator, you set the logical position back to before the first element H. Most of the logic involved is fairly straightforward, although there’s plenty of room for off-by-one errors; my first implementation failed its unit tests for precisely that rea- son. The good news is that it works, and that you only need to implement IEnumerable in IterationSample to complete the example: public IEnumerator GetEnumerator() { return new IterationSampleIterator(this); } I won’t reproduce the combined code here, but it’s available on the book’s website, including listing 6.1, which now produces the expected output. It’s worth bearing in mind that this is a relatively simple example—there’s not a lot of state to keep track of, and there’s no attempt to check whether the collection has changed between iterations. With such a large burden involved to implement a simple iterator, you shouldn’t be surprised at how rarely this pattern was implemented in C# 1. Developers were generally happy to use foreach on the collections provided by the framework, but more direct (and collection-specific) access when it came to their own collections. It took about 40 lines of code to implement the iterator in C# 1. Let’s see if C# 2 can do any better. 6.2 C# 2: Simple iterators with yield statements I’ve always been the kind of person who likes to stay up until midnight on Christmas Eve so I can open a present as soon as Christmas Day arrives. In the same way, I’d find it almost impossible to wait any significant amount of time before showing you how neat the solution is in C# 2. 6.2.1 Introducing iterator blocks and yield return This chapter wouldn’t exist if C# 2 didn’t have a powerful feature that cuts down the amount of code you have to write to implement iterators. In some other topics the amount of code has only been reduced slightly, or the changes have just made something more elegant. In this case, though, the amount of code required is reduced massively. The following listing shows the complete implementation of the GetEnumerator method in C# 2. 164 CHAPTER 6 Implementing iterators the easy way public IEnumerator GetEnumerator() { for (int index = 0; index < values.Length; index++) { yield return values[(index + startingPoint) % values.Length]; } } Four lines of implementation, two of which are just braces. To be clear, that replaces the whole of the IterationSampleIterator class. Completely. At least in the source code…Later on you’ll see what the compiler has done behind the scenes and some of the quirks of the implementation it’s provided, but for the moment let’s look at the source code. The method looks perfectly normal until you see the use of yield return. That’s what tells the C# compiler that this isn’t a normal method, but one implemented with an iterator block. The method is declared to return an IEnumerator, and you can only use iterator blocks to implement methods1 that have a return type of IEnumerable, IEnumerator, or one of the generic equivalents. The yield type of the iterator block is object if the declared return type of the method is a nongeneric interface, or it’s the type argument of the generic interface otherwise. For instance, a method declared to return IEnumerable would have a yield type of string. No normal return statements are allowed within iterator blocks—only yield return. All yield return statements in the block must try to return a value compati- ble with the yield type of the block. In the previous example, you couldn’t write yield return 1; in a method declared to return IEnumerable. RESTRICTIONS ON YIELD RETURN There are a few further restrictions on yield statements. You can’t use yield return inside a try block if it has any catch blocks, and you can’t use yield return or yield break (which we’ll come to shortly) in a finally block. That doesn’t mean you can’t use try/ catch or try/finally blocks inside iterators—it just restricts what you can do in them. If you want to know more about why these restrictions exist, Eric Lippert has a whole series of blog posts about these and other design decisions involving iterators: see http://mng.bz/EJ97. The big idea that you need to get your head around when it comes to iterator blocks is that although you’ve written a method that looks like it executes sequentially, what you’ve actually asked the compiler to do is create a state machine for you. This is neces- sary for exactly the same reason you had to put so much effort into implementing the iterator in C# 1—the caller only wants to see one element at a time, so you need to keep track of what you were doing when you last returned a value. Listing 6.4 Iterating through the sample collection with C# 2 and yield return 1 Or properties, as you’ll see later on. You can’t use an iterator block in an anonymous method, though. 165C# 2: Simple iterators with yield statements When the compiler encounters an iterator block, it creates a nested type for the state machine. This type remembers exactly where you are within the block and the values of local variables (including parameters). The generated class is somewhat simi- lar to the longhand implementation you wrote earlier, in that it keeps all the necessary state as instance variables. Let’s think about what this state machine has to do in order to implement the iterator:  It has to have some initial state.  Whenever MoveNext is called, it has to execute code from the GetEnumerator method until you’re ready to provide the next value (in other words, until you hit a yield return statement).  When the Current property is used, it has to return the last value you yielded.  It has to know when you’ve finished yielding values so that MoveNext can return false. The second point in this list is the tricky one, because the state machine always needs to restart the code from the point it had previously reached. Keeping track of the local variables (as they appear in the method) isn’t too hard—they’re represented by instance variables in the state machine. The restarting aspect is trickier, but the good news is that unless you’re writing a C# compiler yourself, you needn’t care about how it’s achieved: the result from a black-box point of view is that it just works. You can write perfectly normal code within the iterator block, and the compiler is responsible for making sure that the flow of execution is exactly as it would be in any other method. The difference is that a yield return statement appears to only temporarily exit the method—you could think of it as being paused, effectively. Next we’ll examine the flow of execution in more detail and in a more visual way. 6.2.2 Visualizing an iterator’s workflow It may help to think about how iterators execute in terms of a sequence diagram. Rather than drawing the diagram by hand, the following listing prints it out. The iter- ator provides a sequence of numbers (0, 1, 2, -1) and then finishes. The interesting part isn’t the numbers provided so much as the flow of the code. static readonly string Padding = new string(' ', 30); static IEnumerable CreateEnumerable() { Console.WriteLine("{0}Start of CreateEnumerable()", Padding); for (int i=0; i < 3; i++) { Console.WriteLine("{0}About to yield {1}", Padding, i); yield return i; Console.WriteLine("{0}After yield", Padding); } Console.WriteLine("{0}Yielding final value", Padding); Listing 6.5 Showing the sequence of calls between an iterator and its caller 166 CHAPTER 6 Implementing iterators the easy way yield return -1; Console.WriteLine("{0}End of CreateEnumerable()", Padding); } ... IEnumerable iterable = CreateEnumerable(); IEnumerator iterator = iterable.GetEnumerator(); Console.WriteLine("Starting to iterate"); while (true) { Console.WriteLine("Calling MoveNext()..."); bool result = iterator.MoveNext(); Console.WriteLine("... MoveNext result={0}", result); if (!result) { break; } Console.WriteLine("Fetching Current..."); Console.WriteLine("... Current result={0}", iterator.Current); } Listing 6.5 isn’t pretty, particularly around the iteration side of things. In the normal course of events, you could just use a foreach loop, but to show exactly what’s hap- pening when, I had to break the use of the iterator out into pieces. This code broadly does what foreach does, although foreach also calls Dispose at the end. This is important for iterator blocks, as we’ll discuss shortly. As you can see, there’s no differ- ence in the syntax within the iterator method, even though this time you’re returning IEnumerable instead of IEnumerator. Usually you’ll only want to return IEnumerator in order to implement IEnumerable; if you want to just yield a sequence from a method, return IEnumerable instead. Here’s the output from listing 6.5: Starting to iterate Calling MoveNext()... Start of CreateEnumerable() About to yield 0 ... MoveNext result=True Fetching Current... ... Current result=0 Calling MoveNext()... After yield About to yield 1 ... MoveNext result=True Fetching Current... ... Current result=1 Calling MoveNext()... After yield About to yield 2 ... MoveNext result=True Fetching Current... ... Current result=2 Calling MoveNext()... After yield 167C# 2: Simple iterators with yield statements Yielding final value ... MoveNext result=True Fetching Current... ... Current result=-1 Calling MoveNext()... End of CreateEnumerable() ... MoveNext result=False There are several important things to note in this output:  None of the code in CreateEnumerable is called until the first call to MoveNext.  It’s only when you call MoveNext that any real work gets done. Fetching Current doesn’t run any of your code.  The code stops executing at yield return and picks up again just afterward at the next call to MoveNext.  You can have multiple yield return statements in different places in the method.  The code doesn’t end at the last yield return. Instead, the call to MoveNext that causes you to reach the end of the method is the one that returns false. The first point is particularly important, because it means you can’t use an iterator block for any code that has to be executed immediately when the method is called, such as code for argument validation. If you put normal checking into a method implemented with an iterator block, it won’t behave nicely. You’ll almost certainly fall foul of this at some point—it’s an extremely common error, and hard to understand until you think about what the iterator block is doing. You’ll see the solution to the problem in section 6.3.3. There are two things you haven’t seen yet—an alternative way of halting the itera- tion, and how finally blocks work in this somewhat odd form of execution. Let’s take a look at them now. 6.2.3 Advanced iterator execution flow In normal methods, the return statement has two effects. First, it supplies the value the caller sees as the return value. Second, it terminates the execution of the method, executing any appropriate finally blocks on the way out. You’ve seen that the yield return statement temporarily exits the method, but only until MoveNext is called again, and we haven’t examined the behavior of finally blocks at all. How can you really stop the method, and what happens to all of those finally blocks? We’ll start with a fairly simple construct—the yield break statement. ENDING AN ITERATOR WITH YIELD BREAK You can always find a way to give a method a single exit point, and many people work hard to achieve this.2 The same techniques can be applied in iterator blocks. But 2 I find that the hoops you have to jump through to achieve this often make the code much harder to read than having multiple return points, especially as try/finally is available for cleanup and you need to account for the possibility of exceptions occurring anyway. The point is that it can be done. 168 CHAPTER 6 Implementing iterators the easy way should you wish to have an early out, the yield break statement is your friend. This effectively terminates the iterator, making the current call to MoveNext return false. The following listing demonstrates this by counting up to 100 but stopping early if it runs out of time. This code also demonstrates the use of a method parameter in an iterator block and proves that the name of the method is irrelevant.3 static IEnumerable CountWithTimeLimit(DateTime limit) { for (int i = 1; i <= 100; i++) { if (DateTime.Now >= limit) { yield break; } yield return i; } } ... DateTime stop = DateTime.Now.AddSeconds(2); foreach (int i in CountWithTimeLimit(stop)) { Console.WriteLine("Received {0}", i); Thread.Sleep(300); } Typically when you run listing 6.6, you’ll see about seven lines of output. The foreach loop terminates perfectly normally—as far as it’s concerned, the iterator has just run out of elements to iterate over. The yield break statement behaves much like a return statement in a normal method. So far, so simple. There’s one last aspect of execution flow to explore: how and when finally blocks are executed. EXECUTION OF FINALLY BLOCKS You’re used to finally blocks executing whenever you leave the relevant scope. Itera- tor blocks don’t behave quite like normal methods, though. As you’ve seen, a yield return statement effectively pauses the method rather than exiting it. Following that logic, you wouldn’t expect any finally blocks to be executed at that point, and they aren’t. But appropriate finally blocks are executed when a yield break statement is hit, just as you’d expect them to be when returning from a normal method.4 The most common use of finally in an iterator block is to dispose of resources, typically with a convenient using statement. You’ll see a real-world example of this in section 6.3.2, but for now we’re just trying to see how and when finally blocks are Listing 6.6 Demonstration of yield break 3 Note that methods taking ref or out parameters can’t be implemented with iterator blocks. 4 finally blocks also work as expected when execution leaves the relevant scope without reaching either a yield return or a yield break statement. I’m only focusing on the behavior of the two yield statements here because that’s where the flow of execution is new and different. Stops if the time is up 169C# 2: Simple iterators with yield statements executed. The following listing shows this in action—it’s the same code as in listing 6.6, but with a finally block. The changes are shown in bold. static IEnumerable CountWithTimeLimit(DateTime limit) { try { for (int i = 1; i <= 100; i++) { if (DateTime.Now >= limit) { yield break; } yield return i; } } finally { Console.WriteLine("Stopping!"); } } ... DateTime stop = DateTime.Now.AddSeconds(2); foreach (int i in CountWithTimeLimit(stop)) { Console.WriteLine("Received {0}", i); Thread.Sleep(300); } The finally block in listing 6.7 is executed whether the iterator block finishes by counting to 100 or due to the time limit being reached. (It would also execute if the code threw an exception.) But there are other ways you might try to avoid the finally block from being called…Let’s try to be sneaky. You’ve seen that code in the iterator block is only executed when MoveNext is called. So what happens if you never call MoveNext? Or if you call it a few times and then stop? Let’s consider changing the calling part of listing 6.7 to this: DateTime stop = DateTime.Now.AddSeconds(2); foreach (int i in CountWithTimeLimit(stop)) { Console.WriteLine ("Received {0}", i); if (i > 3) { Console.WriteLine("Returning"); return; } Thread.Sleep(300); } Here you’re not stopping early in the iterator code—you’re stopping early in the code using the iterator. The output is perhaps surprising: Listing 6.7 Demonstration of yield break working with try/finally Executes however the loop ends 170 CHAPTER 6 Implementing iterators the easy way Received 1 Received 2 Received 3 Received 4 Returning Stopping! You can see that code is being executed after the return statement in the foreach loop. That doesn’t normally happen unless a finally block is involved—and in this case there are two! You already know about the finally block in the iterator method, but the question is what’s causing it to be executed. I gave a hint about this earlier—foreach calls Dispose on the iterator returned by GetEnumerator in its own finally block (just like the using statement). When you call Dispose on an iterator created with an iterator block before it’s finished iterating, the state machine executes any finally blocks that are in the scope where the code is currently “paused.” That’s a complicated and somewhat detailed explanation, but the result is simpler to express: as long as callers use a foreach loop, finally works within iterator blocks in the way you want it to. You can easily prove that it’s the call to Dispose that triggers this by using the itera- tor manually: DateTime stop = DateTime.Now.AddSeconds(2); IEnumerable iterable = CountWithTimeLimit(stop); IEnumerator iterator = iterable.GetEnumerator(); iterator.MoveNext(); Console.WriteLine("Received {0}", iterator.Current); iterator.MoveNext(); Console.WriteLine("Received {0}", iterator.Current); This time the stopping line is never printed. If you explicitly add a call to Dispose, you’ll see the extra line in the output again. It’s relatively rare that you’ll want to ter- minate an iterator before it’s finished, and it’s relatively rare that you’ll be iterating manually instead of using foreach, but if you do, remember to wrap the iterator in a using statement. We’ve now covered most of the behavior of iterator blocks, but before we end this section, it’s worth considering a few oddities to do with the current Microsoft implementation. 6.2.4 Quirks in the implementation If you compile iterator blocks with the Microsoft C# 2 compiler and look at the result- ing IL in either ildasm or Reflector, you’ll see the nested type that the compiler has generated for you behind the scenes. In my case, when compiling listing 6.4, it was called IterationSample.d__0 (where the angle brackets make it an unspeakable name—nothing to do with generics). I won’t go through exactly what’s generated in detail here, but it’s worth looking at it in Reflector to get a feel for what’s going on, preferably with the language specification next to you, open at section 10.14 171C# 2: Simple iterators with yield statements (“Iterators”); the specification defines different states the type can be in, and this description makes the generated code easier to follow. The bulk of the work is per- formed in MoveNext, which is generally a big switch statement. Fortunately, as developers we don’t need to care much about the hoops the com- piler has to jump through. But there are a few quirks about the implementation that are worth knowing about:  Before MoveNext is called the first time, the Current property will always return the default value for the yield type of the iterator.  After MoveNext has returned false, the Current property will always return the final value yielded.  Reset always throws an exception instead of resetting like your manual imple- mentation did. This is required behavior, laid down in the specification.  The nested class always implements both the generic and nongeneric forms of IEnumerator (and the generic and nongeneric IEnumerable, where appropriate). Failing to implement Reset is reasonable—the compiler can’t work out what you’d need to do in order to reset the iterator, or even whether it’s feasible. Arguably Reset shouldn’t have been in the IEnumerator interface to start with, and I can’t remember the last time I called it. Many collections don’t support it, so callers can’t generally rely on it anyway. Implementing extra interfaces does no harm either. It’s interesting that if your method returns IEnumerable, you end up with one class implementing five interfaces (including IDisposable). The language specification explains it in detail, but the upshot is that as a developer you don’t need to worry. The fact that it implements both IEnumerable and IEnumerator is slightly unusual—the compiler goes to some pains to make sure that the behavior is correct whatever you do with it, but it also manages to create a single instance of the nested type in the common case where you just iter- ate through the collection in the same thread that created it. The behavior of Current is odd—in particular, keeping hold of the last item after supposedly moving off it could keep it from being garbage collected. It’s possible that this may be fixed in a later release of the C# compiler, but it’s unlikely, as it could break existing code (the Microsoft C# compilers shipping with .NET 3.5, 4, and 4.5 behave in the same way). Strictly speaking, it’s correct from the point of view of the C# 2 language specification—the behavior of the Current property is undefined in this situation. It’d be nicer, though, if it implemented the property in the way that the framework documentation suggests, throwing exceptions at appropriate times. Those are a few tiny drawbacks of using the autogenerated code, but sensible callers won’t have any problems—and let’s face it, you’ve saved a lot of code in order to come up with the implementation. This means it makes sense to use iterators more widely than you might’ve done in C# 1. The next section provides some sample code so you can check your understanding of iterator blocks and see how they’re useful in real life rather than in theoretical scenarios. 172 CHAPTER 6 Implementing iterators the easy way 6.3 Real-life iterator examples Have you ever written some code that’s really simple in itself but that makes your proj- ect much neater? It happens to me every so often, and it usually makes me happier than it probably ought to—enough to get strange looks from colleagues, anyway. That sort of childlike delight is particularly strong when it comes to using a new language feature in a way that’s clearly nicer and you’re not just doing it for the sake of playing with new toys. Even now, after using iterators for a few years, I still come across situations where a solution using iterator blocks presents itself, and the resulting code is brief, clean, and easy to understand. In this section I’ll share three such scenarios with you. 6.3.1 Iterating over the dates in a timetable While working on a project involving timetables, I came across a few loops, all of which started like this: for (DateTime day = timetable.StartDate; day <= timetable.EndDate; day = day.AddDays(1)) I was working on this area of code a lot, and I always hated that loop, but it was only when I was reading the code out loud to another developer as pseudocode that I real- ized I was missing a trick. I said something like, “For each day within the timetable…” In retrospect, it’s obvious that what I really wanted was a foreach loop. (This may have been obvious to you from the start—apologies if this is the case. Fortunately I can’t see you looking smug.) The loop is much nicer when rewritten as follows: foreach (DateTime day in timetable.DateRange) In C# 1, I might’ve looked at that as a fond dream but not bothered implementing it; you’ve seen how messy it is to implement an iterator by hand, and the end result only made a few for loops neater in this case. In C# 2, though, it was easy. Within the class representing the timetable, I simply added a property: public IEnumerable DateRange { get { for (DateTime day = StartDate; day <= EndDate; day = day.AddDays(1)) { yield return day; } } } This moved the original loop into the timetable class, but that’s okay—it’s much nicer for it to be encapsulated there, in a property that just loops through the days, yielding them one at a time, than to be in business code that was dealing with those days. If I 173Real-life iterator examples ever wanted to make it more complex (skipping weekends and public holidays, for instance), I could do it in one place and reap the rewards everywhere. This one small change made a massive improvement to the readability of the code base. As it happens, I stopped refactoring at that point in the commercial code. I thought about introducing a Range type to represent a general-purpose range, but as I only needed it in this one situation, it didn’t make sense to expend any more effort on the problem. It turns out that was a wise move. In the first edition of this book, I created just such a type—but it had some shortcomings that were hard to address in a book-friendly manner. I redesigned it significantly for my utility library, but I still have a few misgivings. Types like this often sound simpler than they really are, and soon you end up with a corner case to be handled at every turn. The details of the difficulties I encountered don’t really belong in this book—they’re more points about general design than C#—but they’re interesting in their own right, so I’ve writ- ten them up as an article on the book’s website (see http://mng.bz/GAmS). The next example is one of my favorites—it demonstrates everything I love about iterator blocks. 6.3.2 Iterating over lines in a file How often have you read a text file line by line? It’s an incredibly common task. As of .NET 4, the framework finally provides a method to make this easier in File.Read- Lines, but if you’re using an earlier version of the framework, you can write your own version really easily, as I’ll show in the next couple of pages. I dread to think how often I’ve written code like this: using (TextReader reader = File.OpenText(filename)) { string line; while ((line = reader.ReadLine()) != null) { // Do something with line } } We have four separate concepts here:  How to obtain a TextReader  Managing the lifetime of the TextReader  Iterating over the lines returned by TextReader.ReadLine  Doing something with each of those lines Only the first and last of these are generally specific to the situation—the lifetime management and the mechanism for iterating are just boilerplate code. (At least, the lifetime management is simple in C#. Thank goodness for using statements!) There are two ways you could improve things. You could use a delegate—write a utility method that would take a reader and a delegate as parameters, call the delegate for each line in the file, and close the reader at the end. That’s often used as an example 174 CHAPTER 6 Implementing iterators the easy way of closures and delegates, but there’s an alternative that I find more elegant and that fits in much better with LINQ. Instead of passing your logic into a method as a dele- gate, you can use an iterator to return a single line at a time from the file, so you can use the normal foreach loop. You can achieve this using a whole type implementing IEnumerable (I have a LineReader class in my MiscUtil library for this purpose), but a standalone method in another class will work fine, too. It’s really simple, as the next listing proves. static IEnumerable ReadLines(string filename) { using (TextReader reader = File.OpenText(filename)) { string line; while ((line = reader.ReadLine()) != null) { yield return line; } } } ... foreach (string line in ReadLines("test.txt")) { Console.WriteLine(line); } The body of the method is pretty much exactly what you had before, except that what you’re doing with the line is yielding it to the caller when it iterates over the collec- tion. As before, you open the file, read a line at a time, and then close the reader when you’ve finished…although the concept of “when you’ve finished” is more inter- esting in this case than with a using statement in a normal method, where the flow control is more obvious. This is why it’s important that the foreach loop dispose of the iterator—because that’s what makes sure the reader gets cleaned up. The using statement in the iterator method is acting as a try/finally block; that finally block will execute if either you get to the end of the file or you call Dispose on the IEnumerator when you’re part of the way through. It’d be possible for calling code to abuse the IEnumerator returned by ReadLines(...).GetEnumerator() and end up with a resource leak, but that’s usually the case with IDisposable—if you don’t call Dispose, you may leak resources. It’s rarely a problem though, as foreach does the right thing. It’s important to be aware of this potential abuse—if you relied on some sort of try/ finally block in an iterator to grant some permission and then remove it again later, that really would be a security hole. This method encapsulates the first three of the four concepts I listed earlier, but it’s a bit restrictive. It’s reasonable to lump together the lifetime management and iter- ation aspects, but what if you want to read text from a network stream instead? Or if Listing 6.8 Looping over the lines in a file using an iterator block 175Real-life iterator examples you want to use an encoding other than UTF-8? You need to put the first part back in the control of the caller, and the most obvious approach would be to change the method signature to accept a TextReader, like this: static IEnumerable ReadLines(TextReader reader) This is a bad idea, though. You want to take ownership of the reader so that you can clean it up conveniently for the caller, but the fact that you take responsibility for the cleanup means you have to clean it up, as long as the caller uses the result sensibly. The problem is, if something happens before the first call to MoveNext(), you’re never going to have any chance to clean up: none of your code will run. The IEnumerable itself isn’t disposable, and yet it would’ve stored this piece of state, which required disposal. Another problem would occur if GetEnumerator() were called twice: that ought to generate two independent iterators, but they’d both be using the same reader. You could mitigate this somewhat by changing the return type to IEnumerator, but that would mean the result couldn’t be used in a foreach loop, and you still wouldn’t get to run any cleanup code if you never got as far as the first MoveNext() call. Fortunately, there’s a way around this. Just as the code doesn’t get to run immediately, you don’t need the reader immedi- ately. What you need is a way of getting the reader when you need it. You could use an interface to represent the idea of “I can provide a TextReader when you want one,” but the idea of a single method interface should usually make you reach for a delegate. Instead, I’m going to cheat slightly by introducing a delegate that’s part of .NET 3.5. It’s overloaded by different numbers of type parameters, but you only need one: public delegate TResult Func() As you can see, this delegate has no parameters, but it returns a result of the same type as the type parameter. It’s a classic provider or factory signature. In this case, you want to get a TextReader, so you can use Func. The changes to the method are simple: static IEnumerable ReadLines(Func provider) { using (TextReader reader = provider()) { string line; while ((line = reader.ReadLine()) != null) { yield return line; } } } Now the resource is acquired just before you need it, and by that point you’re in the context of IDisposable, so you can release the resource at the appropriate time. Fur- thermore, if GetEnumerator() is called multiple times on the returned value, each call will result in an independent TextReader being created. 176 CHAPTER 6 Implementing iterators the easy way You can easily use anonymous methods to add overloads to open files, optionally specifying an encoding: static IEnumerable ReadLines(string filename) { return ReadLines(filename, Encoding.UTF8); } static IEnumerable ReadLines(string filename, Encoding encoding) { return ReadLines(delegate { return File.OpenText(filename, encoding); }); } This example uses generics, an anonymous method (which captures parameters), and an iterator block. All that’s missing is nullable value types and you’d have the full house of C# 2’s major features. I’ve used this code on a number of occasions, and it’s always much cleaner than the cumbersome code we started off with. As I mentioned earlier, if you’re using a recent version of .NET, you’ve got all this available in File.ReadLines anyway, but it’s still a neat example of just how useful iterator blocks can be. As a final example, let’s get a first taste of LINQ—even though we’ll only use C# 2. 6.3.3 Filtering items lazily using an iterator block and a predicate Even though we haven’t started to look at LINQ properly yet, I’m sure you have some idea of what it’s about: it allows you to query data in a simple and powerful way across multiple data sources, such as in-memory collections and databases. C# 2 doesn’t have any of the language integration for query expressions, nor the lambda expressions and extension methods that can make it so concise, but you can still achieve some of the same effects. One of the core features of LINQ is filtering with the Where method. You provide a collection and a predicate, and the result is a lazily evaluated query that’ll yield only the items in the collection that match the predicate. This is a little like List.FindAll, but it’s lazy and works with any IEnumerable. One of the beautiful things about LINQ5 is that the cleverness is in the design. It’s quite simple to implement LINQ to Objects as we’ll prove now, at least for the Where method. Ironically, even though most of the language features that make LINQ shine are part of C# 3, these are almost all about how you can access methods such as Where, rather than how they’re implemented. The following listing shows a full example, including simple argument validation, and uses the filter to display all the using directives in the source file that contains the sample code itself. 5 Or to be more precise, LINQ to Objects. LINQ providers for databases and the like are far more complex. 177Real-life iterator examples public static IEnumerable Where(IEnumerable source, Predicate predicate) { if (source == null || predicate == null) { throw new ArgumentNullException(); } return WhereImpl(source, predicate); } private static IEnumerable WhereImpl(IEnumerable source, Predicate predicate) { foreach (T item in source) { if (predicate(item)) { yield return item; } } } ... IEnumerable lines = LineReader.ReadLines("../../FakeLinq.cs"); Predicate predicate = delegate(string line) { return line.StartsWith("using"); }; foreach (string line in Where(lines, predicate)) { Console.WriteLine(line); } This example splits the implementation into two parts: argument validation and the real business logic of filtering. It’s slightly ugly but entirely necessary for sensible error handling. Suppose you put everything in the same method—what would happen when you called Where(null, null)? The answer is nothing…or, at least, the desired exception wouldn’t be thrown. This is due to the lazy semantics of iterator blocks: none of the code in the body of the method runs until the first call to Move- Next(), as you saw in section 6.2.2. Typically you want to check the preconditions to the method eagerly—there’s no point in delaying the exception, and it just makes debugging harder. The standard workaround for this is to split the method in half, as in listing 6.9. First you check the arguments B in a normal method, and then you call the method implemented using an iterator block to lazily process the data as and when it’s requested C. The iterator block itself is mind-numbingly straightforward: for each item in the original collection, you test the predicate D and yield the value if it matches. If it doesn’t match, you try the next item, and so on until you find something that does match, or you run out of items. It’s straightforward, but a C# 1 implementation would’ve been much harder to follow (and couldn’t have been generic, of course). Listing 6.9 Implementing LINQ’s Where method using iterator blocks Eagerly checks argumentsB Lazily processes dataC Tests current item against predicateD 178 CHAPTER 6 Implementing iterators the easy way The final piece of code to demonstrate the method in action uses the previous example to provide the data—in this case, the source code of the implementation. The predicate simply tests a line to see whether it begins with “using”—it could con- tain far more complicated logic, of course. I’ve created separate variables for the data and the predicate just to make the formatting clearer, but it could all have been writ- ten inline. It’s important to note the principal difference between this example and the equivalent that could’ve been achieved using File.ReadAllLines and Array .FindAll. This implementation is entirely lazy and streaming. Only a single line from the source file is ever required in memory at a time. Of course, that wouldn’t matter in this particular case where the file is small—but if you imagine a multigigabyte log file, you can see the benefits of this approach. I hope these examples have given you an inkling of why iterator blocks are so important—as well as perhaps a desire to hurry on and find out more about LINQ. Before that, I’d like to mess with your mind a bit and introduce you to a thoroughly bizarre (but really neat) use of iterators. 6.4 Pseudo-synchronous code with the Concurrency and Coordination Runtime The Concurrency and Coordination Runtime (CCR) is a library developed by Microsoft to offer an alternative way of writing asynchronous code that’s amenable to complex coordination. At the time of this writing, it’s only available as part of the Microsoft Robotics Studio (see http://www.microsoft.com/robotics). Microsoft has been put- ting a lot of resources into concurrency in various projects, most notably the Task Par- allel Library introduced in .NET 4, and the asynchronous language features in C# 5 (supported by a lot of asynchronous APIs). But I wanted to use the CCR to show you how iterator blocks can change the whole execution model. Indeed, it’s no coinci- dence that this early foray into an alternative approach to concurrency uses iterator blocks to change the execution model; the similarities between the state machines generated for iterator blocks and those used for asynchronous functions in C# 5 are striking. The sample code does work (against dummy services) but the ideas are more important than the details. Suppose you’re writing a server that needs to handle lots of requests. As part of dealing with those requests, you need to first call a web service to fetch an authentica- tion token, and then use that token to get data from two independent data sources (say, a database and another web service). You then process that data and return the result. Each of the fetch stages could take a while—perhaps a few seconds. Normally you might consider the simple synchronous route or the stock asynchronous approach. The synchronous version might look something like this: HoldingsValue ComputeTotalStockValue(string user, string password) { Token token = AuthService.Check(user, password); Holdings stocks = DbService.GetStockHoldings(token); 179Pseudo-synchronous code with the Concurrency and Coordination Runtime StockRates rates = StockService.GetRates(token); return ProcessStocks(stocks, rates); } That’s easy to understand, but if each request takes 2 seconds, the whole operation will take 6 seconds and tie up a thread for the whole time it’s running. If you want to scale up to hundreds of thousands of requests running in parallel, you’re in trouble. Now let’s consider a fairly simple asynchronous version, which avoids tying up a thread when nothing’s happening6 and uses parallel calls where possible: void StartComputingTotalStockValue(string user, string password) { AuthService.BeginCheck(user, password, AfterAuthCheck, null); } void AfterAuthCheck(IAsyncResult result) { Token token = AuthService.EndCheck(result); IAsyncResult holdingsAsync = DbService.BeginGetStockHoldings (token, null, null); StockService.BeginGetRates(token, AfterGetRates, holdingsAsync); } void AfterGetRates(IAsyncResult result) { IAsyncResult holdingsAsync = (IAsyncResult)result.AsyncState; StockRates rates = StockService.EndGetRates(result); Holdings stocks = DbService.EndGetStockHoldings(holdingsAsync); OnRequestComplete(ProcessStocks(stocks, rates)); } This is much harder to read and understand—and that’s only a simple version. The coordination of the two parallel calls is only achievable in a simple way because you don’t need to pass any other state around, and even so it’s not ideal. If the stock ser- vice call completes quickly, you’ll still block a thread-pool thread waiting for the data- base call to complete. More importantly, it’s far from obvious what’s going on, because the code jumps between different methods. By now you may be asking yourself where iterators come into the picture. Well, the iterator blocks provided by C# 2 effectively allow you to pause current execution at certain points of the flow through the block and then come back to the same place, with the same state. The clever folks designing the CCR realized that that’s exactly what’s needed for a continuation-passing style of coding. You need to tell the system that there are certain operations you need to perform—including starting other opera- tions asynchronously—but that you’re then happy to wait until the asynchronous operations have finished before you continue. You do this by providing the CCR with an implementation of IEnumerator (where ITask is an interface defined by the CCR). Here’s some code to achieve the same results using this style: 6 Well, mostly. It might still be inefficient, as you’ll see in a moment. 180 CHAPTER 6 Implementing iterators the easy way static IEnumerator ComputeTotalStockVal.(str.user,str.pass) { string token = null; yield return Arbiter.Receive(false, AuthService.CcrCheck(user, pass), delegate(string t) { token = t; }); IEnumerable stocks = null; IDictionary rates = null; yield return Arbiter.JoinedReceive(false, DbService.CcrGetStockHoldings(token), StockService.CcrGetRates(token), delegate(IEnumerable s, IDictionary r) { stocks = s; rates = r; }); OnRequestComplete(ComputeTotal(stocks, rates)); } Confused? I certainly was when I first saw it, but now I’m in awe of how neat it is. The CCR calls into your code (with a call to MoveNext on the iterator), and you execute until and including the first yield return statement. The CcrCheck method within AuthService kicks off an asynchronous request, and the CCR waits (without using a dedicated thread) until it has completed, calling the supplied delegate to handle the result. It then calls MoveNext again, and your method continues. This time you kick off two requests in parallel and ask the CCR to call another delegate with the results of both operations when they’ve both finished. After that, MoveNext is called for a final time, and you get to complete the request processing. Although it’s obviously more complicated than the synchronous version, it’s still all in one method, it gets executed in the order written, and the method itself can hold the state (in the local variables, which become state in the extra type generated by the compiler). It’s fully asynchronous, using as few threads as it can get away with. I haven’t shown any error handling, but that’s also available in a sensible fashion that forces you to think about the issue at appropriate places. I’ve deliberately not gone into the details of the Arbiter class, the ITask interface, and so forth here. I’m not trying to promote the CCR in this section, although it’s fas- cinating to read about and experiment with; I suspect that asynchronous functions in C# 5 will have much more impact on mainstream developers. My point here has been to show that iterators can be used in radically different contexts that have little to do with traditional collections. At the heart of this use of them is the idea of a state machine: two of the tricky aspects of asynchronous development are handling state and effectively pausing until something interesting happens. Iterator blocks are a nat- ural fit for both of these problems, although you’ll see in chapter 15 how more tar- geted language support makes things much cleaner. 6.5 Summary C# supports many patterns indirectly, in terms of it being feasible to implement them in C#. But relatively few patterns are directly supported in terms of language features being specifically targeted at a particular pattern. In C# 1, the iterator pattern was 181Summary directly supported from the point of view of the calling code, but not from the per- spective of the collection being iterated over. Writing a correct implementation of IEnumerable was time consuming and error-prone, without being interesting. In C# 2, the compiler does all the mundane work for you, building a state machine to cope with the callback nature of iterators. It should be noted that iterator blocks have one aspect in common with the anony- mous methods you saw in chapter 5, even though the actual features are very differ- ent. In both cases, extra types may be generated, and a potentially complicated code transformation is applied to the original source. Compare this with C# 1, where most of the transformations for syntactic sugar (lock, using, and foreach being the most obvious examples) were straightforward. You’ll see this trend toward smarter compila- tion continuing with almost every aspect of C# 3. I’ve shown you one piece of LINQ-related functionality in this chapter: filtering a collection. IEnumerable is one of the most important types in LINQ, and if you ever want to write your own LINQ operators on top of LINQ to Objects,7 you’ll be eter- nally grateful to the C# team for including iterator blocks in the language. In addition to seeing some real-life examples of the use of iterators, we’ve looked at how one particular library has used them in a fairly radical way that has little to do with what likely comes to mind when you think about iteration over a collection. It’s worth bearing in mind that many languages have also looked at this sort of problem before—in computer science, the term coroutine is applied to concepts of this nature, and that’s how they’re referred to in the Unity 3D game development toolset, where again they’re used for asynchrony. Different languages have historically supported them to a greater or lesser extent, with tricks being applicable to simulate them some- times. For example, Simon Tatham has an excellent article on how even C can express coroutines if you’re willing to bend coding standards somewhat (see his “Coroutines in C” article at http://mng.bz/H8YX). You’ve seen that C# 2 makes coroutines easy to write and use. Now that you’ve seen some major and sometimes mind-warping language changes focused around a few key features, the next chapter will be a change of pace. It describes a number of small changes that make C# 2 more pleasant to work with than its predecessor. The designers learned from the little niggles of the past and produced a language that has fewer rough edges, more scope for dealing with awkward backward-compatibility cases, and a better story around working with generated code. Each feature is relatively straightforward, but there are quite a few of them. 7 This is less daunting and more fun than it sounds. We’ll look at a few guidelines around this topic in chap- ter 12. 182 Concluding C# 2: the final features So far we’ve looked at the four biggest new features in C# 2: generics, nullable types, delegate enhancements, and iterator blocks. Each addresses a fairly complex requirement, which is why we’ve gone into them in some depth. The remaining new features of C# 2 knock a few rough edges off C# 1. They’re little niggles that the language designers decided to correct—either areas where the language needed a bit of improvement for its own sake, or where the experience of working with code generation and native code could be made more pleasant. This chapter covers  Partial types  Static classes  Separate getter/setter property access  Namespace aliases  Pragma directives  Fixed-size buffers  Friend assemblies 183Partial types Over time, Microsoft has received a lot of feedback from the C# community (and its own developers, no doubt) about areas where C# hasn’t gleamed quite as brightly as it might. Several smaller changes made it into C# 2 along with the larger ones, alle- viating some of these small pain points. None of the features in this chapter is particularly difficult, and we’ll go through them fairly quickly. Don’t underestimate how important they are, though. Just because a topic can be explored in a few pages doesn’t mean it’s useless. You’re likely to use some of these features on a frequent basis. Here’s a quick rundown of the fea- tures covered in this chapter and their uses, so you know what to expect:  Partial types—The ability to write the code for a type in multiple source files. This is particularly handy for types where part of the code is autogenerated and the rest is written manually.  Static classes—For tidying up utility classes so that the compiler can spot when you’re trying to use them inappropriately, and making your intentions clearer.  Separate getter/setter property access—Finally, the ability to have a public getter and a private setter for properties! (That’s not the only combination available, but it’s the most common.)  Namespace aliases—Ways out of sticky situations where type names aren’t unique.  Pragma directives—Compiler-specific instructions for actions such as suppressing specific warnings for a particular section of code.  Fixed-size buffers—More control over how structs handle arrays in unsafe code.  InternalsVisibleToAttribute (friend assemblies)—A feature spanning language, framework, and runtime, this allows selected assemblies more access when required. You may be itching to get on to the sexy stuff from C# 3 by this point, and I don’t blame you. Nothing in this chapter is going to set the world on fire—but each of these features can make your life more pleasant, or dig you out of a hole in some cases. Hav- ing dampened your expectations somewhat, the first feature is actually pretty nifty. 7.1 Partial types The first change we’ll look at is in response to the power struggle that was usually involved when using code generators with C# 1. For Windows Forms, the designer in Visual Studio required its own regions of code that couldn’t be touched by developers, within the same file that developers had to edit for user interface functionality. This was clearly a brittle situation. In other cases, code generators create source that’s compiled alongside manually written code. In C# 1, adding extra functionality involved deriving new classes from the autogenerated ones, which is ugly. There are plenty of other scenarios where hav- ing an unnecessary link in the inheritance chain can cause problems or reduce encap- sulation. For instance, if two different parts of your code want to call each other, you 184 CHAPTER 7 Concluding C# 2: the final features need virtual methods for the parent type to call the child, and protected methods for the reverse situation, where normally you’d use two private nonvirtual methods. C# 2 allows more than one file to contribute to a type, and IDEs can extend this notion so that some of the code used for a type may not even be visible as C# source code at all. Types built from multiple source files are called partial types. In this section we’ll also discuss partial methods, which are only relevant in partial types and allow a rich but efficient way of adding manually written hooks into auto- generated code. This is actually a C# 3 feature (this time based on feedback about C# 2), but it’s more logical to discuss it when we examine partial types than to wait until the next part of the book. 7.1.1 Creating a type with multiple files Creating a partial type is a cinch—you just need to include the partial contextual keyword in the declaration for the type in each file it occurs in. A partial type can be declared within as many files as you like, although all the examples in this section use two. The compiler effectively combines all the source files together before compiling. This means that code in one file can call code in another and vice versa, as shown in figure 7.1—there’s no need for forward references or other tricks. You can’t write half of a member in one file and half of it in another—each individ- ual member has to be completely contained within its own file. You can’t start a method in one file and finish it in another, for example.1 There are a few obvious restrictions about the declarations of the type—the declarations have to be compatible. Any file can specify interfaces to be implemented (and they don’t have to be implemented in that file), any file can specify the base type, and any file can specify constraints on a type parameter. But if multiple files specify a base type, those base types have to be the same, and if multiple files specify type constraints, the constraints have to be identical. The 1 There’s an exception here: partial types can contain nested partial types spread across the same set of files. partial class Example { void FirstMethod() { SecondMethod(); } void ThirdMethod() { } } partial class Example { void SecondMethod() { ThirdMethod(); } } Example2.csExample1.cs Figure 7.1 Code in partial types is able to see all of the members of the type, regardless of which file each member is in. 185Partial types following listing shows an example of the flexibility afforded (while not doing anything even remotely useful). // Example1.cs using System; partial class Example : IEquatable where TFirst : class { public bool Equals(string other) { return false; } } // Example2.cs using System; partial class Example : EventArgs, IDisposable { public void Dispose() { } } I stress that this listing is solely for the purpose of talking about what’s legal in a decla- ration—the types involved were only picked for convenience and familiarity. You can see that both declarations (B and D) contribute to the list of interfaces that must be implemented. In this example, each file implements the interfaces it declares, and that’s a common scenario, but it would be legal to move the implementation of IDisposable E to Example1.cs and the implementation of IEquatable C to Example2.cs. I’ve used the ability to specify interfaces separately from the imple- mentation myself, encapsulating methods with the same signature generated for mul- tiple different types into an interface. The code generator doesn’t know about the interface, so it doesn’t know to declare that the type implements it. Only the first declaration B specifies any type constraints, and only the second D specifies a base class. If the first declaration B had specified a base class, it would have to be EventArgs, and if the second declaration had specified any type constraints, they’d have to be exactly as in the first. In particular, you can’t specify a type constraint for TSecond in the second declaration, even though it’s not mentioned in the first. Both types have to have the same access modifier, if any—you can’t make one declara- tion internal and the other public, for example. Essentially, the rules around com- bining files allow flexibility in most cases while encouraging consistency. In single file types, the initialization of member and static variables is guaranteed to occur in the order they appear in the file, but there’s no guaranteed order when multiple files are involved. Relying on the order of declaration within the file is brittle Listing 7.1 Demonstration of mixing declarations of a partial type Specifies interface and type parameter constraint B Implements IEquatable C Specifies base class and interface D Implements IDisposableE 186 CHAPTER 7 Concluding C# 2: the final features to start with—it leaves your code open to subtle bugs if a developer decides to “harm- lessly” move things around—so it’s worth avoiding this situation where you can. You should particularly avoid it with partial types. Now that you know what you can and can’t do, let’s take a closer look at why you’d want to do it. 7.1.2 Uses of partial types As I mentioned earlier, partial types are primarily useful in conjunction with designers and other code generators. If a code generator has to modify a file that’s owned by a developer, there’s always a risk of things going wrong. With the partial types model, a code generator can own the file where it’ll work and completely overwrite the whole file every time it wants to. Some code generators may choose not to generate a C# file at all until the build is well under way. For instance, Snippy has Extensible Application Markup Language (XAML) files that describe the user interface. When the project is built, each XAML file is converted into a C# file in the obj directory (the filenames end with .g.cs to show they’ve been generated) and compiled along with the partial class providing extra code for that type (typically event handlers and extra construction code). This com- pletely prevents developers from tweaking the generated code, at least without going to the extreme lengths of hacking the build file. I’ve been careful to use the phrase code generator instead of designer because there are plenty of code generators around other than designers. For instance, in Visual Stu- dio, web service proxies are generated as partial classes, and you may have your own tools that generate code based on other data sources. One reasonably common exam- ple of this is object-relational mapping (ORM)—some ORM tools use database entity descriptions from a configuration file (or straight from the database) and generate partial classes representing those entities. Likewise my .NET port of the Google Proto- col Buffers serialization framework generates partial classes—a feature that has proven useful even within the implementation itself. This makes it straightforward to add behavior to the type—overriding virtual meth- ods of the base class, adding new members with business logic, and so forth. It’s a great way of letting the developer and the tool work together, rather than constantly squabbling about who’s in charge. One scenario that’s occasionally useful is for one file to be generated containing multiple partial types, and then some of those types are enhanced in other files, with one manually generated file per type. To return to the ORM example, the tool could generate a single file containing all the entity definitions, and some of those entities could have extra code provided by the developer, using one file per entity. This keeps the number of automatically generated files low, but still provides good visibility of the manual code involved. 187Partial types Figure 7.2 shows how the uses of partial types for XAML and entities are similar, but with slightly different timing involved when it comes to creating the autogenerated C# code. A somewhat different use of partial types is as an aid to refactoring. Sometimes a type gets too big and assumes too many responsibilities. One first step to dividing the bloated type into smaller, more coherent types can be to split it into a partial type over two or more files. This can be done with no risk and in an experimental manner, mov- ing methods between files until each file addresses a single concern. Although the next step of splitting the type up is still far from automatic, it should be a lot easier to see the end goal. One final use to mention: unit testing. Often the set of unit tests for a class can end up being much larger than the implementation itself. One way to split the tests into more understandable chunks is to use partial types. You can still easily run all the tests for a type in one go (since you still have a single test class), but you can easily see the tests for different areas of functionality in different files. By hand-editing the project file, you can even have the same parent/child expansion in Solution Explorer as you see when partial types are used for Visual Studio’s generated code. This won’t be to everyone’s taste, but I’ve found it to be a useful way of managing tests. When partial types first appeared in C# 2, no one knew exactly how they’d be used. One feature that was almost immediately requested was a way to provide optional extra code for generated methods to call. This need has been addressed by C# 3 with partial methods. GuiPage.xaml.cs (Handwritten C#) GuiPage.xaml (XAML) GuiPage.g.cs (C#) GuiPage type (Part of an assembly) XAML to C# converter (Build time) Customer.cs (Handwritten C#) Schema/model (Database, XML, etc.) GeneratedEntities.cs (C# — includes partial Customer class) Customer type (Part of an assembly) Code generator (Prebuild) Using XAML for declarative UI design Prebuilding partial classes for database entities C# compilation C# compilation Figure 7.2 Comparison between XAML precompilation and autogenerated entity classes 188 CHAPTER 7 Concluding C# 2: the final features 7.1.3 Partial methods—C# 3 only! To reiterate my previous explanation, the rest of this part of the book just deals with C# 2 features, but partial methods don’t fit with any of the other C# 3 features and they do fit in well when describing partial types. Apologies for any confusion this may cause. Back to the feature: sometimes you want to be able to specify behavior in a manu- ally created file and use that behavior from an automatically generated file. For instance, in a class that has lots of automatically generated properties, you might want to be able to specify code to be executed to validate a new value for some of those properties. Another common scenario is for a code generator to include construc- tors—manually written code may want to hook into object construction to set default values, perform some logging, and so forth. In C# 2, these requirements can only be met either by using events that the manu- ally generated code can subscribe to, or by making the automatically generated code assume that the handwritten code will include methods of a particular name—making all the code fail to compile unless the relevant methods are provided. Alternatively, the generated code can provide a base class with virtual methods that do nothing by default. The manually generated code can then derive from the class and override some or all of the methods. All of these solutions are somewhat messy. C# 3’s partial methods effectively pro- vide optional hooks that have no cost whatsoever if they’re not implemented—any calls to the unimplemented partial methods are removed by the compiler. This allows tools to be very generous in terms of the hooks they provide. In the compiled code, you only pay for what you use. It’s easiest to understand this with an example. The following listing shows a par- tial type specified in two files, with the constructor in the automatically generated code calling two partial methods, one of which is implemented in the manually gen- erated code. // Generated.cs using System; partial class PartialMethodDemo { public PartialMethodDemo() { OnConstructorStart(); Console.WriteLine("Generated constructor"); OnConstructorEnd(); } partial void OnConstructorStart(); partial void OnConstructorEnd(); } // Handwritten.cs Listing 7.2 A partial method called from a constructor 189Partial types using System; partial class PartialMethodDemo { partial void OnConstructorEnd() { Console.WriteLine("Manual code"); } } As shown in listing 7.2, partial methods are declared just like abstract methods: by pro- viding the signature without any implementation but using the partial modifier. Sim- ilarly, the actual implementations just have the partial modifier but are otherwise like normal methods. Calling the parameterless constructor of PartialMethodDemo would result in Generated constructor and then Manual code being printed out. If you examined the IL for the constructor, you wouldn’t see a call to OnConstructorStart because it no longer exists—there’s no trace of it anywhere in the compiled type. Because the method may not exist, partial methods must have a return type of void, and they can’t take out parameters. They have to be private, but they can be static and/or generic. If the method isn’t implemented in one of the files, the whole statement calling it is removed, including any argument evaluations. If evaluating any of the arguments has a side effect that you want to occur whether or not the partial method is implemented, you should perform the evaluation sepa- rately. For instance, suppose you have the following code: LogEntity(LoadAndCache(id)); Here LogEntity is a partial method, and LoadAndCache loads an entity from the data- base and inserts it into the cache. You might want to use this instead: MyEntity entity = LoadAndCache(id); LogEntity(entity); That way, the entity is loaded and cached regardless of whether an implementation has been provided for LogEntity. Of course, if the entity can be loaded equally cheaply later on, and may not even be required, you should leave the statement in the first form and avoid an unnecessary load in some cases. To be honest, unless you’re writing your own code generators, you’re more likely to be implementing partial methods than declaring and calling them. If you’re only implementing them, you don’t need to worry about the argument evaluation side of things. In summary, partial methods in C# 3 allow generated code to interact with hand- written code in a rich manner without any performance penalties for situations where the interaction is unnecessary. This is a natural continuation of the C# 2 partial types feature, which enables a much more productive relationship between code generators and developers. 190 CHAPTER 7 Concluding C# 2: the final features The next feature is entirely different. It’s a way of telling the compiler more about the intended nature of a type so that it can perform more checking on both the type itself and any code using it. 7.2 Static classes The second new feature is in some ways completely unnecessary—it just makes things tidier and more elegant when you write utility classes. Everyone has utility classes. I haven’t seen a significant project in either Java or C# that didn’t have at least one class consisting solely of static methods. The classic exam- ple in developer code is a type with string helper methods, doing anything from escap- ing, reversing, smart replacing—you name it. An example from the framework is the System.Math class. The key features of a utility class are as follows:  All members are static (except a private constructor).  The class derives directly from object.  Typically there’s no state at all, unless some caching or a singleton is involved.  There are no visible constructors.  The class is sealed if the developer remembers to do so. The last two points are optional, and if there are no visible constructors (including protected ones), the class is effectively sealed anyway. Both of them help make the pur- pose of the class more obvious, though. The following listing gives an example of a C# 1 utility class. Then we’ll look at how C# 2 improves matters. public sealed class NonStaticStringHelper { private NonStaticStringHelper() { } public static string Reverse(string input) { char[] chars = input.ToCharArray(); Array.Reverse(chars); return new string(chars); } } The class is sealed B so that no one tries to derive from it. Inheritance is supposed to be about specialization, and there’s nothing to specialize here, as all the members are static D except the private constructor C. That constructor may seem odd at first sight—why have it at all if it’s private and never going to be used? The reason is that if you don’t supply any constructors for a class, the C# 1 compiler will always provide a Listing 7.3 A typical C# 1 utility class Seals class to prevent derivationB Prevents instantiation from other codeC All methods are staticD 191Static classes default constructor that’s public and parameterless. In this case, you don’t want any visi- ble constructors, so you have to provide a private one. This pattern works reasonably well, but C# 2 makes it explicit and actively prevents the type from being misused. First, we’ll look at what changes are needed to turn list- ing 7.3 into a proper static class, as defined in C# 2. As you can see in the following list- ing, little action is required. using System; public static class StringHelper { public static string Reverse(string input) { char[] chars = input.ToCharArray(); Array.Reverse(chars); return new string(chars); } } You use the static modifier in the class declaration this time instead of sealed, and you don’t include a constructor at all—those are the only code differences. The C# 2 compiler knows that a static class shouldn’t have any constructors, so it doesn’t pro- vide a default one. In fact, the compiler enforces a number of constraints on the class definition:  It can’t be declared as abstract or sealed, although it’s implicitly both.  It can’t specify any implemented interfaces.  It can’t specify a base type.  It can’t include any nonstatic members, including constructors.  It can’t include any operators.  It can’t include any protected or protected internal members. It’s worth noting that although all the members must be static, you have to explicitly make them static. Although nested types are implicitly static members of the enclos- ing class, the nested type itself can be a nonstatic type if that’s required. The compiler doesn’t just put constraints on the definition of static classes—it also guards against their misuse. Because it knows that there can never be any instances of the class, it prevents any use that would require one. For instance, all of the following are invalid when StringHelper is a static class: StringHelper variable = null; StringHelper[] array = null; public void Method1(StringHelper x) {} public StringHelper Method1() { return null; } List x = new List(); Listing 7.4 The same utility class as in listing 7.3, but converted into a C# 2 static class 192 CHAPTER 7 Concluding C# 2: the final features None of these is prevented if the class follows the C# 1 pattern, but all of them are essentially useless. In short, static classes in C# 2 don’t allow you to do anything you couldn’t do before, but they prevent you from doing things that you shouldn’t have been doing anyway. They also explicitly state your intentions. By making a class static, you’re saying that you definitely don’t want any instances to be created. It’s not just a quirk of the implementation; it’s a design choice. The next feature on the list has a more positive feel. It’s aimed at a specific— although widely encountered—situation, and allows a solution that’s neither ugly nor breaks encapsulation, which was the choice available in C# 1. 7.3 Separate getter/setter property access I’ll admit to being bemused when I first saw that C# 1 didn’t allow you to have a public getter and a private setter for properties. This isn’t the only combination of access modifiers that’s prohibited by C# 1, but it’s the most commonly desired one. In fact, in C# 1 both the getter and the setter need to have the same accessibility—it’s declared as part of the property declaration rather than as part of the getter or setter. There are perfectly good reasons to want different accessibility for the getter and the setter. Often you may want some validation, logging, locking, or other code to be executed when changing a variable that backs the property, but you don’t want to make the property writable to code outside the class. In C# 1 the alternatives were either to break encapsulation by making the property publicly writable against your better judgment or to write a SetXXX() method in the class to do the setting, which frankly looks ugly when you’re used to real properties. C# 2 fixes the problem by allowing either the getter or the setter to explicitly have more restrictive access than that declared for the property itself. This is most easily seen with an example: string name; public string Name { get { return name; } private set { // Validation, logging etc here name = value; } } In this case, the Name property is effectively read-only to all other types,2 but you can use the familiar property syntax for setting the property within the type itself. The same syntax is also available for indexers as well as properties. You could make the set- ter more public than the getter (a protected getter and a public setter, for example), but that’s a pretty rare situation, in the same way that write-only properties are few and far between compared with read-only properties. 2 Except nested types, which always have access to private members of their enclosing types. 193Namespace aliases TRIVIA: THE ONLY PLACE WHERE “PRIVATE” IS REQUIRED Everywhere else in C#, the default access modifier in any given situation is the most private one possi- ble. For example, if something can be declared to be private, it will default to private if you don’t specify any access modifiers. This is a nice element of lan- guage design, because it’s hard to get it wrong accidentally; if you want some- thing to be more public than it is, you’ll notice when you try to use it. But if you accidentally make something too public, the compiler can’t help you spot the problem. Specifying the access of a property getter or setter is the one exception to this rule—if you don’t specify anything, the default is to give the getter or setter the same access as the overall property itself. Note that you can’t declare the property itself to be private and make the getter pub- lic—you can only make a particular getter or setter more private than the property. Also, you can’t specify an access modifier for both the getter and the setter—that would be silly, as you could declare the property itself to be whichever is the more public of the two modifiers. This aid to encapsulation is extremely welcome. There’s still nothing in C# 2 to stop other code in the same class from bypassing the property and going directly to whatever fields are backing it, unfortunately. As you’ll see in the next chapter, C# 3 fixes this in one particular case, but not in general. We’ll now move from a feature you may want to use regularly to one that you’ll want to avoid most of the time—it allows your code to be absolutely explicit in terms of which types it’s referring to, but at a significant cost to readability. 7.4 Namespace aliases Namespaces are primarily intended as a means of organizing types into a useful hier- archy. They also allow you to keep fully qualified names of types distinct even when the unqualified names may be the same. This shouldn’t be seen as an invitation to reuse unqualified type names without good cause, but there are times when it’s the natural thing to do. An example of this is the unqualified name Button. There are two classes with that name in the .NET 2.0 Framework: System.Windows.Forms.Button and System.Web.UI .WebControls.Button. Although they’re both called Button, it’s easy to tell them apart by their namespaces. This mirrors real life closely—you may know several people called Jon, but you’re unlikely to know anyone else called Jon Skeet. If you’re talking with friends in a particular context, you may be able to use just the name Jon without specifying which one you’re talking about, but in other contexts you may need to pro- vide more exact information. The using directive of C# 1 (not to be confused with the using statement that calls Dispose automatically) was available in two flavors: one created an alias for a namespace or type (for example, using Out = System.Console;) and the other intro- duced a namespace into the list of contexts the compiler would search when looking for a type (for example, using System.IO;). By and large, this was adequate, but there 194 CHAPTER 7 Concluding C# 2: the final features were a few situations that the language couldn’t cope with. In some other cases, auto- matically generated code would have to go out of its way to make absolutely sure that the right namespaces and types were being used whatever happened. C# 2 fixes these problems, bringing additional expressiveness to the language. You can now write code that’s guaranteed to mean what you want it to regardless of which other types, assemblies, and namespaces are introduced. These extreme measures are rarely needed outside automatically generated code, but it’s nice to know that they’re there when you need them. In C# 2 there are three types of aliases: the namespace aliases of C# 1, the global namespace alias, and extern aliases. We’ll start off with the one type of alias that was already present in C# 1, but we’ll introduce a new way of using aliases to ensure that the compiler knows to treat it as an alias rather than checking to see whether it’s the name of another namespace or type. 7.4.1 Qualifying namespace aliases Even in C# 1, it was a good idea to avoid namespace aliases wherever possible. Every so often you might find that one type name clashed with another—as with the previous Button example—so you either had to specify the full name including the namespace every time you used it, or you needed an alias that distinguished the two, in some ways acting like a shortened form of the namespace. The following listing shows an exam- ple where the two types of Button are used, qualified by an alias. using System; using WinForms = System.Windows.Forms; using WebForms = System.Web.UI.WebControls; class Test { static void Main() { Console.WriteLine(typeof(WinForms.Button)); Console.WriteLine(typeof(WebForms.Button)); } } Listing 7.5 compiles without any errors or warnings, although it’s still not as pleasant as it would be if you only needed to deal with one kind of Button to start with. There’s a problem, though—what if someone were to introduce a type or namespace called WinForms or WebForms? The compiler wouldn’t know what WinForms.Button meant and would use the type or namespace in preference to the alias. You want to be able to tell the compiler that you need it to treat WinForms as an alias, even though it’s avail- able elsewhere. C# 2 introduces the ::namespace alias qualifier syntax to do this, as shown in the fol- lowing listing. Listing 7.5 Using aliases to distinguish between different Button types 195Namespace aliases using System; using WinForms = System.Windows.Forms; using WebForms = System.Web.UI.WebControls; class WinForms {} class Test { static void Main() { Console.WriteLine(typeof(WinForms::Button)); Console.WriteLine(typeof(WebForms::Button)); } } Instead of WinForms.Button, listing 7.6 uses WinForms::Button, and the compiler is happy. If you change the :: back to . you’ll get a compilation error. So if you use :: everywhere you use an alias, you’ll be fine, right? Well, not quite... 7.4.2 The global namespace alias There’s one part of the namespace hierarchy that you can’t define your own alias for: the root of it, or the global namespace. Suppose you have two classes, both named Configuration—one within a namespace of MyCompany and the other with no namespace specified at all. How can you refer to the root Configuration class from within the MyCompany namespace? You can’t use a normal alias, and if you just specify Configuration, the compiler will use MyCompany.Configuration. In C# 1, there was no way of getting around this. Again, C# 2 comes to the rescue, allowing you to use global::Configuration to tell the compiler exactly what you want. The following listing demonstrates both the problem and the solution. using System; class Configuration {} namespace Chapter7 { class Configuration {} class Test { static void Main() { Console.WriteLine(typeof(Configuration)); Console.WriteLine(typeof(global::Configuration)); Console.WriteLine(typeof(global::Chapter7.Test)); } } } Listing 7.6 Using :: to tell the compiler to use aliases Listing 7.7 Use of the global namespace alias to specify the desired type exactly 196 CHAPTER 7 Concluding C# 2: the final features Most of listing 7.7 just sets up the situation—the three lines within Main are the inter- esting ones. The first line prints Chapter7.Configuration as the compiler resolves Configuration to that type before moving out to the namespace root. The second line indicates that the type has to be in the global namespace, so it simply prints Configuration. I included the third line to demonstrate that by using the global alias, you can still refer to types within namespaces, but you have to specify the fully quali- fied name. At this point, you can get to any uniquely named type, using the global namespace alias if necessary. If you ever write a code generator where the code doesn’t need to be readable, you might want to use this feature liberally to make sure that you always refer to the correct type regardless of any other types that are present by the time the code is compiled. But what do you do if the type’s name isn’t unique even when you include its namespace? The plot thickens... 7.4.3 Extern aliases At the start of this section, I referred to human names as examples of namespaces and contexts. I specifically said that you’re unlikely to know more than one person called Jon Skeet. But I know that there is more than one person with my name, and it’s not beyond the realm of possibility that you might know two or more of us. In this case, in order to specify which one you mean, you’d have to provide some more information beyond just the full name—the reason you know the particular person, or the country they live in, or something similarly distinctive. C# 2 lets you specify that extra information in the form of an extern alias—a name that exists not only in your source code, but also in the parameters you pass to the compiler. For the Microsoft C# compiler, this means specifying the assembly that contains the types in question. Suppose that two assemblies—First.dll and Second.dll—both contain a type called Demo.Example. You can’t just use the fully qualified name to distinguish them, as they both have the same fully qualified name. Instead, you can use extern aliases to specify which you mean. The following listing shows an example of the C# code involved, along with the command line needed to compile it. // Compile with // csc Test.cs /r:FirstAlias=First.dll /r:SecondAlias=Second.dll extern alias FirstAlias; extern alias SecondAlias; using System; using FD = FirstAlias::Demo; class Test { static void Main() Listing 7.8 Working with different types of the same type in different assemblies Specifies two extern aliasesB Refers to extern alias with namespace aliasC 197Pragma directives { Console.WriteLine(typeof(FD.Example)); Console.WriteLine(typeof(SecondAlias::Demo.Example)); } } The code in listing 7.8 is straightforward. The first thing you have to do is introduce the two extern aliases B. After that, you can use them either via namespace aliases (C and D) or directly E. In fact, a normal using directive without an alias (such as using FirstAlias::Demo;) would’ve allowed you to use the name Example without any further qualification at all. One extern alias can cover multiple assemblies, and several extern aliases can all refer to the same assembly, although I’d think carefully before using either of these features, and particularly before combining them. To specify an extern alias in Visual Studio, just select the assembly reference within Solution Explorer and modify the Aliases value in the Properties window, as shown in figure 7.3. Hopefully I don’t need to persuade you to avoid this kind of situation whenever you can. It can be necessary to work with assemblies from dif- ferent third parties who happen to have used the same fully qualified type names, at which point you’d otherwise be stuck. Where you have more control over the naming, though, make sure that your names never lead you into this territory. The next feature is almost a metafeature. The exact functionality it provides depends on which compiler you’re using, because its purpose is to enable control over compiler-specific features. We’ll concentrate on the Microsoft compiler. 7.5 Pragma directives Describing pragma directives in general is extremely easy: a pragma directive is a pre- processing directive represented by a line beginning with #pragma. The rest of the line can contain any text at all. The result of a pragma directive can’t change the behavior of the program to contravene anything within the C# language specification, but it can do anything outside the scope of the specification. If the compiler doesn’t under- stand a particular pragma directive, it can issue a warning, but not an error. That’s basically everything the specification has to say on the subject. The Micro- soft C# compiler understands two pragma directives: warnings and checksums. 7.5.1 Warning pragmas Occasionally, the C# compiler issues warnings that are justifiable but annoying. The correct response to a compiler warning is almost always to fix your code—it's rarely made any worse by fixing the cause of the warning, and usually it’s improved. Uses namespace aliasD Uses extern alias directlyE Figure 7.3 Part of the Properties window of Visual Studio 2010, showing an extern alias of FirstAlias for the First.dll reference 198 CHAPTER 7 Concluding C# 2: the final features But sometimes there’s a good reason to ignore a warning, and that’s what warn- ing pragmas are available for. As an example, you’ll create a private field that’s never read from or written to. It’s almost always going to be useless...unless you happen to know that it’ll be used by reflection. The following listing is a complete class demon- strating this. public class FieldUsedOnlyByReflection { int x; } If you try to compile listing 7.9, you’ll get a warning message like this: FieldUsedOnlyByReflection.cs(3,9): warning CS0169: The private field 'FieldUsedOnlyByReflection.x' is never used That’s the output from the command-line compiler. In the Error List window of Visual Studio, you can see the same information (plus the project it’s in) except that you don’t get the warning number (CS0169). To find the number, you need to either select the warning and bring up the help related to it, or look in the Output window, where the full text is shown. You need the number in order to make the code compile without warnings, as shown in the following listing. public class FieldUsedOnlyByReflection { #pragma warning disable 0169 int x; #pragma warning restore 0169 } Listing 7.10 is self-explanatory—the first pragma disables the specified warning, and the second one restores it. It’s good practice to disable warnings for as short a time as you can, so that you don’t miss any warnings you genuinely ought to fix. If you want to disable or restore multiple warnings in a single line, just use a comma-separated list of warning numbers. If you don’t specify any warning numbers at all, the result is to disable or restore all warnings in one fell swoop, but that’s a bad idea in almost every imaginable scenario. 7.5.2 Checksum pragmas You’re unlikely to need the second form of pragma recognized by the Microsoft com- piler. It supports the debugger by allowing it to check that it’s found the right source file. Normally when a C# file is compiled, the compiler generates a checksum from the file and includes it in the debugging information. When the debugger needs to locate a source file and finds multiple potential matches, it can generate the check- sum itself for each of the candidate files and see which is correct. Listing 7.9 Class containing an unused field Listing 7.10 Disabling (and restoring) warning CS0169 199Fixed-size buffers in unsafe code When an ASP.NET page is converted into C#, the generated file is what the C# com- piler sees. The generator calculates the checksum of the .aspx page and uses a check- sum pragma to tell the C# compiler to use that checksum instead of calculating one from the generated page. The syntax of the checksum pragma is #pragma checksum "filename" "{guid}" "checksum bytes" The GUID indicates which hashing algorithm has been used to calculate the check- sum. The documentation for the CodeChecksumPragma class gives GUIDs for SHA-1 and MD5, should you ever wish to implement your own dynamic compilation framework with debugger support. It’s possible that future versions of the C# compiler will include more pragma directives, and other compilers (such as the Mono compiler, mcs) could have their own support for different features. Consult your compiler documentation for the most up-to-date information. The next feature is another one that you may never use, but if you ever do, it’s likely to make your life somewhat simpler. 7.6 Fixed-size buffers in unsafe code When calling into native code with P/Invoke, it’s not unusual to find yourself dealing with a structure that’s defined to have a buffer of a particular length within it. Prior to C# 2, such structures were difficult to handle directly, even with unsafe code. Now you can declare a buffer of the right size to be embedded directly with the rest of the data for the structure. This capability isn’t just available for calling native code, although that’s its primary use. You could use it to easily populate a data structure directly corresponding to a file format, for instance. The syntax is simple, and once again we’ll demonstrate it with an example. To create a field that embeds an array of 20 bytes within its enclosing struc- ture, you’d use fixed byte data[20]; This would allow data to be used as if it were a byte* (a pointer to byte data), although the implementation used by the C# compiler is to create a new nested type within the declaring type and apply the new FixedBuffer attribute to the variable itself. The CLR then takes care of allocating the memory appropriately. One downside of this feature is that it’s only available within unsafe code: the enclos- ing structure has to be declared in an unsafe context, and you can only use the fixed- size buffer member within an unsafe context. This limits the situations in which it’s use- ful, but it can still be a nice trick to have up your sleeve. Also, fixed-size buffers are only applicable to primitive types and can’t be members of classes (only structures). There are remarkably few Windows APIs where this feature is directly useful. Numerous situations call for a fixed array of characters—the TIME_ZONE_INFORMATION 200 CHAPTER 7 Concluding C# 2: the final features structure, for example—but unfortunately fixed-size buffers of characters appear to be handled poorly by P/Invoke, with the marshaler getting in the way. The following listing shows one example, though—a console application that dis- plays the colors available in the current console window. It uses an API function, Get- ConsoleScreenBufferEx, that was introduced in Windows Vista and Windows Server 2008 and that retrieves extended console information. The following listing displays all 16 colors in hexadecimal format (bbggrr). using System; using System.Runtime.InteropServices; struct COORD { public short X, Y; } struct SMALL_RECT { public short Left, Top, Right, Bottom; } unsafe struct CONSOLE_SCREEN_BUFFER_INFOEX { public int StructureSize; public COORD ConsoleSize, CursorPosition; public short Attributes; public SMALL_RECT DisplayWindow; public COORD MaximumWindowSize; public short PopupAttributes; public int FullScreenSupported; public fixed int ColorTable[16]; } static class FixedSizeBufferDemo { const int StdOutputHandle = -11; [DllImport("kernel32.dll")] static extern IntPtr GetStdHandle(int nStdHandle); [DllImport("kernel32.dll")] static extern bool GetConsoleScreenBufferInfoEx (IntPtr handle, ref CONSOLE_SCREEN_BUFFER_INFOEX info); unsafe static void Main() { IntPtr handle = GetStdHandle(StdOutputHandle); CONSOLE_SCREEN_BUFFER_INFOEX info; info = new CONSOLE_SCREEN_BUFFER_INFOEX(); info.StructureSize = sizeof(CONSOLE_SCREEN_BUFFER_INFOEX); GetConsoleScreenBufferInfoEx(handle, ref info); for (int i=0; i < 16; i++) { Listing 7.11 Demonstration of fixed-size buffers to obtain console color information 201Exposing internal members to selected assemblies Console.WriteLine ("{0:x6}", info.ColorTable[i]); } } } Listing 7.11 uses fixed-size buffers for the table of colors. Before fixed-size buffers, you could have used the API either with a field for each color table entry or by marshaling a normal array as UnmanagedType.ByValArray. But this would’ve created a separate array on the heap instead of keeping the information all within the structure. That’s not a problem here, but in some high-performance situations it’s nice to be able to keep lumps of data together. On a different performance note, if the buffer is part of a data structure on the managed heap, you have to pin it before accessing it. If you do this a lot, it can significantly affect the garbage collector. Stack-based structures don’t have this problem, of course. I don’t claim that fixed-size buffers are a hugely important feature in C# 2—at least, not to most people. I’ve included them here for completeness, and doubtless someone, somewhere, will find them invaluable. The final feature we’ll look at can barely be called a C# 2 language feature at all, but it just about counts. 7.7 Exposing internal members to selected assemblies Some features are obviously in the language—iterator blocks, for example. Some fea- tures obviously belong to the runtime, such as JIT compiler optimizations. Some clearly sit in both camps, such as generics. This last feature has a toe in both but is suf- ficiently odd that it doesn’t merit a mention in either specification. In addition, it uses a term that has different meanings in C++ and VB.NET, adding a third meaning to the mix. To be fair, all the terms are used in the context of access permissions, but they have different effects. 7.7.1 Friend assemblies in the simple case In .NET 1.1 it was entirely accurate to say that something defined to be internal (whether a type, a method, a property, a variable, or an event) could only be accessed within the same assembly in which it was declared.3 In .NET 2.0 that’s still mostly true, but there’s a new attribute that lets you bend the rules slightly: InternalsVisible- ToAttribute, usually referred to as just InternalsVisibleTo. (When applying an attri- bute whose name ends with Attribute, the C# compiler will apply the suffix automatically.) InternalsVisibleTo can only be applied to an assembly (not a specific member), and you can apply it multiple times to the same assembly. I’ll call the assembly con- taining the attribute the source assembly, although this is unofficial terminology. When you apply the attribute, you have to specify another assembly, known as the friend assembly. The result is that the friend assembly can see all the internal members of the 3 Using reflection when running with suitable permissions doesn’t count. 202 CHAPTER 7 Concluding C# 2: the final features source assembly as if they were public. This may sound alarming, but it can be useful, as you’ll see in a minute. The following listing shows this with three classes in three different assemblies. // Compiled to Source.dll using System.Runtime.CompilerServices; [assembly:InternalsVisibleTo("FriendAssembly")] public class Source { internal static void InternalMethod() {} public static void PublicMethod() {} } // Compiled to FriendAssembly.dll public class Friend { static void Main() { Source.InternalMethod(); Source.PublicMethod(); } } // Compiled to EnemyAssembly.dll public class Enemy { static void Main() { // Source.InternalMethod(); Source.PublicMethod(); } } In listing 7.12, a special relationship exists between FriendAssembly.dll and Source.dll, although it only operates one way: Source.dll has no access to internal members of FriendAssembly.dll. If you were to uncomment the line at B, the Enemy class would fail to compile. Why on earth would you want to open up your well-designed assembly to certain assemblies to start with? 7.7.2 Why use InternalsVisibleTo? I rarely use InternalsVisibleTo between two production assemblies. I can see how it could be useful, and I’ve certainly used it for extra access when writing tools, but my primary use of it has always been unit testing. Some say you should only test the public interface to code. Personally I’m happy to test whatever I can in the simplest manner possible. Friend assemblies make that a lot easier: suddenly it’s trivial to test code that only has internal access without taking the dubious step of making members public just for the sake of testing or including the Listing 7.12 Demonstration of friend assemblies Grants additional access Uses additional access within FriendAssembly EnemyAssembly has no special access B Access public method as normal 203Exposing internal members to selected assemblies test code within the production assembly. (It does occasionally mean making mem- bers internal for the sake of testing where they might otherwise be private, but that’s less worrying.) The only downside to this is that the name of your test assembly lives on in your production assembly. In theory, this could represent a security attack vector if your assemblies aren’t signed and if your code normally operates under a restricted set of permissions. (Anyone with full trust could use reflection to access the members in the first place. You could do that yourself for unit tests, but it’s much nastier.) If this ever ends up as a genuine issue for anyone, I’ll be very surprised. But it does bring the option of signing assemblies into the picture. Just when you thought this was a nice, simple little feature... 7.7.3 InternalsVisibleTo and signed assemblies If a friend assembly is signed, the source assembly needs to specify the public key of the friend assembly, to make sure it’s trusting the right code. You need the full public key, not just the public key token. For instance, consider the following command line and output (rewrapped and modified slightly for formatting) used to discover the public key of a signed Friend- Assembly.dll: c:\Users\Jon\Test>sn -Tp FriendAssembly.dll Microsoft (R) .NET Framework Strong Name Utility Version 3.5.21022.8 Copyright (c) Microsoft Corporation. All rights reserved. Public key is 0024000004800000940000000602000000240000525341310004000001 000100a51372c81ccfb8fba9c5fb84180c4129e50f0facdce932cf31fe 563d0fe3cb6b1d5129e28326060a3a539f287aaf59affc5aabc4d8f981 e1a82479ab795f410eab22e3266033c633400463ee7513378bb4ef41fc 0cae5fb03986d133677c82a865b278c48d99dc251201b9c43edd7bedef d4b5306efd0dec7787ec6b664471c2 Public key token is 647b99330b7f792c The source code for the Source class would now need to have this as the attribute: [assembly:InternalsVisibleTo("FriendAssembly,PublicKey=" + "0024000004800000940000000602000000240000525341310004000001" + "000100a51372c81ccfb8fba9c5fb84180c4129e50f0facdce932cf31fe" + "563d0fe3cb6b1d5129e28326060a3a539f287aaf59affc5aabc4d8f981" + "e1a82479ab795f410eab22e3266033c633400463ee7513378bb4ef41fc" + "0cae5fb03986d133677c82a865b278c48d99dc251201b9c43edd7bedef" + "d4b5306efd0dec7787ec6b664471c2")] Unfortunately, you need to either have the public key on one line or use string concat- enation—whitespace in the public key will cause a compilation failure. It’d be a lot more pleasant to look at if you could specify the token instead of the whole key, but fortunately this ugliness is usually confined to AssemblyInfo.cs, so you won’t need to see it often. 204 CHAPTER 7 Concluding C# 2: the final features In theory, it’s possible to have an unsigned source assembly and a signed friend assembly. In practice, that’s not terribly useful, as the friend assembly typically wants to have a reference to the source assembly, and you can’t refer to an unsigned assembly from one that’s signed. Likewise, a signed assembly can’t specify an unsigned friend assembly, so typically you end up with both assemblies being signed if either one of them is. 7.8 Summary This completes our tour of the new features in C# 2. The topics we’ve looked at in this chapter have broadly fallen into two categories: “nice to have” improvements that streamline development, and “hope you don’t need it” features that can get you out of tricky situations when you need them. To make an analogy between C# 2 and improve- ments to a house, the major features from our earlier chapters are comparable to full- scale additions. Some of the features we’ve seen in this chapter (such as partial types and static classes) are more like redecorating a bedroom, and features such as namespace aliases are akin to fitting smoke alarms—you may never see a benefit, but it’s nice to know they’re there if you ever need them. The range of features in C# 2 is broad—the designers tackled many of the areas where developers were feeling pain, without any one overarching goal. That’s not to say the features don’t work well together—nullable value types wouldn’t be feasible without generics, for instance—but there’s no one aim that every feature contributes to, unless you count general productivity. Now that we’ve finished examining C# 2, it’s time to move on to C# 3, where the picture is very different. Nearly every feature in C# 3 forms part of the grand picture of LINQ, a conglomeration of technologies that massively simplifies many tasks. Part 3 C# 3: Revolutionizing data access There’s no doubt that C# 2 is a significant improvement over C# 1. The ben- efits of generics, in particular, are fundamental to other changes, not just in C# 2 but also in C# 3. But C# 2 is in some sense a piecemeal collection of features. Don’t get me wrong: they fit together nicely enough, but they address a set of individual issues. That was appropriate at that stage of C#’s development, but C# 3 is different. Almost every feature in C# 3 enables one specific technology: LINQ. Many of the features are useful outside this context, and you certainly shouldn’t confine yourself to only using them when you happen to be writing a query expression, but it’d be equally silly not to recognize the complete picture created by the set of jigsaw puzzle pieces presented in the following five chapters. When I originally wrote about C# 3 and LINQ in 2007, I was highly impressed on a somewhat academic level. The more deeply you study the language, the more clearly you see the harmony between the various elements that have been introduced. The elegance of query expressions—and in particular the ability to use the same syntax for both in-process queries and providers like LINQ to SQL—was very appealing. LINQ had a great deal of promise. Now, years later, I can look back on the promises and see how they’ve played out. In my experience with the community—particularly on Stack Overflow—it’s obvious that LINQ has been widely adopted and really has changed how we approach many data-oriented tasks. Database providers aren’t restricted to those from Microsoft—LINQ to NHibernate and SubSonic are just two of the other options available. Microsoft hasn’t stopped innovating around LINQ either; in chapter 12 you’ll see Parallel LINQ and Reactive Extensions, two very different ways of handling data that still use the familiar LINQ operators. And then there’s LINQ to Objects—the simplest, most predictable, almost mundane LINQ provider, and the one that’s most pervasive in industry. The days of writing yet another filtering loop, yet another piece of code to find some maximum value, yet another check to see whether any items in a collection satisfy some condition have gone—and good riddance. Despite the broad adoption of LINQ, I still see a number of questions that make it clear that some developers regard LINQ as a sort of magic black box. What’s going to happen when I use a query expression, compared with using extension methods directly? When does the data actually get read? How can I make it work more effi- ciently? Though you can learn a lot of LINQ just by playing with it and looking at examples in blog posts, you’ll get a great deal more out of it by seeing how it all works at a language level and then learning about what the various libraries do for you. This is not a book about LINQ—I’m still concentrating on the language features that enable LINQ, rather than going into details of concurrency considerations for the Entity Framework and so on. But once you’ve seen the language elements individually and how they fit together, you’ll be in a much better position to learn the details of specific providers. 207 Cutting fluff with a smart compiler We’ll start looking at C# 3 in the same way that we finished looking at C# 2—with a collection of relatively simple features. These are just the first small steps on the path to LINQ. Each of them can be used outside that context, but almost all are important for simplifying code to the extent that LINQ requires in order to be effective. One important point to note is that although two of the biggest features of C# 2—generics and nullable types—required CLR changes, there were no signifi- cant changes to the CLR that shipped with .NET 3.5. There were some tweaks, but nothing fundamental. The framework grew to support LINQ, and a few more fea- tures were introduced to the base class library, but that’s a different matter. It’s This chapter covers  Automatically implemented properties  Implicitly typed local variables  Object and collection initializers  Implicitly typed arrays  Anonymous types 208 CHAPTER 8 Cutting fluff with a smart compiler worth being clear in your mind which changes are only in the C# language, which are library changes, and which are CLR changes. Almost all of the new features exposed in C# 3 are due to the compiler being will- ing to do more work for you. You saw some evidence of this in part of the book—par- ticularly with anonymous methods and iterator blocks—and C# 3 continues in the same vein. In this chapter, you’ll meet the following features that are new to C# 3:  Automatically implemented properties—Remove the drudgery of writing simple properties backed directly by fields  Implicitly typed local variables—Reduce redundancy from local variable declara- tions by inferring the variable type from the initial value  Object and collection initializers—Simplify the creation and initialization of objects in single expressions  Implicitly typed arrays—Reduce redundancy from array-creation expressions by inferring the array type from the contents  Anonymous types—Enable the creation of ad hoc types to contain simple properties In addition to describing what the new features do, I’ll make recommendations about their use. Many of the features of C# 3 require a certain amount of discretion and restraint on the part of the developer. That’s not to say they’re not powerful and incredibly useful—quite the opposite—but the temptation to use the latest and greatest funky syntax shouldn’t be allowed to overrule the drive toward clear and readable code. The considerations I’ll discuss in this chapter (and the rest of the book) will rarely be black and white. Perhaps more than ever before, readability is in the eye of the beholder, and as you become more comfortable with the new features, they’re likely to become more readable to you. I should stress, though, that unless you have good reason to suppose you’ll be the only one to ever read your code, you should consider the needs and views of your colleagues carefully. That’s enough navel gazing for the moment. We’ll start off with a feature that shouldn’t cause any controversy. Simple but effective, automatically implemented properties just make life better. 8.1 Automatically implemented properties The first feature we’ll discuss is probably the simplest in the whole of C# 3. It’s even simpler than any of the new features in C# 2. Despite that—or possibly because of that—it’s also immediately applicable in many, many situations. When you read about iterator blocks in chapter 6, you may not immediately have thought of any areas of your current code base that could be improved by using them, but I’d be surprised to find any nontrivial C# 2 program that couldn’t be modified to use automatically implemented properties. This fabulously simple feature allows you to express trivial properties with less code than before. 209Automatically implemented properties What do I mean by a trivial property? I mean one that’s read/write and that stores its value in a straightforward private variable without any validation or other custom code. Trivial properties only take a few lines of code, but that’s still a lot when you con- sider that you’re expressing a very simple concept. C# 3 reduces the verbosity by applying a simple compile-time transformation, as shown in figure 8.1. The code at the bottom of figure 8.1 isn’t quite valid C#, of course. The field has an unspeakable name to prevent naming collisions, in the same way as you’ve seen before for anonymous methods and iterator blocks. But that’s effectively the code that’s gen- erated by the automatically implemented property at the top. Where previously you might have been tempted to use a public variable for the sake of simplicity, there’s now even less excuse for not using a property instead. This is par- ticularly true for throwaway code, which we all know tends to live far longer than anticipated. TERMINOLOGY: AUTOMATIC PROPERTY OR AUTOMATICALLY IMPLEMENTED PROPERTY? When automatically implemented properties were first dis- cussed, long before the full C# 3 specification was published, they were called automatic properties. Personally, I find this less of a mouthful than the full name, and it’s more widely used in the community. There’s no risk of ambiguity, so for the rest of this book, I’ll use automatic property and automatically implemented property synonymously. The feature of C# 2 that allows you to specify different access for the getter and the setter is still available here, and you can also create static automatic properties. But static automatic properties are almost always pointless. Although most types don’t claim to have thread-safe instance members, publicly visible static members usually should be thread-safe, and the compiler doesn’t do anything to help you in this respect. The following listing gives an example of a safe, but useless, static automatic public string Name { get; set; } private string k__BackingField; public string Name { get { return k__BackingField; } set { k__BackingField = value; } } ...is compiled as... Figure 8.1 Transformation of an automatically implemented property 210 CHAPTER 8 Cutting fluff with a smart compiler property that counts how many instances of a class have been created, along with instance properties for the name and age of a person. public class Person { public string Name { get; private set; } public int Age { get; private set; } private static int InstanceCounter { get; set; } private static readonly object counterLock = new object(); public InstanceCountingPerson(string name, int age) { Name = name; Age = age; lock (counterLock) { InstanceCounter++; } } } In this listing, you use a lock to make sure you don’t have threading problems, and you’d also need to use the same lock whenever you accessed the property. There are better alternatives here involving the Interlocked class, but they require access to fields. In short, the only scenario in which I can see static automatic properties being useful is where the getter is public, the setter is private, and the setter is only called within the type initializer. The other properties in listing 8.1, representing the name and age of the person, tell a much happier tale—using automatic properties is a no-brainer here. Where you have properties that you’d have implemented trivially in previous versions of C#, there’s no benefit in not using automatic properties.1 One slight wrinkle occurs if you use automatic properties when writing your own structs: all of your constructors need to explicitly call the parameterless constructor— this()—so that the compiler knows that all the fields have been definitely assigned. You can’t set the fields directly because they’re anonymous, and you can’t use the prop- erties until all the fields have been set. The only way of proceeding is to call the param- eterless constructor, which will set the fields to their default values. For example, if you wanted to create a struct with a single integer property, this wouldn’t be valid: public struct Foo { public int Value { get; private set; } Listing 8.1 Counting instances awkwardly with a static automatic property 1 Certainly for read/write properties, anyway. If you’re creating a read-only property, you may choose to use a read-only backing field and a property with just a getter to return it. This prevents you from accidentally writ- ing to the property within the class, which would be possible with a “public read, private write” automatic property. Declares properties with public getters Declares private static property and lock Uses lock for safe property access 211Implicit typing of local variables public Foo(int value) { this.Value = value; } } But it’s fine if you explicitly chain to the parameterless constructor: public struct Foo { public int Value { get; private set; } public Foo(int value) : this() { this.Value = value; } } That’s all there is to automatically implemented properties. There are no bells and whistles to them. For instance, there’s no way of declaring them with initial default val- ues, and no way of making them genuinely read-only (a private setter is as close as you can get). If all the C# 3 features were that simple, we could cover everything in a single chap- ter. Of course, that’s not the case, but there are some features that don’t take too much explanation. The next topic removes duplicate code in another common but specific situation—declaring local variables. 8.2 Implicit typing of local variables In chapter 2 we discussed the nature of the C# 1 type system. In particular, I stated that it was static, explicit, and safe. That’s still true in C# 2, and in C# 3 it’s still almost completely true. The static and safe parts are still true (ignoring explicitly unsafe code, just as we did in chapter 2), and most of the time it’s still explicitly typed—but you can ask the compiler to infer the types of local variables for you.2 8.2.1 Using var to declare a local variable In order to use implicit typing, all you need to do is replace the type part of a normal local variable declaration with var. Certain restrictions exist (we’ll come to those in a moment), but essentially it’s as easy as changing this: MyType variableName = someInitialValue; to this: var variableName = someInitialValue; The results of the two lines (in terms of compiled code) are exactly the same, assuming that the type of someInitialValue is MyType. The compiler simply takes the compile- time type of the initialization expression and makes the variable have that type too. 2 C# 4 changes the game yet again, allowing you to use dynamic typing where you want to, as you’ll see in chap- ter 14. One step at a time—C# was still fully statically typed up to and including version 3. 212 CHAPTER 8 Cutting fluff with a smart compiler The type can be any normal .NET type, including generics, delegates, and interfaces. The variable is still statically typed; you just haven’t written the name of the type in your code. This is important to understand, as it goes to the heart of what a lot of developers initially fear when they see this feature—that var makes C# dynamic or weakly typed. That’s not true at all. The best way of explaining this is to show you some invalid code: var stringVariable = "Hello, world."; stringVariable = 0; That doesn’t compile because the type of stringVariable is System.String, and you can’t assign the value 0 to a string variable. In many dynamic languages, the code would have compiled, leaving the variable with no particularly useful type as far as the compiler, IDE, or runtime environment is concerned. Using var is not like using a VARIANT type from COM or VB6. The variable is statically typed; the type has just been inferred by the compiler. I apologize if I seem to be laboring this point somewhat, but it’s incredibly important, and it’s been the cause of a lot of confusion. In Visual Studio, you can tell which type the compiler has used for the variable by hovering over the var part of the declaration, as shown in figure 8.2. Note how the type parameters for the generic Dictionary type are also explained. If this looks familiar, that’s because it’s exactly the same behavior you get when you declare local variables explicitly. Tooltips aren’t just available at the point of declaration, either. As you’d probably expect, the tooltip displayed when you hover over the variable name later on in the code indicates the type of the variable too. This is shown in figure 8.3, where the same declaration is used and then I’ve hovered over a use of the variable. Again, that’s exactly the same behavior as you’d see with a normal local variable declaration. INVALID Figure 8.2 Hovering over var in Visual Studio displays the type of the declared variable. Figure 8.3 Hovering over the use of an implicitly typed local variable displays its type. 213Implicit typing of local variables There are two reasons for bringing up Visual Studio in this context. The first is that it’s more evidence of the static typing involved—the compiler clearly knows the type of the variable. The second is to point out that you can easily discover the type involved, even from deep within a method. This’ll be important when we talk about the pros and cons of using implicit typing in a minute. First, though, I ought to men- tion some limitations. 8.2.2 Restrictions on implicit typing You can’t use implicit typing for every variable in every case. You can only use it when all of the following points are true:  The variable being declared is a local variable, rather than a static or instance field.  The variable is initialized as part of the declaration.  The initialization expression isn’t a method group or anonymous function3 (without casting).  The initialization expression isn’t null.  Only one variable is declared in the statement.  The type you want the variable to have is the compile-time type of the initializa- tion expression.  The initialization expression doesn’t involve the variable being declared.4 The third and fourth points are interesting. You can’t write this: var starter = delegate() { Console.WriteLine(); }; That’s because the compiler doesn’t know what type to use. You can write this: var starter = (ThreadStart) delegate() { Console.WriteLine(); }; But if you’re going to do that, you’d be better off explicitly declaring the variable in the first place. The same is true in the null case—you could cast the null appropri- ately, but there’d be no point. Note that you can use the result of method calls or properties as the initialization expression—you’re not limited to constants and constructor calls. For instance, you could use this: var args = Environment.GetCommandLineArgs(); In that case, args would be of type string[]. In fact, initializing a variable with the result of a method call is likely to be the most common situation where implicit typing is used, as part of LINQ. You’ll see all that later on—just bear it in mind as the exam- ples progress. 3 The term anonymous function covers both anonymous methods and lambda expressions, which we’ll delve into in chapter 9. 4 It’d be highly unusual to do so anyway, but with normal declarations it’s possible if you try hard enough. INVALID 214 CHAPTER 8 Cutting fluff with a smart compiler It’s also worth noting that you are allowed to use implicit typing for the local vari- ables declared in the first part of a using, for, or foreach statement. For example, the following are all valid (with appropriate bodies, of course): for (var i = 0; i < 10; i++) using (var x = File.OpenText("test.dat")) foreach (var s in Environment.GetCommandLineArgs()) These variables would end up with types of int, StreamReader, and string, respectively. Of course, just because you can do this doesn’t mean you should. Let’s look at the reasons for and against using implicit typing. 8.2.3 Pros and cons of implicit typing The question of when it’s a good idea to use implicit typing is the cause of a lot of community discussion. Views range from “everywhere” to “nowhere” with plenty of more balanced approaches between the two. You’ll see in section 8.5 that in order to use another of C# 3’s features—anonymous types—you often need to use implicit typ- ing. You could avoid anonymous types as well, of course, but that’s throwing the baby out with the bathwater. The main reason for using implicit typing (leaving anonymous types aside for the moment) is not that it reduces the number of keystrokes required to enter the code, but that it makes the code less cluttered (and therefore more readable) on the screen. In particular, when generics are involved, the type names can get very long. Figures 8.2 and 8.3 used a type of Dictionary>, which is 33 charac- ters. By the time you have that twice on a line (once for the declaration and once for the initialization), you end up with a massive line just for declaring and initializing a single variable. An alternative is to use an alias, but that puts the real type involved a long way (conceptually at least) from the code that uses it. When reading the code, there’s no point in seeing the same long type name twice on the same line when it’s obvious that they should be the same. If the declaration isn’t visible on the screen, you’re in the same boat whether implicit typing was used or not (all the ways you’d use to find out the variable type are still valid), and if it is visible, the expression used to initialize the variable tells you the type anyway. Additionally, using var changes the emphasis of the code. Sometimes you want the reader to pay close attention to the precise types involved because they’re significant. For example, even though the generic SortedList and SortedDictionary types have similar APIs, they have different performance characteristics, and that may be impor- tant for your particular piece of code. Other times, all you really care about is the operations that are being performed; you wouldn’t really mind if the expression used to initialize the variable changed, as long as you could achieve the same goals.5 Using 5 I realize this sounds a little like duck typing: “As long as it can quack, I’m happy.” The difference is that you’re still checking quackability at compile time, not execution time. 215Implicit typing of local variables var allows the reader to focus on the use of a variable rather than the declaration—the what rather than the how of the code. All of this sounds good, so what are the arguments against implicit typing? Para- doxically enough, readability is the most important one, despite also being an argu- ment in favor of implicit typing! By not being explicit about what type of variable you’re declaring, you may be making it harder to work it out when reading the code. It breaks the “state what you’re declaring, then what value it’ll start off with” mindset that keeps the declaration and the initialization separate. To what extent that’s an issue depends on both the reader and the initialization expression involved. If you’re explicitly calling a constructor, it’ll always be pretty obvious what type you’re creating. If you’re calling a method or using a property, it depends on how obvious the return type is when looking at the call. Integer literals are an example where it’s harder to guess the type of an expression than you might suppose. How quickly can you work out the type of each of the variables declared here? var a = 2147483647; var b = 2147483648; var c = 4294967295; var d = 4294967296; var e = 9223372036854775807; var f = 9223372036854775808; The answers are int, uint, uint, long, long, and ulong, respectively—the type used depends on the value of the expression. There’s nothing new here in terms of the handling of literals—C# has always behaved like this—but implicit typing makes it eas- ier to write obscure code in this case. The argument that’s rarely explicitly stated but that I believe is behind a lot of the concern over implicit typing is, “It just doesn’t feel right.” If you’ve been writing in a C- like language for years and years, there’s something unnerving about the whole busi- ness, however much you tell yourself that it’s still static typing under the covers. This may not be a rational concern, but that doesn’t make it any less real. If you’re uncom- fortable, you’re likely to be less productive. If the advantages don’t outweigh your neg- ative feelings, that’s fine. Depending on your personality, you may try to push yourself to become more comfortable with implicit typing, but you certainly don’t have to. 8.2.4 Recommendations Here are some recommendations based on my experience with implicit typing. That’s all they are—recommendations—and you should feel free to take them with a pinch of salt:  If it’s important that someone reading the code knows the type of the variable at a glance, use explicit typing.  If the variable is directly initialized with a constructor and the type name is long (which often occurs with generics), consider using implicit typing. 216 CHAPTER 8 Cutting fluff with a smart compiler  If the precise type of the variable isn’t important, but its general nature is clear from the context, use implicit typing to de-emphasize how the code achieves its aim and concentrate on the higher level of what it’s achieving.  Consult your teammates on the matter when embarking on a new project.  When in doubt, try a line both ways and go with your gut feelings. I used to use explicit typing for production code, except in situations where there was a clear and significant benefit to using implicit typing. Most of my uses of implicit typ- ing were in test code (and throwaway code). Nowadays I’m more ambivalent and frankly inconsistent. I’ll happily use implicit typing in production code just for a bit of added simplicity, even when the type names involved aren’t too onerous. Although consistency in some aspects of coding style is quite important, I haven’t found this mix-and-match approach to cause any problems. Effectively, my recommendation boils down to not using implicit typing just because it saves a few keystrokes. Where it keeps the code tidier, allowing you to con- centrate on the most important elements of the code, go for it. I’ll use implicit typing extensively in the rest of the book, for the simple reason that code is harder to format in print than on a screen—not as much width is available. We’ll come back to implicit typing when we look at anonymous types, as they cre- ate situations where you’re forced to ask the compiler to infer the types of some vari- ables. Before that, let’s look at how C# 3 makes it easier to construct and populate a new object in one expression. 8.3 Simplified initialization One would’ve thought that object-oriented languages would’ve streamlined object creation long ago. After all, before you start using an object, something has to create it, whether it’s through your code directly or a factory method of some sort. Despite this, few language features in C# 2 are geared toward making life easier when it comes to initialization. If you can’t do what you want using constructor arguments, you’re basi- cally out of luck—you need to create the object, and then manually initialize it with property calls and the like. This is particularly annoying when you want to create a whole bunch of objects in one go, such as in an array or other collection. Without a single-expression way of ini- tializing an object, you’re forced to either use local variables for temporary manipula- tion or create a helper method that performs the appropriate initialization based on parameters. C# 3 comes to the rescue in a number of ways, as you’ll see in this section. 8.3.1 Defining some sample types The expressions we’ll use in this section are called object initializers. These are just ways of specifying initialization that should occur after an object has been created. You can set properties, set properties of properties (don’t worry, it’s simpler than it sounds), and add to collections that are accessible via properties. 217Simplified initialization To demonstrate all this, we’ll use a Person class again. It has the name and age we’ve used before, exposed as writable properties. We’ll provide both a parameterless constructor and one that accepts the name as a parameter. We’ll also add a list of friends and the person’s home location, both of which are accessible as read-only properties but can still be modified by manipulating the retrieved objects. A simple Location class provides Country and Town properties to represent the person’s home. The following listing shows the complete code for the classes. public class Person { public int Age { get; set; } public string Name { get; set; } List friends = new List(); public List Friends { get { return friends; }} Location home = new Location(); public Location Home { get { return home; }} public Person() {} public Person(string name) { Name = name; } } public class Location { public string Country { get; set; } public string Town { get; set; } } Listing 8.2 is straightforward, but it’s worth noting that both the list of friends and the home location are created in a blank way when the person is created, rather than being left as null references. The friends and home location properties are read-only, too. That’ll be important later on—but for the moment let’s look at the properties representing the name and age of a person. 8.3.2 Setting simple properties Now that you have a Person type, it’s time to create some instances of it using the new features of C# 3. In this section, we’ll look at setting the Name and Age properties— we’ll come to the others later. Object initializers are most commonly used to set properties, but everything shown here also applies to fields. In a well-encapsulated system, though, you’re unlikely to have access to fields unless you’re creating an instance of a type within that type’s own code. It’s worth knowing that you can use fields, of course—so for the rest of the sec- tion, just read property and field whenever the text says property. Listing 8.2 A fairly simple Person class used for further demonstrations 218 CHAPTER 8 Cutting fluff with a smart compiler With that out of the way, let’s get down to business. Suppose you want to create a person called Tom, who is 9 years old. Prior to C# 3, there were two ways this could be achieved: Person tom1 = new Person(); tom1.Name = "Tom"; tom1.Age = 9; Person tom2 = new Person("Tom"); tom2.Age = 9; The first version uses the parameterless constructor and then sets both properties. The second version uses the constructor overload, which sets the name, and then sets the age afterward. Both of these options are still available in C# 3, but there are other alternatives: Person tom3 = new Person() { Name = "Tom", Age = 9 }; Person tom4 = new Person { Name = "Tom", Age = 9 }; Person tom5 = new Person("Tom") { Age = 9 }; The part in braces at the end of each line is the object initializer. Again, it’s just com- piler trickery. The IL used to initialize tom3 and tom4 is identical, and is nearly the same as that used for tom1.6 Predictably, the code for tom5 is nearly the same as for tom2. Note how the initialization of tom4 omits the parentheses for the constructor. You can use this shorthand for types with a parameterless constructor, which is what gets called in the compiled code. After the constructor has been called, the specified properties are set in the obvi- ous way. They’re set in the order specified in the object initializer, and you can only specify a particular property once—you can’t set the Name property twice, for example. (You could call the constructor taking the name as a parameter, and then set the Name property. It would be pointless, but the compiler wouldn’t stop you from doing it.) The expression used as the value for a property can be any expression that isn’t itself an assignment—you can call methods, create new objects (potentially using another object initializer), pretty much anything. You may be wondering just how useful this is—you’ve saved one or two lines of code, but surely that’s not a good enough reason to make the language more compli- cated, is it? There’s a subtle point here, though: you haven’t just created an object in one line—you’ve created it in one expression. That difference can be very important. Suppose you want to create an array of type Person[] with some predefined data in it. Even without using the implicit array typing you’ll see later, the code is neat and readable: 6 In fact, tom1’s new value isn’t assigned until all the properties have been set. A temporary local variable is used until then. This is rarely important but worth knowing to avoid confusion if you happen to break into the debugger halfway through the initializer. 219Simplified initialization Person[] family = new Person[] { new Person { Name = "Holly", Age = 36 }, new Person { Name = "Jon", Age = 36 }, new Person { Name = "Tom", Age = 9 }, new Person { Name = "William", Age = 6 }, new Person { Name = "Robin", Age = 6 } }; In a simple example like this, you could’ve written a constructor taking both the name and age as parameters and initialized the array in a similar way in C# 1 or 2. But appro- priate constructors aren’t always available, and if there are several constructor param- eters, it’s often not clear which one means what, just from the position. By the time a constructor needs to take five or six parameters, I often find myself relying on Intelli- Sense more than I want to. Using the property names is a great boon to readability in such cases.7 This form of object initializer is the one you’ll probably use most often. But there are two other forms—one for setting subproperties, and one for adding to collections. Let’s look at subproperties—properties of properties—first. 8.3.3 Setting properties on embedded objects So far it’s been easy to set the Name and Age properties, but you can’t set the Home property in the same way—it’s read-only. You can set the town and the country of a person, though, by first fetching the Home property and then setting properties on the result. The language specification refers to this as setting the properties of an embedded object. Just to make it clear, what we’re talking about is the following C# 1 code: Person tom = new Person("Tom"); tom.Age = 9; tom.Home.Country = "UK"; tom.Home.Town = "Reading"; When you’re populating the home location, each statement is doing a get to retrieve the Location instance, and then a set on the relevant property on that instance. There’s nothing new in that, but it’s worth slowing your mind down to look at it care- fully; otherwise it’s easy to miss what’s going on behind the scenes. C# 3 allows all of this to be done in one expression, as shown here: Person tom = new Person("Tom") { Age = 9, Home = { Country = "UK", Town = "Reading" } }; 7 C# 4 provides an alternative approach here using named arguments, which you’ll meet in chapter 13. 220 CHAPTER 8 Cutting fluff with a smart compiler The compiled code for these snippets is effectively the same. The compiler spots that to the right side of the = sign is another object initializer, and it applies the properties to the embedded object appropriately. The absence of the new keyword in the part initializing Home is significant. If you need to work out where the compiler is going to create new objects and where it’s going to set properties on existing ones, look for occurrences of new in the initializer. Every time a new object is created, the new keyword appears somewhere. FORMATTING OBJECT INITIALIZER CODE As with almost all C# features, object initializers are whitespace-independent. You can collapse the whitespace in the object initializer, putting it all on one line if you like. It’s up to you to work out where the sweet spot is in balancing long lines against lots of lines. We’ve dealt with the Home property, but what about Tom’s friends? There are proper- ties you can set on a List, but none of them will add entries to the list. It’s time for the next feature—collection initializers. 8.3.4 Collection initializers Creating a collection with some initial values is an extremely common task. Until C# 3 arrived, the only language feature that gave any assistance was array creation, and even that was clumsy in many situations. C# 3 has collection initializers, which allow you to use the same type of syntax as array initializers but with arbitrary collections and with more flexibility. CREATING NEW COLLECTIONS WITH COLLECTION INITIALIZERS As a first example, let’s use the now-familiar List type. In C# 2, you could populate a list either by passing in an existing collection or by calling Add repeatedly after creat- ing an empty list. Collection initializers in C# 3 take the latter approach. Suppose we want to populate a list of strings with some names—here’s the C# 2 code (on the left) and the close equivalent in C# 3 (on the right): Just as with object initializers, you can specify constructor arguments if you want, or use a parameterless constructor either explicitly or implicitly. The use of implicit typ- ing here was partly for space reasons—the names variable could equally well have been declared explicitly. Reducing the number of lines of code (without reducing readabil- ity) is nice, but there are two bigger benefits of collection initializers:  The create-and-initialize part counts as a single expression.  There’s a lot less clutter in the code. List names = new List(); names.Add("Holly"); names.Add("Jon"); names.Add("Tom"); names.Add("Robin"); names.Add("William"); var names = new List { "Holly", "Jon", "Tom", "Robin", "William" }; 221Simplified initialization The first point becomes important when you want to use a collection as either an argument to a method or as one element in a larger collection. That happens relatively rarely (although often enough to still be useful). The second point is the real reason this is a killer feature in my view. If you look at the code on the right, you can easily see the information you need, with each piece of information written only once. The vari- able name occurs once, the type being used occurs once, and each of the elements of the initialized collection appears once. It’s all extremely simple, and much clearer than the C# 2 code, which contains a lot of fluff around the useful bits. Collection initializers aren’t limited to just lists. You can use them with any type that implements IEnumerable, as long as it has an appropriate Add method for each element in the initializer. You can use an Add method with more than one parameter by putting the values within another set of braces. The most common use for this is creating dictionaries. For example, if you wanted a dictionary mapping names to ages, you could use the following code: Dictionary nameAgeMap = new Dictionary { {"Holly", 36 }, {"Jon", 36 }, {"Tom", 9 } }; In this case, the Add(string, int) method would be called three times. If multiple Add methods are available, different elements of the initializer can call different over- loads. If no compatible overload is available for a specified element, the code will fail to compile. There are two interesting points about the design decision here:  The fact that the type has to implement IEnumerable is never used by the compiler.  The Add method is only found by name—there’s no interface requirement specifying it. These are both pragmatic decisions. Requiring IEnumerable to be implemented is a reasonable attempt to check that the type really is a collection of some sort, and using any accessible overload of the Add method (rather than requiring an exact signature) allows for simple initializations, such as the earlier dictionary example. An early draft of the C# 3 specification required ICollection to be imple- mented instead, and the implementation of the single-parameter Add method (as specified by the interface) was called rather than allowing different overloads. This sounds more pure, but there are far more types that implement IEnumerable than ICollection, and using the single-parameter Add method would be inconvenient. For example, in this case it would’ve forced you to explicitly create an instance of a KeyValuePair for each element of the initializer. Sacrificing a bit of aca- demic purity has made the language far more useful in real life. 222 CHAPTER 8 Cutting fluff with a smart compiler POPULATING COLLECTIONS WITHIN OTHER OBJECT INITIALIZERS So far we’ve only looked at collection initializers used in a standalone fashion to cre- ate whole new collections. They can also be combined with object initializers to popu- late embedded collections. To demonstrate this, we’ll go back to the Person example. The Friends property is read-only, so you can’t create a new collection and specify that as the collection of friends, but you can add to whatever collection is returned by the property’s getter. The way you do this is similar to the syntax you’ve already seen for setting properties of embedded objects, but you just specify a collection initializer instead of a sequence of properties. Let’s see this in action by creating another Person instance for Tom, this time with some of his friends. Person tom = new Person { Name = "Tom", Age = 9, Home = { Town = "Reading", Country = "UK" }, Friends = { new Person { Name = "Alberto" }, new Person("Max"), new Person { Name = "Zak", Age = 7 }, new Person("Ben"), new Person("Alice"), { Age = 9, Home = { Town = "Twyford", Country = "UK" } } } }; Listing 8.3 uses all the features of object and collection initializers we’ve come across. The main part of interest is the collection initializer, which itself uses lots of different forms of object initializers internally. Note that you’re not creating a new collection here, just adding to an existing one. (If the property had a setter, you could create a new collection and still use collection initializer syntax.) You could’ve gone further, specifying friends of friends, friends of friends of friends, and so forth. But you couldn’t specify that Tom is Alberto’s friend. While you’re still initializing an object, you don’t have access to it, so you can’t express cyclic relationships. This can be awkward in a few cases, but it usually isn’t a problem. Collection initialization within object initializers works as a sort of cross between standalone collection initializers and setting embedded object properties. For each element in the collection initializer, the collection property getter (Friends, in this case) is called, and then the appropriate Add method is called on the returned value. The collection isn’t cleared in any way before elements are added. For example, if you were to decide that a person should always be his own friend, and added this to the Listing 8.3 Building up a rich object using object and collection initializers Implicitly calls parameterless constructorSets properties directly Initializes embedded object Initializes collection with further object initializers 223Simplified initialization list of friends within the Person constructor, using a collection initializer would only add extra friends. As you can see, the combination of collection and object initializers can be used to populate whole trees of objects. But when and where is this likely to actually happen? 8.3.5 Uses of initialization features Trying to pin down exactly where these features are useful is reminiscent of being in a Monty Python sketch about the Spanish Inquisition—every time you think you have a reasonably complete list, another common example pops up. I’ll just mention three examples, which I hope will encourage you to consider where else you might use them. CONSTANT COLLECTIONS It’s not uncommon for me to want some kind of collection (often a map) that’s effec- tively constant. Of course, it can’t be a constant as far as the C# language is concerned, but it can be declared static and read-only, with big warnings to say that it shouldn’t be changed. (It’s usually private, so that’s good enough. Alternatively, you can use Read- OnlyCollection.) Typically, this used to involve writing a static constructor or a helper method, just to populate the map. With C# 3’s collection initializers, it’s easy to set the whole thing up inline. SETTING UP UNIT TESTS When writing unit tests, I frequently want to populate an object just for one test, often passing it in as an argument to the method I’m trying to test at the time. Writing all of the initialization longhand can be long-winded and also hides the essential structure of the object from the reader of the code, just as XML creation code can often obscure what the document would look like if you viewed it (appropriately formatted) in a text editor. With appropriate indentation of object initializers, the nested structure of the object hierarchy can become obvious in the very shape of the code, as well as make the values stand out more than they would otherwise. THE BUILDER PATTERN For various reasons, sometimes you want to specify a lot of values for a single method or constructor call. The most common situation in my experience is creating an immutable object. Instead of having a huge set of parameters (which can become a readability problem as the meaning of each argument becomes unclear8), you can use the builder pattern—create a mutable type with appropriate properties, and then pass an instance of the builder into the constructor or method. The framework Process- StartInfo type is a good example of this—the designers could have overloaded Process.Start with many different sets of parameters, but using ProcessStartInfo makes everything clearer. Object and collection initializers allow you to create the builder object in a clearer manner—you can even specify it inline when you call the original member if you want. Admittedly, you still have to write the builder type in the first place, but auto- matic properties help on that front. 8 Named arguments in C# 4 help in this area, admittedly. 224 CHAPTER 8 Cutting fluff with a smart compiler Of course, there are uses beyond these three in ordinary code, and I don’t want to put you off using the new features elsewhere. There’s little reason not to use them, other than possibly confusing developers who aren’t familiar with C# 3 yet. You may decide that using an object initializer just to set one property (as opposed to explicitly setting it in a separate statement) is over the top—that’s a matter of aesthetics, and I can’t give you much objective guidance there. As with implicit typing, it’s a good idea to try the code both ways, and learn to predict your own (and your team’s) reading preferences. So far we’ve looked at a fairly diverse range of features: implementing properties easily, simplifying local variable declarations, and populating objects in single expres- sions. In the remainder of this chapter, we’ll gradually bring these topics together, using more implicit typing and more object population, and creating whole types with- out giving any implementation details. The next topic appears to be quite similar to collection initializers when you look at code using it. I mentioned earlier that array initialization was a bit clumsy in C# 1 and 2. I’m sure it won’t surprise you to learn that it’s been streamlined for C# 3. Let’s take a look. 8.4 Implicitly typed arrays In C# 1 and 2, initializing an array as part of a variable declaration and initialization statement was quite neat, but if you wanted to do it anywhere else, you had to specify the exact array type involved. For example, this compiles without any problem: string[] names = {"Holly", "Jon", "Tom", "Robin", "William"}; This doesn’t work for parameters, though—suppose you wanted to make a call to MyMethod, declared as void MyMethod(string[] names). This code won’t work: MyMethod({"Holly", "Jon", "Tom", "Robin", "William"}); Instead, you have to tell the compiler what type of array you want to initialize: MyMethod(new string[] {"Holly", "Jon", "Tom", "Robin", "William"}); C# 3 allows something in between: MyMethod(new[] {"Holly", "Jon", "Tom", "Robin", "William"}); Clearly the compiler needs to work out what type of array to use. It starts by forming a set containing all the compile-time types of the expressions inside the braces. If there’s exactly one type in that set that all the others can be implicitly converted to, that’s the type of the array. Otherwise (or if all the values are typeless expressions, such as constant null values or anonymous methods, with no casts) the code won’t compile. Note that only the types of the expressions are considered as candidates for the overall array type. This means that occasionally you might have to explicitly cast a value to a less-specific type. For instance, this won’t compile: new[] { new MemoryStream(), new StringWriter() } INVALID INVALID 225Anonymous types There’s no conversion from MemoryStream to StringWriter or vice versa. Both are implicitly convertible to object and IDisposable, but the compiler only considers types that are in the original set produced by the expressions themselves. If you change one of the expressions in this situation so that its type is either object or IDisposable, the code compiles: new[] {(IDisposable) new MemoryStream(), new StringWriter() } The type of this last expression is implicitly IDisposable[]. Of course, at that point you might as well explicitly state the type of the array just as you would in C# 1 and 2, to make it clearer what you’re trying to achieve. Compared with the earlier features, implicitly typed arrays are a bit of an anticli- max. I find it hard to get excited about them, even though they do make life simpler in cases where an array is passed as an argument. The designers haven’t gone mad, though—there’s one important situation in which this implicit typing is absolutely crucial. That’s when you don’t know (and can’t know) the name of the type of the ele- ments of the array. How can you possibly get into this peculiar state? Read on… 8.5 Anonymous types Implicit typing, object and collection initializers, and implicit array typing are all use- ful in their own right, to a greater or lesser extent. But they also serve a higher pur- pose—they make it possible to work with this chapter’s final feature, anonymous types. In turn, anonymous types serve the higher purpose of LINQ. 8.5.1 First encounters of the anonymous kind It’s much easier to explain anonymous types when you already have some idea of what they are through an example. I’m sorry to say that without the use of extension meth- ods and lambda expressions, the examples in this section are likely to be a little con- trived, but there’s a chicken-and-egg situation here: anonymous types are most useful within the context of the more advanced features, but we need to cover the building blocks before we can look at much of the bigger picture. Stick with it—it will make sense in the long run, I promise. Let’s pretend we didn’t have the Person class, and the only properties we cared about were the name and age. The following listing shows how you could still build objects with those properties, without ever declaring a type. var tom= new { Name = "Tom", Age = 9 }; var holly = new { Name = "Holly", Age = 36 }; var jon = new { Name = "Jon", Age = 36 } ; Console.WriteLine("{0} is {1} years old", jon.Name, jon.Age); As you can tell from listing 8.4, the syntax for initializing an anonymous type is similar to the object initializers you saw in section 8.3.2—it’s just that the name of the type is Listing 8.4 Creating objects of an anonymous type with Name and Age properties 226 CHAPTER 8 Cutting fluff with a smart compiler missing between new and the opening brace. Here you’re using implicitly typed local variables because that’s all you can use (other than object of course)—you don’t have a type name to declare the variable with. As you can see from the last line, the type has properties for Name and Age, both of which can be read and which will have the values specified in the anonymous object initializer used to create the instance, so in this case the output is Jon is 36 years old. The properties have the same types as the expres- sions in the initializers—string for Name and int for Age. Just as in normal object ini- tializers, the expressions used in anonymous object initializers can call methods or constructors, fetch properties, perform calculations—whatever you need to do. You might now be starting to see why implicitly typed arrays are important. Sup- pose you want to create an array containing the whole family, and then iterate through it to work out the total age.9 The following listing does just that, and it dem- onstrates a few other interesting features of anonymous types at the same time. var family = new[] { new { Name = "Holly", Age = 36 }, new { Name = "Jon", Age = 36 }, new { Name = "Tom", Age = 9 }, new { Name = "Robin", Age = 6 }, new { Name = "William", Age = 6 } }; int totalAge = 0; foreach (var person in family) { totalAge += person.Age; } Console.WriteLine("Total age: {0}", totalAge); Putting together listing 8.5 and what you learned about implicitly typed arrays in sec- tion 8.4, you can deduce something important: all the people in the family are of the same type. If each use of an anonymous object initializer C referred to a different type, the compiler couldn’t infer an appropriate type for the array B. Within any given assem- bly, the compiler treats two anonymous object initializers as the same type if there are the same number of properties, with the same names and types in the same order. In other words, if you swapped the Name and Age properties in one of the initializers, there’d be two different types involved; likewise, if you introduced an extra property in one line, or used a long instead of an int for the age of one person, another anon- ymous type would’ve been introduced. At that point, the type inference for the array would fail. 9 If you already know LINQ, you may feel that this is a quaint way of summing the ages. I agree, calling family.Sum(p => p.Age) would be a lot neater—but let’s take things one step at a time. Listing 8.5 Populating an array using anonymous types and then finding the total age Uses an implicitly typed array initializerB Uses same anonymous type five timesC Uses implicit typing for person D Sums agesE 227Anonymous types IMPLEMENTATION DETAIL: HOW MANY TYPES? If you ever decide to look at the IL (or decompiled C#) for an anonymous type generated by Microsoft’s com- piler, be aware that although two anonymous object initializers with the same property names in the same order but using different property types will pro- duce two different types, they’ll actually be generated from a single generic type. The generic type is parameterized, but the closed, constructed types will be different because they’ll be given different type arguments for the differ- ent initializers. Notice that you can use a foreach statement to iterate over the array, just as you would any other collection. The type involved is inferred D, and the type of the person vari- able is the same anonymous type you used in the array. Again, you can use the same variable for different instances because they’re all of the same type. Listing 8.5 also proves that the Age property really is strongly typed as an int— otherwise trying to sum the ages E wouldn’t compile. The compiler knows about the anonymous type, and Visual Studio is even willing to share the information via tool- tips, in case you’re uncertain. Figure 8.4 shows the result of hovering over the person part of the person.Age expression from listing 8.5. Now that you’ve seen anonymous types in action, let’s go back and look at what the compiler is actually doing. 8.5.2 Members of anonymous types Anonymous types are created by the compiler and included in the compiled assembly in the same way as the extra types for anonymous methods and iterator blocks. The CLR treats them as perfectly ordinary types, and so they are—if you later move from an anonymous type to a normal, manually coded type with the behavior described in this section, you shouldn’t see anything change. Anonymous types contain the following members:  A constructor taking all the initialization values. The parameters are in the same order as they were specified in the anonymous object initializer, and they have the same names and types.  Public read-only properties.  Private read-only fields backing the properties.  Overrides for Equals, GetHashCode, and ToString. Figure 8.4 Hovering over a variable that’s declared (implicitly) to be of an anonymous type shows the details of that anonymous type. 228 CHAPTER 8 Cutting fluff with a smart compiler That’s it. There are no implemented interfaces, no cloning or serialization capabili- ties—just a constructor, some properties, and the normal methods from object. The constructor and the properties do the obvious things. Equality between two instances of the same anonymous type is determined in the natural manner, compar- ing each property value in turn using the property type’s Equals method. The hash code generation is similar, calling GetHashCode on each property value in turn and combining the results. The exact method for combining the various hash codes together to form one composite hash is unspecified, and you shouldn’t write code that depends on it anyway—you just need to be confident that two equal instances will return the same hash, and two unequal instances will usually return different hashes. All of this only works if the Equals and GetHashCode implementations of all the differ- ent types involved as properties conform to the normal rules, of course. Because the properties are read-only, all anonymous types are immutable as long as the types used for their properties are immutable. This provides you with all the normal benefits of immutability—being able to pass values to methods without fear of them changing, simple sharing of data across threads, and so forth. VB ANONYMOUS TYPE PROPERTIES ARE MUTABLE BY DEFAULT Anonymous types are also available in Visual Basic 9 onward. But, by default, their properties are mutable; you need to declare any properties you want to be immutable with the Key modifier. Only properties declared as keys are used in hashing and equality comparisons. This is easy to overlook when converting code from one language to another. We’re almost done with anonymous types now. But there’s one slight wrinkle still to talk about—a shortcut for a situation that’s fairly common in LINQ. 8.5.3 Projection initializers The anonymous object initializers you’ve seen so far have all been lists of name/value pairs—Name="Jon", Age=36 and the like. As it happens, I’ve always used constants because they make for smaller examples, but in real code you often want to copy prop- erties from an existing object. Sometimes you’ll want to manipulate the values in some way, but often a straight copy is enough. Again, without LINQ it’s hard to give convincing examples of this, but let’s go back to our Person class and suppose we had a good reason to want to convert a collection of Person instances into a similar collection where each element has just a name and a flag to say whether that person is an adult. Given an appropriate person variable, you could use something like this: new { Name = person.Name, IsAdult = (person.Age >= 18) } That works, and for just a single property the syntax for setting the name (the part in bold) isn’t too clumsy, but if you were copying several properties it would get tiresome. C# 3 provides a shortcut: if you don’t specify the property name, but just the expression to evaluate for the value, it’ll use the last part of the expression as the 229Anonymous types name, provided it’s a simple field or property. This is called a projection initializer. It means you can rewrite the previous code as follows: new { person.Name, IsAdult = (person.Age >= 18) } It’s common for all the bits of an anonymous object initializer to be projection initial- izers—it typically happens when you’re taking some properties from one object and some properties from another, often as part of a join operation. Anyway, I’m getting ahead of myself. The following listing shows the previous code in action, using the List .ConvertAll method and an anonymous method. List family = new List { new Person { Name = "Holly", Age = 36 }, new Person { Name = "Jon", Age = 36 }, new Person { Name = "Tom", Age = 9 }, new Person { Name = "Robin", Age = 6 }, new Person { Name = "William", Age = 6 } }; var converted = family.ConvertAll(delegate(Person person) { return new { person.Name, IsAdult = (person.Age >= 18) }; } ); foreach (var person in converted) { Console.WriteLine("{0} is an adult? {1}", person.Name, person.IsAdult); } In addition to the use of a projection initializer for the Name property, listing 8.6 shows the value of delegate type inference and anonymous methods. Without them, you couldn’t have retained the strong typing of converted, because you wouldn’t have been able to specify what the TOutput type parameter of Converter should be. As it is, you can iterate through the new list and access the Name and IsAdult properties as if you were using any other type. Don’t spend too long thinking about projection initializers at this point—the important thing is to be aware that they exist so you won’t get confused when you see them later. In fact, that advice applies to this entire section on anonymous types, so without going into details, let’s look at why they’re present at all. 8.5.4 What’s the point? I hope you’re not feeling cheated at this point, but I sympathize if you are. Anony- mous types are a fairly complex solution to a problem we haven’t really encountered yet. But I bet you have seen part of the problem before, really. If you’ve ever done any real-life work involving databases, you’ll know that you don’t always want all of the data that’s available on all the rows that match your query Listing 8.6 Transformation from Person to a name and adulthood flag 230 CHAPTER 8 Cutting fluff with a smart compiler criteria. Often it’s not a problem to fetch more than you need, but if you only need 2 columns out of the 50 in the table, you wouldn’t bother to select all 50, would you? The same problem occurs in nondatabase code. Suppose you have a class that reads a log file and produces a sequence of log lines with many fields. Keeping all of the information might be far too memory-intensive if you only care about a couple of fields from the log. LINQ lets you filter that information easily. But what’s the result of that filtering? How can you keep some data and discard the rest? How can you easily keep some derived data that isn’t directly represented in the original form? How can you combine pieces of data that may not initially have been consciously associated, or that may only have a relationship in a particular situation? Effectively, you want a new data type, but manually creating such a type in every situa- tion is tedious, particularly when you have tools such as LINQ available that make the rest of the process so simple. Figure 8.5 shows the three elements that make anony- mous types a powerful feature. If you find yourself creating a type that’s only used in a single method, and that only contains fields and trivial properties, consider whether an anonymous type would be appropriate. I suspect that usually, when you find yourself leaning toward anony- mous types, you could also use LINQ to help you. If you find yourself using the same sequence of properties for the same purpose in several places, though, you might want to consider creating a normal type for the pur- pose, even if it still just contains trivial properties. Anonymous types naturally infect whatever code they’re used in with implicit typing, which is often fine, but can be a nuisance at other times. In particular, it means you can’t easily create a method to return an instance of that type in a strongly typed way. As with the previous features, use anonymous types when they genuinely make the code simpler to work with, not just because they’re new and cool. Avoiding excessive dat accumulation Tailoring data encapsulation to one situation Avoiding manual “turn the handle” coding Anonymous types AAAAvoiding manual “turn the handle” coding AAAAvoiding excessiveAA dat accumulation Anonymous types Tailoring data encapsulation to one situation Avoiding excessive data accumulation Tailoring data encapsulation to one situation Avoiding manual turn-the-handle coding Anonymous types Figure 8.5 Anonymous types allow you to keep just the data you need for a particular situation, in a form that’s tailored to that situation, without the tedium of writing a fresh type each time. 231Summary 8.6 Summary What a seemingly mixed bag of features! You’ve seen four features that are quite simi- lar, at least in syntax: object initializers, collection initializers, implicitly typed arrays, and anonymous types. The other two features—automatic properties and implicitly typed local variables—are somewhat different. Likewise, most of the features would’ve been useful individually in C# 2, whereas implicitly typed arrays and anonymous types only pay back the cost of learning about them when the rest of the C# 3 features are brought into play. So what do these features really have in common? They all relieve the developer of tedious coding. I’m sure you don’t enjoy writing trivial properties any more than I do, or setting several properties, one at a time, using a local variable—particularly when you’re trying to build up a collection of similar objects. Not only do the new features of C# 3 make it easier to write the code, they also make it easier to read it, at least when they’re applied sensibly. In the next chapter, we’ll look at a major new language feature, along with a frame- work feature it provides direct support for. If you thought anonymous methods made creating delegates easy, just wait until you see lambda expressions. 232 Lambda expressions and expression trees In chapter 5 you saw how C# 2 made delegates much easier to use due to implicit con- versions of method groups, anonymous methods, and return type and parameter variance. This is enough to make event subscription significantly simpler and more readable, but delegates in C# 2 are still too bulky to be used all the time; a page of code full of anonymous methods is painful to read, and you wouldn’t want to start putting multiple anonymous methods in a single statement on a regular basis. One of the fundamental building blocks of LINQ is the ability to create pipe- lines of operations, along with any state required by those operations. These opera- tions can express all kinds of logic about data: how to filter it, how to order it, how This chapter covers  Lambda expression syntax  Conversions from lambdas to delegates  Expression tree framework classes  Conversions from lambdas to expression trees  Why expression trees matter  Changes to type inference and overload resolution 233 to join different data sources together, and much more. When LINQ queries are exe- cuted in-process, those operations are usually represented by delegates. Statements containing several delegates are common when manipulating data with LINQ to Objects,1 and lambda expressions in C# 3 make all of this possible without sacri- ficing readability. (While I’m mentioning readability, this chapter uses lambda expres- sion and lambda interchangeably.) IT’S ALL GREEK TO ME The term lambda expression comes from lambda calculus, also written as λ-calculus, where λ is the Greek letter lambda. This is an area of math and computer science dealing with defining and applying functions. It’s been around for a long time and is the basis of functional languages such as ML. The good news is that you don’t need to know lambda calculus to use lambda expressions in C# 3. Executing delegates is only part of the LINQ story. To use databases and other query engines efficiently, you need a different representation of the operations in the pipe- line—a way to treat code as data that can be examined programmatically. The logic within the operations can then be transformed into a different form, such as a web service call, a SQL or LDAP query—whatever’s appropriate. Although it’s possible to build up representations of queries in a particular API, it’s usually tricky to read and it sacrifices a lot of compiler support. This is where lambdas save the day again: not only can they be used to create delegate instances, but the C# compiler can also transform them into expression trees—data structures representing the logic of the lambda expressions—so that other code can examine it. In short, lambda expressions are the idiomatic way of representing the operations in LINQ data pipelines—but we’ll take things one step at a time, examining them in a fairly isolated way before we embrace the whole of LINQ. In this chapter we’ll look at both ways of using lambda expressions, although for the moment our coverage of expression trees will be relatively basic—we won’t create any SQL just yet. With that theory under your belt, you should be relatively comfort- able with lambda expressions and expression trees by the time we hit the really impressive stuff in chapter 12. In the final part of this chapter we’ll examine how type inference has changed for C# 3, mostly due to lambdas with implicit parameter types. This is a bit like learning how to tie shoelaces: far from exciting, but without this ability you’ll trip over yourself when you start running. Let’s begin by seeing what lambda expressions look like. We’ll start with an anony- mous method and gradually transform it into shorter and shorter forms. 1 LINQ to Objects handles sequences of data within the same process. By contrast, providers such as LINQ to SQL offload the work to other out-of-process systems—databases, for example. 234 CHAPTER 9 Lambda expressions and expression trees 9.1 Lambda expressions as delegates In many ways, lambda expressions can be seen as an evolution of anonymous methods from C# 2. Lambda expressions can do almost everything that anonymous methods can, and they’re almost always more readable and compact.2 In particular, the behav- ior of captured variables is exactly the same in lambda expressions as in anonymous methods. In the most explicit form, not much difference exists between the two, but lambda expressions have a lot of shortcuts available that make them compact in com- mon situations. Like anonymous methods, lambda expressions have special conver- sion rules—the type of the expression isn’t a delegate type in itself, but it can be converted into a delegate instance in various ways, both implicitly and explicitly. The term anonymous function covers anonymous methods and lambda expressions, and in many cases the same conversion rules apply to both of them. We’ll start with a simple example, initially expressed as an anonymous method. You’ll create a delegate instance that takes a string parameter and returns an int (which is the length of the string). First you need to choose a delegate type to use; for- tunately, .NET 3.5 comes with a whole family of generic delegate types to help you out. 9.1.1 Preliminaries: Introducing the Func<…> delegate types There are five generic Func delegate types in .NET 3.5’s System namespace. There’s nothing special about Func—it’s just handy to have some predefined generic types that are capable of handling many situations. Each delegate signature takes between zero and four parameters, the types of which are specified as type parameters.3 The last type parameter is used for the return type in each case. Here are the signatures of all the Func delegate types in .NET 3.5: TResult Func() TResult Func(T arg) TResult Func(T1 arg1, T2 arg2) TResult Func(T1 arg1, T2 arg2, T3 arg3) TResult Func(T1 arg1, T2 arg2, T3 arg3, T4 arg4) For example, Func is equivalent to a delegate type of the form public delegate int SomeDelegate(string arg1, double arg2) The Action<...> set of delegates provides the equivalent functionality when you want a void return type. The single parameter form of Action existed in .NET 2.0, but the rest are new to .NET 3.5. If four arguments aren’t enough for you, then .NET 4 has the answer: it expands both the Action<...> and Func<...> families to take up to 16 arguments, so Func has an eye-watering 17 type parameters. This is primarily to help support the Dynamic Language Runtime (DLR) that you’ll meet in chapter 14, and you’re unlikely to need to deal with it directly. 2 The one feature available to anonymous methods but not lambda expressions is the ability to concisely ignore parameters. Look back at section 5.4.3 for more details if you’re interested, but in practice it’s not something you’ll really miss with lambda expressions. 3 You may remember that you met the version without any parameters (but one type parameter) in chapter 6. 235Lambda expressions as delegates For this example, you need a type that takes a string parameter and returns an int, so you can use Func. 9.1.2 First transformation to a lambda expression Now that you know the delegate type, you can use an anonymous method to create your delegate instance. The following listing shows this and executes the delegate instance afterward, so you can see it working. Func returnLength; returnLength = delegate (string text) { return text.Length; }; Console.WriteLine(returnLength("Hello")); Listing 9.1 prints 5, just as you’d expect it to. I’ve separated the declaration of return- Length from the assignment to it, to keep it on one line—it’s easier to keep track of that way. The anonymous method expression is the part in bold; that’s the part you’ll convert into a lambda expression. The most long-winded form of a lambda expression is this: (explicitly-typed-parameter-list) => { statements } The => part is new to C# 3 and tells the compiler that you’re using a lambda expres- sion. Most of the time, lambda expressions are used with a delegate type that has a nonvoid return type, and the syntax is slightly less intuitive when there isn’t a result. This is another indication of the changes in idiom between C# 1 and C# 3. In C# 1, delegates were usually used for events and rarely returned anything. In LINQ they’re usually used as part of a data pipeline, taking input and returning a result to say what the projected value is, or whether the item matches the current filter, and so forth. With the explicit parameters and statements in braces, this version looks very simi- lar to an anonymous method. The following listing is equivalent to listing 9.1, but it uses a lambda expression. Func returnLength; returnLength = (string text) => { return text.Length; }; Console.WriteLine(returnLength("Hello")); Again, I’ve used bold to indicate the expression used to create the delegate instance. When reading lambda expressions, it helps to think of the => part as “goes to,” so the example in listing 9.2 could be read “text goes to text.Length.” Since this is the only part of the listing that’s interesting for a while, I’ll show it alone from now on. You can replace the bold text from listing 9.2 with any of the lambda expressions listed in this section and the result will be the same. Listing 9.1 Using an anonymous method to create a delegate instance Listing 9.2 A long-winded first lambda expression, similar to an anonymous method 236 CHAPTER 9 Lambda expressions and expression trees The same rules that govern return statements in anonymous methods apply to lambdas: you can’t try to return a value from a lambda expression with a void return type, whereas if there’s a nonvoid return type, every code path has to return a compat- ible value.4 It’s all pretty intuitive and rarely gets in the way. So far, we haven’t saved much space or made things particularly easy to read. Let’s start applying the shortcuts. 9.1.3 Using a single expression as the body The form we’ve looked at so far uses a full block of code to return the value. This is flexible—you can have multiple statements, perform loops, return from different places in the block, and so on, just as with anonymous methods. But most of the time, you can easily express the whole of the body in a single expression, the value of which is the result of the lambda.5 In these cases, you can specify just that expression, with- out any braces, return statements, or semicolons. The format then is (explicitly-typed-parameter-list)=>expression In our example, this means that the lambda expression becomes (string text) => text.Length That’s starting to look simpler already. Now, what about that parameter type? The compiler already knows that instances of Func take a single string parameter, so you should be able to just name that parameter. 9.1.4 Implicitly typed parameter lists Most of the time, the compiler can guess the parameter types without you explicitly stating them. In these cases, you can write the lambda expression as (implicitly-typed-parameter-list)=>expression An implicitly typed parameter list is just a comma-separated list of names, without the types. You can’t mix and match for different parameters—either the whole list is explicitly typed, or it’s all implicitly typed. Also, if any of the parameters are out or ref parameters, you’re forced to use explicit typing. In our example, it’s fine, so your lambda expression is just (text) => text.Length That’s getting pretty short now. There’s not a lot more you could get rid of. The parentheses seem redundant, though. 4 Code paths throwing exceptions don’t need to return a value, of course, and neither do detectable infinite loops. 5 You can still use this syntax for a delegate with a void return type if you only need one statement. You omit the semicolon and the braces, basically. 237Lambda expressions as delegates 9.1.5 Shortcut for a single parameter When the lambda expression only needs a single parameter, and that parameter can be implicitly typed, C# 3 allows you to omit the parentheses, so it now has this form: parameter-name => expression The final form of your lambda expression is therefore text => text.Length You may be wondering why there are so many special cases with lambda expres- sions—none of the rest of the language cares whether a method has one parameter or more, for instance. Well, what sounds like a very specific case actually turns out to be extremely common, and the improvement in readability from removing the paren- theses from the parameter list can be significant when there are many lambdas in a short piece of code. It’s worth noting that you can put parentheses around the whole lambda expres- sion if you want to, just like other expressions. Occasionally this helps readability, such as when you’re assigning the lambda to a variable or property—otherwise, the equals symbols can get confusing, at least to start with. Most of the time it’s perfectly readable without any extra syntax at all. The following listing shows this in the context of our original code. Func returnLength; returnLength = text => text.Length; Console.WriteLine(returnLength("Hello")); At first you may find listing 9.3 confusing to read, in the same way that anonymous methods appear strange to many developers until they get used to them. In normal use, you’d declare the variable and assign the value to it in the same expression, making it even clearer. When you are used to lambda expressions, though, you can appreciate how concise they are. It’d be hard to imagine a shorter, clearer way of creating a dele- gate instance.6 You could change the variable name text to something like x, and in full LINQ that’s often useful, but longer names give valuable information to the reader. I’ve shown this transformation over the course of a few pages, but figure 9.1 makes it clear just how much extraneous syntax you’ve saved. The decision of whether to use the short form for the body of the lambda expres- sion, specifying just an expression instead of a whole block, is completely independent from the decision about whether to use explicit or implicit parameters. This example has taken us down one route of shortening the lambda, but we could’ve started off by Listing 9.3 A concise lambda expression 6 That’s not to say it’s impossible. Some languages allow closures to be represented as simple blocks of code with a magic variable name to represent the common case of a single parameter. 238 CHAPTER 9 Lambda expressions and expression trees making the parameters implicit. When you’re comfortable with lambda expressions, you won’t think about this at all—you’ll just write the shortest available form naturally. HIGHER-ORDER FUNCTIONS The body of a lambda expression can itself contain a lambda expression, and this tends to be as confusing as it sounds. Alterna- tively, the parameter to a lambda expression can be another delegate, which is just as bad. Both of these are examples of higher-order functions. If you enjoy feeling dazed and confused, have a look at some of the downloadable source code. Although I’m being flippant, this approach is common in functional programming and can be useful. It just takes a certain degree of perseverance to get into the right mindset. So far we’ve only dealt with a single lambda expression, putting it into different forms. Let’s look at a few examples to make things more concrete before we examine the details. 9.2 Simple examples using List and events When we look at extension methods in chapter 10, we’ll use lambda expressions all the time. Until then, List and event handlers give us the best examples. We’ll start off with lists, using automatically implemented properties, implicitly typed local vari- ables, and collection initializers for the sake of brevity. We’ll then call methods that take delegate parameters, using lambda expressions to create the delegates, of course. 9.2.1 Filtering, sorting, and actions on lists Remember the FindAll method on List—it takes a Predicate and returns a new list with all the elements from the original list that match the predicate. The Sort method takes a Comparison and sorts the list accordingly. Finally, the ForEach method takes an Action to perform on each element. delegate(String text) { return text.Length; } (string text) => { return text.Length; } (string text) => text.Length (text) => text.Length text => text.Length Convert to lambda expression Single expression, so no braces required Let the compiler infer the parameter type Remove unnecessary parentheses Start with an anonymous method Figure 9.1 Lambda syntax shortcuts 239Simple examples using List and events Listing 9.4 uses lambda expressions to provide the delegate instance to each of these methods. The sample data in question is the name and year of release for various films. You print out the original list, then create and print out a filtered list of only old films, and then sort and print out the original list ordered by name. (It’s interesting to consider how much more code would’ve been required to do the same thing in C# 1.) class Film { public string Name { get; set; } public int Year { get; set; } } ... var films = new List { new Film { Name = "Jaws", Year = 1975 }, new Film { Name = "Singing in the Rain", Year = 1952 }, new Film { Name = "Some like it Hot", Year = 1959 }, new Film { Name = "The Wizard of Oz", Year = 1939 }, new Film { Name = "It's a Wonderful Life", Year = 1946 }, new Film { Name = "American Beauty", Year = 1999 }, new Film { Name = "High Fidelity", Year = 2000 }, new Film { Name = "The Usual Suspects", Year = 1995 } }; Action print = film => Console.WriteLine("Name={0}, Year={1}", film.Name, film.Year); films.ForEach(print); films.FindAll(film => film.Year < 1960) .ForEach(print); films.Sort((f1, f2) => f1.Name.CompareTo(f2.Name)); films.ForEach(print); The first half of listing 9.4 involves setting up the data. This code uses a named type just to make life easier—an anonymous type would’ve meant a few more hoops to jump through in this particular case. Before you use the newly created list, you create a delegate instance B, which you’ll use to print out the items of the list. You use this delegate instance three times, which is why I created a variable to hold it rather than using a separate lambda expression each time. It just prints a single element, but by passing it into List.ForEach you can dump the whole list to the console. A subtle but important point is that the semicolon at the end of this statement is part of the assignment state- ment, not part of the lambda expression. If you were using the same lambda expres- sion as an argument in a method call, there wouldn’t be a semicolon directly after Console.WriteLine(...). Listing 9.4 Manipulating a list of films using lambda expressions Creates reusable list-printing delegate B Prints original listC Creates filtered listD Sorts original listE 240 CHAPTER 9 Lambda expressions and expression trees The first list you print out is just the original one without any modifications C. You then find all the films in your list that were made before 1960 and print those out D. This is done with another lambda expression, which is executed for each film in the list—it only has to determine whether a single film should be included in the filtered list. The source code uses the lambda expression as a method argument, but really the compiler has created a method like this: private static bool SomeAutoGeneratedName(Film film) { return film.Year < 1960; } The method call to FindAll is then effectively this: films.FindAll(new Predicate(SomeAutoGeneratedName)) The lambda expression support here is just like the anonymous method support in C# 2; it’s all cleverness on the part of the compiler. (In fact, the Microsoft compiler is even smarter in this case—it realizes it can get away with reusing the delegate instance if the code is ever called again, so it caches it.) Sorting the list is also achieved using a lambda expression E, which compares any two films using their names. I have to confess that explicitly calling CompareTo yourself is a bit ugly. In the next chapter you’ll see how the OrderBy extension method allows you to express ordering in a neater way. Let’s look at another example, this time using lambda expressions with event handling. 9.2.2 Logging in an event handler If you think back to chapter 5, in section 5.9 you saw an easy way of using anonymous methods to log which events were occurring, but you could only use a compact syntax because you didn’t mind losing the parameter information. What if you wanted to log both the nature of the event and information about its sender and arguments? Lambda expressions enable this in a neat way, as shown in the following listing. static void Log(string title, object sender, EventArgs e) { Console.WriteLine("Event: {0}", title); Console.WriteLine(" Sender: {0}", sender); Console.WriteLine(" Arguments: {0}", e.GetType()); foreach (PropertyDescriptor prop in TypeDescriptor.GetProperties(e)) { string name = prop.DisplayName; object value = prop.GetValue(e); Console.WriteLine(" {0}={1}", name, value); } } Listing 9.5 Logging events using lambda expressions 241Expression trees ... Button button = new Button { Text = "Click me" }; button.Click += (src, e) => Log("Click", src, e); button.KeyPress += (src, e) => Log("KeyPress", src, e); button.MouseClick += (src, e) => Log("MouseClick", src, e); Form form = new Form { AutoSize = true, Controls = { button }}; Application.Run(form); Listing 9.5 uses lambda expressions to pass the event name and parameters to the Log method, which logs details of the event. You don’t log the details of the event source, beyond whatever its ToString override returns, because an overwhelming amount of information is associated with controls. But you use reflection over property descrip- tors to show the details of the EventArgs instance passed to you. Here’s some sample output from when you click the button: Event: Click Sender: System.Windows.Forms.Button, Text: Click me Arguments: System.Windows.Forms.MouseEventArgs Button=Left Clicks=1 X=53 Y=17 Delta=0 Location={X=53,Y=17} Event: MouseClick Sender: System.Windows.Forms.Button, Text: Click me Arguments: System.Windows.Forms.MouseEventArgs Button=Left Clicks=1 X=53 Y=17 Delta=0 Location={X=53,Y=17} All of this is possible without lambda expressions, of course, but it’s a lot neater than it would’ve been otherwise. Now that you’ve seen lambdas being converted into delegate instances, it’s time to look at expression trees, which represent lambda expressions as data instead of code. 9.3 Expression trees The idea of code as data is an old one, but it hasn’t been used much in popular pro- gramming languages. You could argue that all .NET programs use the concept, because the IL code is treated as data by the JIT, which then converts it into native code to run on your CPU. That’s deeply hidden though, and although libraries that manipulate IL programmatically exist, they’re not widely used. Expression trees in .NET 3.5 provide an abstract way of representing some code as a tree of objects. It’s like CodeDOM but operating at a slightly higher level. The pri- mary use of expression trees is in LINQ, and later in this section you’ll see how crucial expression trees are to the whole LINQ story. 242 CHAPTER 9 Lambda expressions and expression trees C# 3 provides built-in support for converting lambda expressions to expression trees, but before we cover that, let’s explore how they fit into the .NET Framework without using any compiler tricks. 9.3.1 Building expression trees programmatically Expression trees aren’t as mystical as they sound, although some of the uses they’re put to look like magic. As the name suggests, they’re trees of objects, where each node in the tree is an expression in itself. Different types of expressions represent the differ- ent operations that can be performed in code: binary operations, such as addition; unary operations, such as taking the length of an array; method calls; constructor calls; and so forth. The System.Linq.Expressions namespace contains the various classes that repre- sent expressions. All of them derive from the Expression class, which is abstract and mostly consists of static factory methods to create instances of other expression classes. It exposes two properties, though:  The Type property represents the .NET type of the evaluated expression—you can think of it like a return type. The type of an expression that fetches the Length property of a string would be int, for example.  The NodeType property returns the kind of expression represented as a member of the ExpressionType enumeration, with values such as LessThan, Multiply, and Invoke. To use the same example, in myString.Length the property access part would have a node type of MemberAccess. There are many classes derived from Expression, and some of them can have many different node types. BinaryExpression, for instance, represents any operation with two operands: arithmetic, logic, comparisons, array indexing, and the like. This is where the NodeType property is important, as it distinguishes between different kinds of expressions that are represented by the same class. I don’t intend to cover every expression class or node type—there are far too many, and MSDN does a perfectly good job of explaining them (see http://mng.bz/3vW3). Instead, we’ll try to get a general feel for what you can do with expression trees. Let’s start off by creating one of the simplest possible expression trees, adding two constant integers together. The following listing creates an expression tree to repre- sent 2+3. Expression firstArg = Expression.Constant(2); Expression secondArg = Expression.Constant(3); Expression add = Expression.Add(firstArg, secondArg); Console.WriteLine(add); Listing 9.6 A simple expression tree, adding 2 and 3 243Expression trees Running listing 9.6 will produce the output (2 + 3), which demonstrates that the various expression classes override ToString to produce human-readable output. Figure 9.2 depicts the tree generated by the code. It’s worth noting that the leaf expressions are created first in the code: you build expressions from the bottom up. This is enforced by the fact that expressions are immutable—once you’ve created an expression, it’ll never change, so you can cache and reuse expressions at will. Now that you’ve built up an expression tree, it’s time to execute it. 9.3.2 Compiling expression trees into delegates One of the types derived from Expression is LambdaExpression. The generic class Expression then derives from LambdaExpression. It’s all slightly confus- ing—figure 9.3 shows the type hierarchy to make things clearer. The difference between Expression and Expression is that the generic class is statically typed to indicate what kind of expression it is, in terms of return type and parameters. Obviously, this is expressed by the TDelegate type param- eter, which must be a delegate type. For instance, the simple addition expression takes no parameters and returns an integer—this is matched by the signature of Func, so you could use an Expression> to represent the expression in a statically typed manner. You do this using the Expression.Lambda method, which has a num- ber of overloads. The examples we’ve looked at use the generic method, which uses a firstArg ConstantExpression NodeType=Constant Type=System.Int32 Value=2 secondArg ConstantExpression NodeType=Constant Type=System.Int32 Value=3 BinaryExpression NodeType=Add Type=System.Int32 Left Right add Figure 9.2 Graphical representation of the expression tree created by listing 9.6 Expression LambdaExpression Expression BinaryExpression (Other types) Figure 9.3 Type hierarchy from Expression up to Expression 244 CHAPTER 9 Lambda expressions and expression trees type parameter to indicate the type of delegate we wanted to represent. See MSDN for alternatives. So, what’s the point of doing this? Well, LambdaExpression has a Compile method that creates a delegate of the appropriate type; Expression has another method by the same name, but statically typed to return a delegate of type TDelegate. This delegate can now be executed in the normal manner, as if it had been created using a normal method or any other means. The following listing shows this in action, with the same expression as before. Expression firstArg = Expression.Constant(2); Expression secondArg = Expression.Constant(3); Expression add = Expression.Add(firstArg, secondArg); Func compiled = Expression.Lambda>(add).Compile(); Console.WriteLine(compiled()); Arguably, listing 9.7 is one of the most convoluted ways of printing out 5 that you could ask for. At the same time, it’s also rather impressive. You’re programmatically creating some logical blocks and representing them as normal objects, and then ask- ing the framework to compile the whole thing into real code that can be executed. You may never need to use expression trees this way, or even build them up program- matically at all, but it’s useful background information that will help you understand how LINQ works. As I said at the beginning of this section, expression trees aren’t too far removed from CodeDOM—Snippy compiles and executes C# code that’s been entered as plain text, for instance. But two significant differences exist between CodeDOM and expres- sion trees. First, in .NET 3.5, expression trees were only able to represent single expressions. They weren’t designed for whole classes, methods, or even just statements. This has changed somewhat in .NET 4, where they’re used to support dynamic typing—you can now create blocks, assign values to variables, and so on. But there are still significant restrictions compared with CodeDOM. Second, C# supports expression trees directly in the language, through lambda expressions. Let’s take a look at that now. 9.3.3 Converting C# lambda expressions to expression trees As you’ve already seen, lambda expressions can be converted to appropriate delegate instances, either implicitly or explicitly. That’s not the only conversion that’s available. You can also ask the compiler to build an expression tree from your lambda expres- sion, creating an instance of Expression at execution time. For example, the following listing shows a much shorter way of creating the “return 5” expression, compiling it, and then invoking the resulting delegate. Listing 9.7 Compiling and executing an expression tree 245Expression trees Expression> return5 = () => 5; Func compiled = return5.Compile(); Console.WriteLine(compiled()); In the first line of listing 9.8, the () => 5 part is the lambda expression. You don’t need any casts because the compiler can verify everything as it goes. You could’ve writ- ten 2+3 instead of 5, but the compiler would’ve optimized the addition away for you. The important point to take away is that the lambda expression has been converted into an expression tree. THERE ARE LIMITATIONS Not all lambda expressions can be converted to expression trees. You can’t convert a lambda with a block of statements (even just one return statement) into an expression tree—it has to be in the form that evaluates a single expression, and that expression can’t contain assign- ments. This restriction applies even in .NET 4 with its extended abilities for expression trees. Although these are the most common restrictions, they’re not the only ones—the full list isn’t worth describing here, as this issue comes up so rarely. If there’s a problem with an attempted conversion, you’ll find out at compile time. Let’s look at a more complicated example to see how things work, particularly with respect to parameters. This time you’ll write a predicate that takes two strings and checks to see if the first one begins with the second. The code is simple when written as a lambda expression. Expression> expression = (x, y) => x.StartsWith(y); var compiled = expression.Compile(); Console.WriteLine(compiled("First", "Second")); Console.WriteLine(compiled("First", "Fir")); The expression tree itself is more complicated, especially by the time you’ve converted it into an instance of LambdaExpression. The next listing shows how it could be built in code. MethodInfo method = typeof(string).GetMethod ("StartsWith", new[] { typeof(string) }); var target = Expression.Parameter(typeof(string), "x"); var methodArg = Expression.Parameter(typeof(string), "y"); Expression[] methodArgs = new[] { methodArg }; Expression call = Expression.Call(target, method, methodArgs); Listing 9.8 Using lambda expressions to create expression trees Listing 9.9 Demonstration of a more complicated expression tree Listing 9.10 Building a method call expression tree in code Builds up parts of method call B CCreates CallExpression from parts 246 CHAPTER 9 Lambda expressions and expression trees var lambdaParameters = new[] { target, methodArg }; var lambda = Expression.Lambda> (call, lambdaParameters); var compiled = lambda.Compile(); Console.WriteLine(compiled("First", "Second")); Console.WriteLine(compiled("First", "Fir")); As you can see, listing 9.10 is considerably more involved than the version with the C# lambda expression. But it does make it more obvious exactly what’s involved in the tree and how parameters are bound. You start off by working out everything you need to know about the method call that forms the body of the final expression B: the target of the method (the string you’re calling StartsWith on); the method itself (as a MethodInfo); and the list of arguments (in this case, just the one). It so happens that your method target and argu- ment will both be parameters passed into the expression, but they could be other types of expressions—constants, the results of other method calls, property evalua- tions, and so forth. After building the method call as an expression C, you then need to convert it into a lambda expression D, binding the parameters as you go. You reuse the same ParameterExpression values you created as information for the method call: the order in which they’re specified when creating the lambda expression is the order in which they’ll be picked up when you eventually call the delegate. Figure 9.4 shows the same final expression tree graphically. To be picky, even though it’s still called an expression tree, the fact that you reuse the parameter expres- sions (and you have to—creating a new one with the same name and attempting to bind parameters that way causes an exception at execution time) means that it’s not really a tree in the purest sense. Glancing at the complexity of figure 9.4 and listing 9.10 without trying to look at the details, you’d be forgiven for thinking that you were doing something really com- plicated, when in fact it’s just a single method call. Imagine what the expression tree for a genuinely complex expression would look like—and then be grateful that C# 3 can create expression trees from lambda expressions! For one final way of looking at the same idea, Visual Studio 2010 and 2012 provide a built-in visualizer for expression trees.7 This can be useful if you’re trying to work out how to build up an expression tree in code, and you want to get an idea of what it should look like; write a lambda expression that does what you want with some dummy data, look at the visualization in the debugger, and then work out how to build similar trees with the information you have in your real code. The visualizer relies on changes within .NET 4, so it won’t work with projects targeting .NET 3.5. Fig- ure 9.5 shows the visualization for the StartsWith example. 7 If you’re using Visual Studio 2008, you can download some sample code from MSDN to build a similar visu- alizer (see http://mng.bz/g6xd), but obviously it’s easier to use the one shipped with Visual Studio if you have a recent enough version. Converts call into LambdaExpression D 247Expression trees method MethodInfo for string.StartsWith(string) target ParameterExpression NodeType=Parameter Type=System.String Name="x" call MethodCallExpression NodeType=Call Type=System.Boolean Method Object methodArgs Collection of Expressions Arguments methodArg ParameterExpression NodeType=Parameter Type=System.String Name="y" lambda Expression NodeType=Lambda Type=System.Boolean Body lambdaParameters Collection of ParameterExpressions Parameters (Contains) (Contains) (Contains) Figure 9.4 Graphical representation of an expression tree that calls a method and uses parameters from a lambda expression Figure 9.5 Debugger visualization of an expression tree 248 CHAPTER 9 Lambda expressions and expression trees The .Lambda and .Call parts of the visualization correspond to your calls to Expression.Lambda and Expression.Call; $x and $y correspond to the parameter expressions. The visualization is the same whether the expression tree has been built up explicitly through code or using a lambda expression conversion. One small point to note is that although the C# compiler builds expression trees in the compiled code using code similar to listing 9.10, it has one shortcut up its sleeve: it doesn’t need to use normal reflection to get the MethodInfo for string.StartsWith. Instead, it uses the method equivalent of the typeof operator. This is only available in IL, not in C# itself, and the same operator is used to create delegate instances from method groups. Now that you’ve seen how expression trees and lambda expressions are linked, let’s take a brief look at why they’re so useful. 9.3.4 Expression trees at the heart of LINQ Without lambda expressions, expression trees would have relatively little value. They’d be an alternative to CodeDOM in cases where you only wanted to model a sin- gle expression instead of whole statements, methods, types, and so forth, but the ben- efit would still be limited. The reverse is also true to a limited extent: without expression trees, lambda expressions would certainly be less useful. Having a more compact way of creating del- egate instances would still be welcome, and the shift toward a more functional mode of development would still be viable. Lambda expressions are particularly effective when combined with extension methods, as you’ll see in the next chapter, but with expression trees in the picture as well, things get a lot more interesting. What do you get when you combine lambda expressions, expression trees, and extension methods? The answer is, “the language side of LINQ,” pretty much. The extra syntax you’ll see in chapter 11 is icing on the cake, but the story would still have been compelling with just those three ingredients. For a long time you could have either nice compile-time checking or the ability to tell another platform to run some code, usually expressed as text (SQL queries being the most obvious example). But you couldn’t do both at the same time. By combining lambda expressions that provide compile-time checks with expres- sion trees that abstract the execution model away from the desired logic, you can have the best of both worlds, within reason. At the heart of out-of-process LINQ providers is the idea that you can produce an expression tree from a familiar source language (C#, in this case) and use the result as an intermediate format that can then be converted into the native language of the target platform—SQL, for example. In some cases, there may not be a simple native language so much as a native API, making different web service calls depending on what the expression represents, perhaps. Figure 9.6 shows the different paths of LINQ to Objects and LINQ to SQL. In some cases, the conversion may try to perform all the logic on the target plat- form, whereas other cases may use the compilation facilities of expression trees to 249Expression trees execute some of the expression locally and some elsewhere. We’ll look at some of the details of this conversion step in chapter 12, but you should bear this end goal in mind as we explore extension methods and LINQ syntax in chapters 10 and 11. NOT ALL CHECKING CAN BE DONE BY THE COMPILER When expression trees are examined by some sort of converter, some cases generally have to be rejected. For instance, although it’s possible to convert a call to string.StartsWith into a similar SQL expression, a call to string.IsInterned doesn’t make sense in a database environment. Expression trees allow a large amount of compile-time safety, but the compiler can only check that the lambda expres- sion can be converted into a valid expression tree; it can’t make sure that the expression tree will be suitable for its eventual use. Although the most common uses of expression trees are related to LINQ, that’s not always the case… 9.3.5 Expression trees beyond LINQ Bjarne Stroustrup once said, “I wouldn’t like to build a tool that could only do what I had been able to imagine for it.” Though expression trees were introduced into .NET primarily for LINQ, both the community and Microsoft have found other uses for them since then. This section is far from comprehensive, but it might give you a few ideas of where expression trees might help you. Query results C# compiler LINQ to SQL provider Executed at database and fetched back Query results C# compiler Delegate code executed directly in the CLR LINQ to Objects LINQ to SQL Compile time Execution time C# compiler C# query code with lambda expressions C# compiler C# query code with lambda expressions IL using expression trees E t d t Dynamic SQL IL using delegates Figure 9.6 Both LINQ to Objects and LINQ to SQL start with C# code and end with query results. The ability to execute the code remotely comes through expression trees. 250 CHAPTER 9 Lambda expressions and expression trees OPTIMIZING THE DYNAMIC LANGUAGE RUNTIME We’ll see a lot more of the Dynamic Language Runtime (DLR) in chapter 14, when we talk about dynamic typing in C#, but expression trees are a core part of the architec- ture. They have three properties that make them attractive to the DLR:  They’re immutable, so you can cache them safely.  They’re composable, so you can build complex behavior out of simple building blocks.  They can be compiled into delegates that are JIT-compiled into native code as normal. The DLR has to make decisions about how to handle various expressions where the meaning can change subtly based on different rules. Expression trees allow these rules (and the results) to be transformed into code that’s close to what you’d write by hand if you knew all the rules and results you’d seen so far. It’s a powerful concept, and one that allows dynamic code to execute surprisingly quickly. REFACTOR-PROOF REFERENCES TO MEMBERS In section 9.3.3 I mentioned that the compiler can emit references to MethodInfo val- ues much like the typeof operator can. Unfortunately, C# doesn’t have the same abil- ity, which means the only way of telling one piece of general-purpose, reflection-based code to “use the property called BirthDate defined in my type” has previously been to use a string literal and make sure that if you change the name of the property, you also change the literal. Using C# 3, you can build an expression tree representing a prop- erty reference using a lambda expression. The method can then dissect the expres- sion tree, work out the property you mean, and do whatever it likes with the information. It can also compile the expression tree into a delegate and use it directly, of course. As an example of how this might be used, you could write this: serializationContext.AddProperty(x => x.BirthDate); The serialization context would then know that you wanted to serialize the BirthDate property, and it could record appropriate metadata and retrieve the value. (Serializa- tion is just one area where you may want a property or method reference; it’s fairly common within reflection-driven code.) If you refactor the BirthDate property to call it DateOfBirth, the lambda expression will change too. Of course, it’s not foolproof— there’s no compile-time check that the expression really evaluates a simple property; that has to be an execution-time check in the AddProperty code. It’s possible that one day C# will gain the ability to do this within the language itself. Such an operator has already been named: infoof, pronounced either “info-of” or “in-foof,” depending on your level of light-heartedness. This has been on the C# team’s possible-feature list for a while, and unsurprisingly Eric Lippert has blogged about it (see http://mng.bz/24y7), but it hasn’t made the cut yet. Maybe in C# 6. 251Changes to type inference and overload resolution SIMPLER REFLECTION The final use I want to mention before we delve into the murky depths of type infer- ence is also about reflection. As I mentioned in chapter 3, arithmetic operators don’t play nicely with generics, which makes it hard to write generic code to (say) add up a series of values. Marc Gravell used expression trees to great effect to provide a generic Operator class and a nongeneric helper class, allowing you to write code such as this: T runningTotal = initialValue; foreach (T item in values) { runningTotal = Operator.Add(runningTotal, item); } This will even work in cases where the values are a different type than the running total—adding a whole sequence of TimeSpan values to a DateTime, for example. It’s possible to do this in C# 2, but it’s significantly more fiddly due to the ways that opera- tors are exposed via reflection, particularly for the primitive types. Expression trees allow the implementation of this magic to be quite clean, and the fact that they’re compiled to normal IL, which is then JIT-compiled, gives great performance. These are just some examples, and no doubt there are many developers busy work- ing on completely different uses. But they mark an end to our direct coverage of lambda expressions and expression trees. You’ll see a good deal more of them when we look at LINQ, but before we go any further, there are a few changes to C# that need some explanation. These are changes to type inference and how the compiler selects between overloaded methods. 9.4 Changes to type inference and overload resolution The steps involved in type inference and overload resolution were altered in C# 3 to accommodate lambda expressions and to make anonymous methods more useful. This may not count as a new feature of C# as such, but it can be important to under- stand what the compiler is going to do. If you find details like this tedious and irrele- vant, feel free to skip to the chapter summary, but remember that this section exists, so you can read it if you run across a compilation error related to this topic and can’t understand why your code doesn’t work. (Alternatively, you might want to come back to this section if you find your code does compile, but you don’t think it should!) Even within this section, I won’t go into every nook and cranny—that’s what the language specification is for; the details are in the C# 5 specification, section 7.5.2 (“Type inference”). Instead, I’ll give an overview of the new behavior, providing exam- ples of common cases. The primary reason for changing the specification is to allow lambda expressions to work in a concise fashion, which is why I’ve included the topic in this particular chapter. Let’s first look a little deeper at what problems you’d have run into if the C# team had stuck with the old rules. 252 CHAPTER 9 Lambda expressions and expression trees 9.4.1 Reasons for change: streamlining generic method calls Type inference occurs in a few situations. You’ve already seen it apply to implicitly typed arrays, and it’s also required when you try to implicitly convert a method group to a delegate type. This can be particularly confusing when the conversion occurs when you’re using a method group as an argument to another method. With over- loading of the method being called, and overloading of methods within the method group, and the possibility of generic methods getting involved, the set of potential conversions may be enormous. By far the most common situation for type inference is when you’re calling a generic method without specifying any type arguments. This happens all the time in LINQ—the way that query expressions work depends heavily on this ability. It’s all han- dled so smoothly that it’s easy to ignore how much the compiler has to work out on your behalf, all for the sake of making your code clearer and more concise. The rules were reasonably straightforward in C# 2, although method groups and anonymous methods weren’t always handled as well as you might’ve liked. The type inference process didn’t deduce any information from them, leading to situations where the desired behavior was obvious to developers but not to the compiler. Life is more complicated in C# 3 due to lambda expressions. If you call a generic method using a lambda expression with an implicitly typed parameter list, the compiler needs to work out what types you’re talking about before it can check the lambda expres- sion’s body. This is much easier to see in code than in words. The following listing gives an example of the kind of issue I’m referring to: calling a generic method using a lambda expression. static void PrintConvertedValue (TInput input, Converter converter) { Console.WriteLine(converter(input)); } ... PrintConvertedValue("I'm a string", x => x.Length); The PrintConvertedValue method in listing 9.11 simply takes an input value and a delegate that can convert that value into a different type. It’s completely generic—it makes no assumptions about the type parameters TInput and TOutput. Now, look at the types of the arguments you’re calling it with in the bottom line of the listing. The first argument is clearly a string, but what about the second? It’s a lambda expression, so you need to convert it into a Converter, and that means you need to know the types of TInput and TOutput. Remember from section 3.3.2 that the type inference rules of C# 2 were applied to each argument individually, with no way of using the types inferred from one argu- ment to another. In this case, these rules would’ve stopped you from finding the types Listing 9.11 Example of code requiring the new type inference rules 253Changes to type inference and overload resolution of TInput and TOutput for the second argument, so the code in listing 9.11 would’ve failed to compile. Our eventual goal is to understand what makes listing 9.11 compile in C# 3 (and it does, I promise you), but we’ll start with something more modest. 9.4.2 Inferred return types of anonymous functions The following listing shows another example of some code that looks like it should compile but doesn’t under the type inference rules of C# 2. delegate T MyFunc(); static void WriteResult(MyFunc function) { Console.WriteLine(function()); } ... WriteResult(delegate { return 5; }); Compiling listing 9.12 under C# 2 gives an error like this: error CS0411: The type arguments for method 'Snippet.WriteResult(Snippet.MyFunc)' cannot be inferred from the usage. Try specifying the type arguments explicitly. You can fix the error in two ways—either specify the type argument explicitly (as sug- gested by the compiler) or cast the anonymous method to a concrete delegate type: WriteResult(delegate { return 5; }); WriteResult((MyFunc)delegate { return 5; }); Both of these work, but they’re ugly. You might like the compiler to perform the same kind of type inference as for nondelegate types, using the type of the returned expres- sion to infer the type of T. That’s exactly what C# 3 does for both anonymous methods and lambda expressions, but there’s one catch. Although in many cases only one return statement is involved, there can sometimes be more. The following listing is a slightly modified version of listing 9.12, where the anony- mous method sometimes returns an integer and sometimes returns an object. delegate T MyFunc(); static void WriteResult(MyFunc function) { Console.WriteLine(function()); } ... WriteResult(delegate { if (DateTime.Now.Hour < 12) Listing 9.12 Attempting to infer the return type of an anonymous method Listing 9.13 Code returning an integer or an object depending on the time of day Declares delegate type: Func isn’t in .NET 2.0 Declares generic method with delegate parameter Requires type inference for T 254 CHAPTER 9 Lambda expressions and expression trees { return 10; } else { return new object(); } }); The compiler uses the same logic to determine the return type in this situation as it does for implicitly typed arrays, as described in section 8.4. It forms a set of all the types from the return statements in the body of the anonymous function8 (in this case, int and object) and checks to see if exactly one of the types can be implicitly con- verted to from all the others. There’s an implicit conversion from int to object (via boxing) but not from object to int, so the inference succeeds with object as the inferred return type. If there are no types matching that criterion (or more than one), no return type can be inferred and you’ll get a compilation error. Now you know how to work out the return type of an anonymous function, but what about lambda expressions where the parameter types can be implicitly defined? 9.4.3 Two-phase type inference The details of type inference in C# 3 are much more complicated than they are for C# 2. It’s rare that you’ll need to reference the specification for the exact behavior, but if you do, I recommend you write down all the type parameters, arguments, and so forth on a piece of paper, and then follow the specification step by step, carefully not- ing down every action it requires. You’ll end up with a sheet full of fixed and unfixed type variables, with a different set of bounds for each of them. A fixed type variable is one that the compiler has decided the value of; otherwise it’s unfixed. A bound is a piece of information about a type variable. In addition to a bunch of notes, I suspect you’ll get a headache; this stuff isn’t pretty. I’ll present a more fuzzy way of thinking about type inference—one that’s likely to serve just as well as knowing the specification and that will be a lot easier to under- stand. The fact is, if the compiler doesn’t perform type inference in exactly the way you want it to, it’ll almost certainly result in a compilation error rather than code that builds but doesn’t behave properly. If your code doesn’t build, try giving the compiler more information—it’s as simple as that. But here’s roughly what’s changed for C# 3. The first big difference is that the method arguments work as a team in C# 3. In C# 2, every argument was used to try to pin down some type parameters exactly, and the compiler would complain if any two arguments came up with different results for a particular type parameter, even if they were compatible. In C# 3, arguments can 8 Returned expressions that don’t have a type, such as null or another lambda expression, aren’t included in this set. Their validity is checked later, once a return type has been determined, but they don’t contribute to that decision. Return type is int Return type is object 255Changes to type inference and overload resolution contribute pieces of information—types that must be implicitly convertible to the final fixed value of a particular type variable. The logic used to come up with that fixed value is the same as for inferred return types and implicitly typed arrays. The following listing shows an example of this without using any lambda expres- sions or even anonymous methods. static void PrintType(T first, T second) { Console.WriteLine(typeof(T)); } ... PrintType(1, new object()); Although the code in listing 9.14 is syntactically valid in C# 2, it wouldn’t build; type inference would fail, because the first parameter would decide that T must be int and the second parameter would decide that T must be object. In C# 3, the compiler determines that T should be object in exactly the same way that it did for the inferred return type in listing 9.13. In fact, the inferred return type rules are effectively one example of the more general process in C# 3. The second change is that type inference is now performed in two phases. The first phase deals with normal arguments where the types involved are known to begin with. This includes anonymous functions where the parameter list is explicitly typed. The second phase then kicks in, where implicitly typed lambda expressions and method groups have their types inferred. The idea is to see whether any of the infor- mation the compiler has pieced together so far is enough to work out the parameter types of the lambda expression (or method group). If it is, the compiler can then examine the body of the lambda expression and work out the inferred return type which is often another of the type parameters it’s looking for. If the second phase gives some more information, the compiler goes through it again, repeating until either it runs out of clues or it’s worked out all the type parameters involved. Figure 9.7 shows this in flowchart form, but please bear in mind that this is a heav- ily simplified version of the algorithm. Let’s look at two examples to show how it works. First we’ll take the code we started the section with—listing 9.11: static void PrintConvertedValue (TInput input, Converter converter) { Console.WriteLine(converter(input)); } ... PrintConvertedValue("I'm a string", x => x.Length); Listing 9.14 Flexible type inference combining information from multiple arguments 256 CHAPTER 9 Lambda expressions and expression trees The type parameters you need to work out here are TInput and TOutput. The steps performed are as follows: 1 Phase 1 begins. 2 The first parameter is of type TInput, and the first argument is of type string. You infer that there must be an implicit conversion from string to TInput. 3 The second parameter is of type Converter, and the second argument is an implicitly typed lambda expression. No inference is per- formed—you don’t have enough information. 4 Phase 2 begins. 5 TInput doesn’t depend on any unfixed type parameters, so it’s fixed to string. Are there any unfixed type variables left? Finished: type inference succeeded Finished: type inference failed (compile error) No No Yes Yes Phase 1 Phase 2 s Have we made any progress this iteration? Infer information from explicitly typed arguments Fix type variables which don't depend on anything still being decided Infer more information based on newly fixed type parameters Figure 9.7 The two-phase type inference flow 257Changes to type inference and overload resolution 6 The second argument now has a fixed input type but an unfixed output type. You can consider it to be (string x) => x.Length and infer the return type as int. Therefore an implicit conversion must take place from int to TOutput. 7 Phase 2 repeats. 8 TOutput doesn’t depend on anything unfixed, so it’s fixed to int. 9 There are now no unfixed type parameters, so inference succeeds. Complicated, eh? Still, it does the job—the result is what you’d want (TInput=string, TOutput=int), and everything compiles without any problems. The importance of phase 2 repeating is best shown with another example. Listing 9.15 shows two conversions being performed, with the output of the first one becoming the input of the second. Until you’ve worked out the output type of the first conversion, you don’t know the input type of the second, so you can’t infer its output type either. static void ConvertTwice (TInput input, Converter firstConversion, Converter secondConversion) { TMiddle middle = firstConversion(input); TOutput output = secondConversion(middle); Console.WriteLine(output); } ... ConvertTwice("Another string", text => text.Length, length => Math.Sqrt(length)); The first thing to notice is that the method signature appears to be pretty horrific. It’s not too bad when you stop being scared and just look at it carefully, and certainly the example usage makes it more obvious. You take a string and perform a conversion on it—the same conversion as before, just a length calculation. You then take that length (an int) and find its square root (a double). Phase 1 of type inference tells the compiler that there must be a conversion from string to TInput. The first time through phase 2, TInput is fixed to string and you infer that there must be a conversion from int to TMiddle. The second time through phase 2, TMiddle is fixed to int and you infer that there must be a conversion from double to TOutput. The third time through phase 2, TOutput is fixed to double and type inference succeeds. When type inference has finished, the compiler can look at the code within the lambda expression properly. CHECKING THE BODY OF A LAMBDA EXPRESSION The body of a lambda expres- sion cannot be checked until the input parameter types are known. The lambda expression x => x.Length is valid if x is an array or a string, but invalid in Listing 9.15 Multistage type inference 258 CHAPTER 9 Lambda expressions and expression trees many other cases. This isn’t a problem when the parameter types are explic- itly declared, but with an implicit parameter list, the compiler needs to wait until it’s performed the relevant type inference before it can try to work out what the lambda expression means. These examples have shown only one change working at a time, but in practice there can be several pieces of information about different type variables, potentially discov- ered in different iterations of the process. In an effort to save your sanity (and mine), I won’t present any more complicated examples—hopefully you understand the gen- eral mechanism, even if the exact details are hazy. Although it may seem as if this kind of situation will occur so rarely that it’s not worth having such complex rules to cover it, in fact it’s common in C# 3, particularly with LINQ. You could easily use type inference extensively without thinking about it— it’s likely to become second nature to you. If it fails and you wonder why, you can always revisit this section and the language specification. We need to cover one more change, but you’ll be glad to hear it’s easier than type inference. Let’s look at method overloading. 9.4.4 Picking the right overloaded method Overloading occurs when there are multiple methods available with the same name but different signatures. Sometimes it’s obvious which method is appropriate, because it’s the only one with the right number of parameters, or it’s the only one where all the arguments can be converted into the corresponding parameter types. The tricky bit comes when there are multiple methods that could be the right one. The rules in section 7.5.3 of the specification (“Overload Resolution”) are quite com- plicated (yes, again), but the key part is the way that each argument type is converted into the parameter type.9 For instance, consider these method signatures as if they were both declared in the same type: void Write(int x) void Write(double y) The meaning of a call to Write(1.5) is obvious, because there’s no implicit conver- sion from double to int, but a call to Write(1) is trickier. There is an implicit conver- sion from int to double, so both methods are possible. At that point, the compiler considers the conversion from int to int and from int to double. A conversion from any type to itself is defined to be better than any conversion to a different type, so the Write(int x) method is better than Write(double y) for this particular call. When there are multiple parameters, the compiler has to make sure there’s a best method to use. One method is better than another if all the argument conversions involved are at least as good as the corresponding conversions in the other method, and at least one conversion is strictly better. As a simple example, suppose you had this: 9 I’m assuming that all the methods are declared in the same class. When inheritance is involved as well, it becomes even more complicated. That aspect hasn’t changed in C# 3, though. 259Changes to type inference and overload resolution void Write(int x, double y) void Write(double x, int y) A call to Write(1, 1) would be ambiguous, and the compiler would force you to add a cast to at least one of the parameters to make it clear which method you meant to call. Each overload has one better argument conversion, so neither of them is the best method. That logic still applies to C# 3, but with one extra rule about anonymous functions, which never specify a return type. In this case, the inferred return type (as described in section 9.4.2) is used in the better-conversion rules. Let’s look at an example of the kind of situation that needs the new rule. The fol- lowing listing contains two methods with the name Execute and a call using a lambda expression. static void Execute(Func action) { Console.WriteLine("action returns an int: " + action()); } static void Execute(Func action) { Console.WriteLine("action returns a double: " + action()); } ... Execute(() => 1); The call to Execute in listing 9.16 could’ve been written with an anonymous method or a method group instead—the same rules are applied whatever kind of conversion is involved. Which Execute method should be called? The overloading rules say that when two methods are both applicable after performing conversions on the argu- ments, those argument conversions are examined to see which one is better. The con- versions here aren’t from a normal .NET type to the parameter type—they’re from a lambda expression to two different delegate types. Which conversion is better? Surprisingly enough, the same situation in C# 2 would result in a compilation error—there was no language rule covering this case. In C# 3, the method with the Func parameter would be chosen. The extra rule that has been added can be paraphrased this way: If an anonymous function can be converted to two delegate types that have the same parameter list but different return types, the delegate conversions are judged by the conversions from the inferred return type to the delegates’ return types. That’s pretty much gibberish without referring to an example. Let’s look back at list- ing 9.16 where you’re converting from a lambda expression with no parameters and an inferred return type of int to either Func or Func. The parameter lists are the same (empty) for both delegate types, so the rule applies. You then just Listing 9.16 Sample of overloading choice influenced by delegate return type 260 CHAPTER 9 Lambda expressions and expression trees need to find the better conversion: int to int, or int to double. This puts you in more familiar territory; as you saw earlier, the int to int conversion is better. Listing 9.16 therefore prints out action returns an int: 1. 9.4.5 Wrapping up type inference and overload resolution This section has been pretty heavy. I would’ve loved to have made it simpler, but it’s a fundamentally complicated topic. The terminology involved doesn’t make it any eas- ier, especially as parameter type and type parameter mean completely different things! Congratulations if you made it through and actually understood it all. Don’t worry if you didn’t; hopefully next time you read through the section, it’ll shed more light on the topic—particularly after you’ve run into situations where it’s relevant to your own code. For the moment, here are the most important points:  Anonymous functions (anonymous methods and lambda expressions) have inferred return types based on the types of all the return statements.  Lambda expressions can only be understood by the compiler when the types of all the parameters are known.  Type inference no longer requires that each argument independently come to exactly the same conclusion about type parameters, as long as the results stay compatible.  Type inference is now multistage: the inferred return type of one anonymous function can be used as a parameter type for another.  Finding the best overloaded method when anonymous functions are involved takes the inferred return type into account. Even that short list is pretty daunting in terms of the sheer density of technical terms. Again, don’t fret if it doesn’t all make sense. In my experience things just work the way you want them to most of the time. 9.5 Summary In C# 3, lambda expressions almost entirely replace anonymous methods. Anonymous methods are supported for the sake of backward compatibility, but idiomatic, freshly written C# 3 code will contain few of them. You’ve seen how lambda expressions are more than just a compact syntax for dele- gate creation. They can be converted into expression trees, subject to some limita- tions. The expression trees can then be processed by other code, possibly performing equivalent actions in different execution environments. Without this ability, LINQ would be restricted to in-process queries. Our discussion of type inference and overloading was a necessary evil to some extent; very few people actually enjoy discussing the sort of rules that are required, but it’s important to have at least a passing understanding of what’s going on. Before we all feel too sorry for ourselves, spare a thought for the poor language designers who have to live and breathe this kind of thing, making sure the rules are consistent and 261Summary don’t fall apart in nasty situations. Then pity the testers who have to try to break the implementation. That’s it in terms of describing lambda expressions, but you’ll see a lot more of them in the rest of the book. For instance, the next chapter is all about extension methods. Superficially, they’re completely separate from lambda expressions, but in reality the two features are often used together. 262 Extension methods I’m not a fan of inheritance. Or rather, I’m not a fan of a number of places where inheritance has been used in code that I’ve maintained, or class libraries I’ve worked with. As with so many things, it’s powerful when used properly, but its design over- head is often overlooked and can become painful over time. It’s sometimes used as a way of adding extra behavior and functionality to a class, even when no real infor- mation about the object is being added—where nothing is being specialized. Sometimes that’s appropriate—if objects of the new type should carry around the details of the extra behavior—but often it’s not. Often it’s just not possible to use inheritance in this way in the first place, such as when you’re working with a value type, a sealed class, or an interface. The alternative is usually to write a bunch of static methods, most of which take an instance of the type in question as at least This chapter covers  Writing extension methods  Calling extension methods  Method chaining  Extension methods in .NET 3.5  Other uses for extension methods 263Life before extension methods one of their parameters. This works fine, without the design penalty of inheritance, but it tends to make code look ugly. C# 3 introduced the idea of extension methods, which have the benefits of the static methods solution and also improve the readability of code that calls them. They let you call static methods as if they were instance methods of a completely different class. Don’t panic—it’s not as crazy or as arbitrary as it sounds. In this chapter we’ll first look at how to use extension methods and how to write them. We’ll then examine a few of the extension methods provided by .NET 3.5 and see how they can be chained together easily. This chaining ability is an important part of the reason for introducing extension methods to the language in the first place, and it’s an important part of LINQ.1 Finally, we’ll consider some of the pros and cons of using extension methods instead of plain static methods. First, though, let’s take a closer look at why extension methods are sometimes desirable compared with what’s available in C# 1 and 2, particularly when you create utility classes. 10.1 Life before extension methods You may be getting a sense of déjà vu at this point, because utility classes came up in chapter 7 when we looked at static classes. If you wrote a lot of C# 2 code before using C# 3, you should look at your static classes—many of the methods in them may be good candidates for converting into extension methods. That’s not to say that all exist- ing static classes are a good fit, but you may well recognize the following traits:  You want to add some members to a type.  You don’t need to add any more data to the instances of the type.  You can’t change the type itself, because it’s in someone else’s code. One slight variation on this is where you want to work with an interface instead of a class, adding useful behavior while only calling methods on the interface. A good example of this is IList. Wouldn’t it be nice to be able to sort any (mutable) implementation of IList? It’d be horrendous to force each implementation of the interface to implement sorting, but it’d be nice from the point of view of the user of the list. The thing is, IList provides all the building blocks for a completely generic sort routine (several, in fact), but you can’t put that implementation in the interface. IList could’ve been specified as an abstract class instead, and the sorting func- tionality included that way, but as C# and .NET have single inheritance of implementa- tion, that would’ve placed a significant restriction on the types deriving from it. An extension method on IList would allow you to sort any IList implementa- tion, making it appear as if the list itself provided the functionality. 1 If you’re getting fed up with hearing about how many features are “an important part of LINQ,” I don’t blame you, but that’s part of its greatness. There are lots of small parts, but the sum of them is very shiny. The fact that each feature can be used independently is an added bonus. 264 CHAPTER 10 Extension methods You’ll see later that a lot of the functionality of LINQ is built on extension methods over interfaces. For the moment, though, we’ll use a different type for our examples: System.IO.Stream, the bedrock of binary communication in .NET. Stream itself is an abstract class with several concrete derived classes, such as NetworkStream, FileStream, and MemoryStream. Unfortunately, there are a few pieces of functionality that would’ve been handy to include in Stream that just aren’t there. The missing features I’m most often aware of are the ability to read the whole of a stream into memory as a byte array, and the ability to copy the contents of one stream into another.2 Both of these features are frequently implemented badly, making assumptions about streams that just aren’t valid—the most common misconception being that Stream.Read will completely fill the buffer if the data doesn’t run out first. NOT SO “MISSING” AFTER ALL One of these features has been added to .NET 4: Stream now has a CopyTo method. This is useful in terms of demonstrating one slightly brittle aspect of extension methods, and we’ll come back to it in section 10.2.3. ReadFully is still missing, but it should be used carefully any- way: you should only try to read the entirety of a stream if you’re confident it actually has an end and that all the data fits into memory. Streams are under no obligation to have a finite amount of data. It’d be nice to have the functionality in a single place, rather than duplicating it in sev- eral projects. That’s why I wrote the StreamUtil class in my miscellaneous utility library. The real code contains a fair amount of error checking and other functional- ity, but the following listing shows a cut-down version that’s more than adequate for our needs. using System.IO; public static class StreamUtil { const int BufferSize = 8192; public static void Copy(Stream input, Stream output) { byte[] buffer = new byte[BufferSize]; int read; while ((read = input.Read(buffer, 0, buffer.Length)) > 0) { output.Write(buffer, 0, read); } } public static byte[] ReadFully(Stream input) { using (MemoryStream tempStream = new MemoryStream()) 2 Due to the nature of streams, this copying doesn’t necessarily duplicate the data—it just reads it from one stream and writes it to another. Although copy isn’t a strictly accurate term in this sense, the difference is usu- ally irrelevant. Listing 10.1 A simple utility class to provide extra functionality for streams 265Extension method syntax { Copy(input, tempStream); return tempStream.ToArray(); } } } The implementation details don’t matter much, although it’s worth noting that the ReadFully method calls the Copy method—that’ll be useful to demonstrate a point about extension methods later. The class is easy to use—the following listing shows how you can write a web response to disk, for example. WebRequest request = WebRequest.Create("http://manning.com"); using (WebResponse response = request.GetResponse()) using (Stream responseStream = response.GetResponseStream()) using (FileStream output = File.Create("response.dat")) { StreamUtil.Copy(responseStream, output); } Listing 10.2 is quite compact, and the StreamUtil class has taken care of looping and asking the response stream for more data until it’s all been received. It’s done its job as a utility class perfectly reasonably. Even so, it doesn’t feel very object-oriented. It’d be better to ask the response stream to copy itself to the output stream, just like the MemoryStream class has a WriteTo method. It’s not a big problem, but it’s a little ugly as it is. Inheritance wouldn’t help you in this situation (you want this behavior to be avail- able for all streams, not just ones you’re responsible for), and you can’t go changing the Stream class itself, so what can you do? With C# 2, you were out of options—you had to stick with the static methods and live with the clumsiness. C# 3 allows you to change your static class to expose its members as extension methods, so you can pre- tend that the methods have been part of Stream all along. Let’s see what changes are required. 10.2 Extension method syntax Extension methods are almost embarrassingly easy to create, and they’re simple to use, too. The considerations around when and how to use them are significantly deeper than the difficulties involved in learning how to write them in the first place. Let’s start by converting the StreamUtil class so it has a couple of extension methods. 10.2.1 Declaring extension methods You can’t use just any method as an extension method—it must have the following characteristics: Listing 10.2 Using StreamUtil to copy a web response stream to a file 266 CHAPTER 10 Extension methods  It must be in a non-nested, nongeneric static class (and therefore must be a static method).  It must have at least one parameter.  The first parameter must be prefixed with the this keyword.  The first parameter can’t have any other modifiers (such as out or ref).  The type of the first parameter must not be a pointer type. That’s it—the method can be generic, return a value, have ref/out parameters other than the first one, be implemented with an iterator block, be part of a partial class, use nullable types—anything, as long as the preceding constraints are met. We’ll call the type of the first parameter the extended type of the method and say that the method extends that type—in this case, we’re extending Stream. This isn’t official terminology from the specification, but it’s a useful piece of shorthand. Not only does the previous list provide all the restrictions, but it also gives the details of what you need to do to turn a normal static method in a static class into an extension method—just add the this keyword. The following listing shows the same class as in listing 10.1, but this time with both methods as extension methods. public static class StreamUtil { const int BufferSize = 8192; public static void CopyTo(this Stream input, Stream output) { byte[] buffer = new byte[BufferSize]; int read; while ((read = input.Read(buffer, 0, buffer.Length)) > 0) { output.Write(buffer, 0, read); } } public static byte[] ReadFully(this Stream input) { using (MemoryStream tempStream = new MemoryStream()) { CopyTo(input, tempStream); return tempStream.ToArray(); } } } Yes, the only big change in listing 10.3 is the addition of the two modifiers shown in bold. I’ve also changed the name of the method from Copy to CopyTo. As you’ll see in a minute, that’ll allow calling code to read more naturally, although it does look slightly strange in the ReadFully method at the moment. Now, it’s not much use having extension methods if you can’t use them… Listing 10.3 The StreamUtil class again, but this time with extension methods 267Extension method syntax 10.2.2 Calling extension methods I’ve mentioned it in passing, but you haven’t yet seen what an extension method actu- ally does. Simply put, it pretends to be an instance method of another type—the type of the first parameter of the method. The transformation of code that uses StreamUtil is as simple as the transforma- tion of the utility class itself. This time, instead of adding something in, we’ll take it away. The following listing is a repeat performance of listing 10.2, but using the new syntax to call CopyTo. I say “new,” but it’s really not new at all—it’s the same syntax you’ve always used for calling instance methods. WebRequest request = WebRequest.Create("http://manning.com"); using (WebResponse response = request.GetResponse()) using (Stream responseStream = response.GetResponseStream()) using (FileStream output = File.Create("response.dat")) { responseStream.CopyTo(output); } In listing 10.4 it at least looks like you’re asking the response stream to do the copying. It’s still StreamUtil doing the work behind the scenes, but the code reads in a more natural way. In fact, the compiler has converted the CopyTo call into a normal static method call to StreamUtil.CopyTo, passing the value of responseStream as the first argument (followed by output as normal). Now that you can see the code in question, I hope you understand why I changed the method name from Copy to CopyTo. Some names work just as well for static meth- ods as instance methods, but you’ll find that others need tweaking to get the maxi- mum readability benefit. If you want to make the StreamUtil code slightly more pleasant, you can change the line of ReadFully that calls CopyTo like this: input.CopyTo(tempStream); At this point, the name change is fully appropriate for all the uses—although there’s nothing to stop you from using the extension method as a normal static method, which is useful when you’re migrating a lot of code. You may have noticed that nothing in these method calls indicates that you’re using an extension method instead of a regular instance method of Stream. This can be seen in two ways: it’s a good thing if your aim is to make extension methods blend in as much as possible and cause little alarm, but it’s a bad thing if you want to be able to immediately see what’s really going on. If you’re using Visual Studio, you can hover over a method call and get an indica- tion in the tooltip when it’s an extension method, as shown in figure 10.1. IntelliSense also indicates when it’s offering an extension method, in both the icon for the method and the tooltip when it’s selected. Of course, you don’t want to have to hover Listing 10.4 Copying a stream using an extension method 268 CHAPTER 10 Extension methods over every method call you make or be super careful with IntelliSense, but most of the time it doesn’t matter whether you’re calling an instance or extension method. There’s still one rather strange thing about this calling code—it doesn’t mention StreamUtil anywhere! How does the compiler know to use the extension method in the first place? 10.2.3 Extension method discovery It’s important to know how to call extension methods, but it’s also important to know how to not call them—how to avoid being presented with unwanted options. To achieve that, you need to know how the compiler decides which extension methods to use in the first place. Extension methods are made available to the code in the same way that classes are made available without qualification—with using directives. When the compiler sees an expression that looks like it’s trying to use an instance method, but none of the instance methods are compatible with the method call (if there’s no method with that name, for instance, or no overload matches the arguments given), it then looks for an appropriate extension method. It considers all the extension methods in all the imported namespaces and the current namespaces, and matches ones where there’s an implicit conversion from the expression type to the extended type. WebRequest request = WebRequest.Create("http://manning.com"); using (WebResponse response = request.GetResponse()) using (Stream responseStream = response.GetResponseStream()0 using (FileStream output = File.Create("response.dat")) { responseStream.CopyTo(output); } py ( p ); (extension) void Stream.CopyTo(Stream output) Figure 10.1 Hovering over a method call in Visual Studio reveals whether the method is an extension method. Implementation detail: how does the compiler spot an extension method? To work out whether it should use an extension method, the compiler has to be able to tell the difference between an extension method and other methods within a static class that happen to have an appropriate signature. It does this by checking whether System.Runtime.CompilerServices.ExtensionAttribute has been applied to the method and the class. This attribute was introduced in .NET 3.5, but the compiler doesn’t check which assembly the attribute comes from. This means that you can still use extension methods even if your project targets .NET 2.0—you just need to define your own attribute with the right name in the right namespace. You can then declare your extension methods as normal, and the attribute will be applied automatically. The compiler also applies the attribute to the assembly con- taining the extension method, but it doesn’t currently require this when searching for extension methods. 269Extension method syntax If multiple applicable extension methods are available for different extended types (using implicit conversions), the most appropriate one is chosen with the better con- version rules used in overloading. For instance, if IDerived inherits from IBase, and there’s an extension method with the same name for both, then the IDerived exten- sion method is used in preference to the one on IBase. Again, this feature is used in LINQ, as you’ll see in section 12.2, where you’ll meet the IQueryable interface. It’s important to note that if an applicable instance method is available, that will always be used before searching for extension methods, but the compiler doesn’t issue a warning if an extension method also matches an existing instance method. For example, .NET 4 has a new Stream method that’s also called CopyTo. It has two over- loads, one of which conflicts with the extension method you just created. The result is that the new method is picked in preference to the extension method, so if you com- pile listing 10.4 against .NET 4, you’ll end up using Stream.CopyTo instead of Stream- Util.CopyTo. You can still call the StreamUtil method statically using the normal syntax of StreamUtil.CopyTo(input, output), but it’ll never be picked as an exten- sion method. In this case, there’s no harm to existing code: the new instance method has the same meaning as your extension method, so it doesn’t matter which one is used. In other cases, there could be subtle differences in semantics that might be hard to spot until the code breaks. Another potential problem with the way that extension methods are made avail- able to code is that it’s very wide-ranging. If there are two classes in the same namespace containing methods with the same extended type, there’s no way of only using the extension methods from one of the classes. Likewise, there’s no way of importing a namespace for the sake of making types available using only their simple names, but without making the extension methods within that namespace available at the same time. You may want to use a namespace that solely contains static classes with extension methods to mitigate this problem, unless the rest of the functionality of the namespace is heavily dependent on the extension methods already (as is the case for System.Linq, for example). One aspect of extension methods can be quite surprising when you first encounter it, but it’s also useful in some situations. It’s all about null references—let’s take a look. 10.2.4 Calling a method on a null reference Anyone who does a significant amount of .NET programming is bound to encounter a NullReferenceException caused by calling a method via a variable whose value turns Introducing your own copies of system types can become problematic when you later need to use a version of the framework that already defines those types. If you do use this technique, it’s worth using preprocessor symbols to only declare the attri- bute conditionally. You can then build one version of your code targeting .NET 2.0 and another targeting .NET 3.5 and higher. 270 CHAPTER 10 Extension methods out to be a null reference. You can’t call instance methods on null references in C# (although IL itself supports it for nonvirtual calls), but you can call extension methods with a null reference. This is demonstrated by the following listing. Note that this isn’t a snippet, since nested classes can’t contain extension methods. using System; public static class NullUtil { public static bool IsNull(this object x) { return x == null; } } public class Test { static void Main() { object y = null; Console.WriteLine(y.IsNull()); y = new object(); Console.WriteLine(y.IsNull()); } } The output of listing 10.5 is True, and then False. If IsNull had been a normal instance method, an exception would’ve been thrown in the second line of Main; instead, IsNull was called with null as the argument. Prior to the advent of extension methods, C# had no way of letting you write the more readable y.IsNull() form safely, requiring NullUtil.IsNull(y) instead. There’s one particularly obvious example in the framework where this aspect of the behavior of extension methods could be useful: string.IsNullOrEmpty. C# 3 allows you to write an extension method that has the same signature (other than the extra parameter for the extended type) as an existing static method on the extended type. To save you reading through that sentence several times, here’s an example— even though the string class has a static, parameterless method IsNullOrEmpty, you can still create and use the following extension method: public static bool IsNullOrEmpty(this string text) { return string.IsNullOrEmpty(text); } At first it seems odd to be able to call IsNullOrEmpty on a variable that’s null without an exception being thrown, particularly if you’re familiar with it as a static method from .NET 2.0. But in my view, code using the extension method is more easily under- standable. For instance, if you read the expression if (name.IsNullOrEmpty()) out loud, it says exactly what it’s doing. Listing 10.5 Extension method being called on a null reference 271Extension methods in .NET 3.5 As always, experiment to see what works for you, but be aware of the possibility of other people using this technique if you’re debugging code. Don’t assume that an exception will be thrown on a method call unless you’re sure it’s not an extension method. Also, think carefully before reusing an existing name for an extension method—the previous extension method could confuse readers who are only familiar with the static method from the framework. CHECKING FOR NULLITY I’m sure that, as a conscientious developer, your pro- duction methods always check their arguments’ validity before proceeding. One question that naturally arises from this quirky feature of extension meth- ods is what exception you should throw when the first argument is null (assuming it’s not meant to be). Should it be ArgumentNullException, as if it were a normal argument, or should it be NullReferenceException, which is what would’ve happened if the extension method had been an instance method to start with? I recommend the former: it’s still an argument, even if the extension method syntax doesn’t make that obvious. This is the route that Microsoft has taken for the extension methods in the framework, so it has the benefit of consistency too. Finally, bear in mind that extension methods can still be called as normal static methods, and in that situation, Argument- NullException is clearly the preferred result. Now that you know the syntax and behavior of extension methods, we can look at some examples of the ones provided in .NET 3.5 as part of the framework. 10.3 Extension methods in .NET 3.5 The biggest use of extension methods in the framework is for LINQ. Some LINQ pro- viders have a few extension methods to help them along, but there are two classes that stand out, both of them appearing in the System.Linq namespace: Enumerable and Queryable. These contain many, many extension methods; most of the ones in Enumerable operate on IEnumerable and most of those in Queryable operate on IQueryable. We’ll look at the purpose of IQueryable in chapter 12, but for the moment let’s concentrate on Enumerable. 10.3.1 First steps with Enumerable Enumerable has a lot of methods in it, and the purpose of this section isn’t to cover all of them, but to give you enough of a feel for them that you’re comfortable going off and experimenting. It’s a joy to play with everything available in Enumerable, and it’s definitely worth firing up Visual Studio or LINQPad for your experiments (rather than using Snippy), as IntelliSense is handy for this kind of activity. Appendix A gives a quick rundown of the behavior of all Enumerable’s methods too. All the complete examples in this section deal with a simple situation: we’ll start with a collection of integers and transform it in various ways. Real-life situations are likely to be somewhat more complicated, usually dealing with business-related types, so at the end of this section I’ll present a couple examples of the transformation side 272 CHAPTER 10 Extension methods of things applied to possible business situations, with full source code available on the book’s website. But those examples are harder to play with than a straightforward col- lection of numbers. It’s worth considering some recent projects you’ve been working on as you read this chapter; see if you can think of situations where you could make your code sim- pler or more readable by using the kind of operations described here. There are a few methods in Enumerable that aren’t extension methods, and we’ll use one of them in the examples for the rest of the chapter. The Range method takes two int parameters: the number to start with and how many results to yield. The result is an IEnumerable that returns one number at a time in the obvious way. To demonstrate the Range method and create a framework to play with, let’s print out the numbers 0 to 9, as shown in the following listing. var collection = Enumerable.Range(0, 10); foreach (var element in collection) { Console.WriteLine(element); } No extension methods are called in listing 10.6, just a plain static method. And yes, it really does just print the numbers 0 to 9—I never claimed this code would set the world on fire. DEFERRED EXECUTION The Range method doesn’t build a list with the appro- priate numbers—it just yields them at the appropriate time. In other words, constructing the enumerable instance doesn’t do the bulk of the work; it gets things ready, so that the data can be provided in a just-in-time fashion at the appropriate point. This is called deferred execution—you saw this sort of behav- ior when we looked at iterator blocks in chapter 6, but you’ll see much more of it in the next chapter. Pretty much the simplest thing you can do with a sequence of numbers that’s already in order is to reverse it. The following listing uses the Reverse extension method to do this—it returns an IEnumerable that yields the same elements as the original sequence, but in the reverse order. var collection = Enumerable.Range(0, 10) .Reverse(); foreach (var element in collection) { Console.WriteLine(element); } Listing 10.6 Using Enumerable.Range to print out the numbers 0 to 9 Listing 10.7 Reversing a collection with the Reverse method 273Extension methods in .NET 3.5 Predictably enough, this prints out 9, then 8, then 7, and so on right down to 0. You called Reverse (seemingly) on an IEnumerable, and the same type has been returned. This pattern of returning one enumerable based on another is pervasive in the Enumerable class. Let’s do something more adventurous now—we’ll use a lambda expression to remove the even numbers. 10.3.2 Filtering with Where and chaining method calls together The Where extension method is a simple but powerful way of filtering collections. It accepts a predicate, which it applies to each of the elements of the original collection. It returns an IEnumerable, and any element that matches the predicate is included in the resulting collection. Listing 10.8 demonstrates this, applying the odd/even filter to the collection of integers before reversing it. You don’t have to use a lambda expression here; for instance, you could use a delegate you’d created earlier, or an anonymous method. In this case (and in many other real-life situations), it’s simple to put the filtering logic inline, and lambda expressions keep the code concise. var collection = Enumerable.Range(0, 10) .Where(x => x % 2 != 0) .Reverse(); foreach (var element in collection) { Console.WriteLine(element); } Listing 10.8 Using the Where method with a lambda expression to find odd numbers Efficiency: buffering versus streaming The extension methods provided by the framework stream or pipe data wherever pos- sible. When an iterator is asked for its next element, it’ll often take an element from the iterator it’s chained to, process that element, and then return something appro- priate, preferably without using any more storage itself. Simple transformations and filters can do this easily, and it’s a powerful way of efficiently processing data where it’s possible, but some operations, such as reversing the order or sorting, require all the data to be available, so it’s all loaded into memory for bulk processing. The dif- ference between this buffered approach and piping is similar to the difference between reading data by loading a whole DataSet versus using a DataReader to pro- cess one record at a time. It’s important to consider what’s required when using LINQ—a single method call can have significant performance implications. Streaming is also known as lazy evaluation, and buffering is also known as eager evaluation. For example, the Reverse method uses deferred execution (it does noth- ing until the first call to MoveNext), but it then eagerly evaluates its data source. Per- sonally, I dislike the terms lazy and eager, as they mean different things to different people (a topic I discuss more in my “Just how lazy are you?” blog entry: http:// mng.bz/3LLM). 274 CHAPTER 10 Extension methods Listing 10.8 prints out the numbers 9, 7, 5, 3, and 1. Hopefully, you’ll have noticed a pattern forming—you’re chaining the method calls together. The chaining idea itself isn’t new. For example, StringBuilder.Replace always returns the instance you call it on, allowing code like this: builder = builder.Replace("<", "<") .Replace(">", ">") ... In contrast, String.Replace returns a string, but a new one each time—this allows chaining, but in a slightly different way. Both patterns are handy to know about; the “return the same reference” pattern works well for mutable types, whereas “return a new instance that’s a copy of the original with some changes” is required for immuta- ble types. Chaining with instance methods like String.Replace and StringBuilder .Replace has always been simple, but extension methods allow static method calls to be chained together. This is one of the primary reasons why extension methods exist. They’re useful for other utility classes, but their true power is revealed in this ability to chain static methods in a natural way. That’s why extension methods primarily show up in Enumerable and Queryable in .NET: LINQ is geared toward this approach to data pro- cessing, with information effectively traveling through pipelines constructed of indi- vidual operations chained together. EFFICIENCY CONSIDERATION: REORDERING METHOD CALLS TO AVOID WASTE I’m not a fan of micro-optimization without good cause, but it’s worth looking at the ordering of the method calls in listing 10.8. You could’ve added the Where call after the Reverse call and achieved the same results, but that would’ve wasted some effort—the Reverse call would’ve had to work out where the even numbers should come in the sequence even though they’ll be discarded from the final result. In this case, it won’t make much difference, but it can have a significant effect on performance in real situations; if you can reduce the amount of wasted work without compromising readability, that’s a good thing. That doesn’t mean you should always put filters at the start of the pipe- line, though; you need to think carefully about any reordering to make sure you get the correct results. There are two obvious ways of writing the first part of listing 10.8 without using the fact that Reverse and Where are extension methods. One is to use a temporary vari- able, which keeps the structure intact: var collection = Enumerable.Range(0, 10); collection = Enumerable.Where(collection, x => x % 2 != 0) collection = Enumerable.Reverse(collection); I hope you’ll agree that the meaning of the code is far less clear here than in listing 10.8. It gets even worse with the other option, which is to keep the single-statement style: 275Extension methods in .NET 3.5 var collection = Enumerable.Reverse (Enumerable.Where (Enumerable.Range(0, 10), x => x % 2 != 0)); The method call order appears to be reversed, because the innermost method call (Range) will be performed first, then the others, with execution working its way out- ward. Even with just three method calls it’s ugly—it becomes far worse for queries involving more operators. Before we move on, let’s think a bit about what the Where method does. 10.3.3 Interlude: haven’t we seen the Where method before? If the Where method feels familiar, it’s because you implemented it in chapter 6. All you need to do is convert listing 6.9 into an extension method and change the dele- gate type from Predicate to Func and you have a perfectly good alterna- tive implementation to Enumerable.Where: public static IEnumerable Where(this IEnumerable source, Func predicate) { if (source == null || predicate == null) { throw new ArgumentNullException(); } return WhereImpl(source, predicate); } private static IEnumerable WhereImpl(IEnumerable source, Func predicate) { foreach (T item in source) { if (predicate(item)) { yield return item; } } } You can change the last part of listing 6.9 to make it look more LINQ-like, too: foreach (string line in LineReader.ReadLines("../../FakeLinq.cs") .Where(line => line.StartsWith("using"))) { Console.WriteLine(line); } This is effectively a LINQ query without using the System.Linq namespace. It would work perfectly well in .NET 2.0 if you declared the appropriate Func delegate and [ExtensionAttribute]. You could even use that implementation for the where clause in a query expression (while still targeting .NET 2.0), as you’ll see in the next chap- ter—but let’s not get ahead of ourselves. 276 CHAPTER 10 Extension methods Filtering is one of the simplest operations in a query, and another is transforming or projecting the results. 10.3.4 Projections using the Select method and anonymous types The most commonly used projection method in Enumerable is Select. It operates on an IEnumerable and projects it into an IEnumerable by way of a Func, which is the transformation to use on each element, speci- fied as a delegate. It’s much like the ConvertAll method in List, but it operates on any enumerable collection and uses deferred execution to perform the projection when each element is requested. When I introduced anonymous types, I said they were useful with lambda expres- sions and LINQ—here’s an example of the kind of thing you can do with them. You currently have the odd numbers from 0 to 9 (in reverse order)—let’s create a type that encapsulates the square root of the number as well as the original number. The fol- lowing listing shows both the projection and a slightly modified way of writing out the results. I’ve adjusted the whitespace solely for the sake of space on the printed page. var collection = Enumerable.Range(0, 10) .Where(x => x % 2 != 0) .Reverse() .Select(x => new { Original = x, SquareRoot = Math.Sqrt(x) }); foreach (var element in collection) { Console.WriteLine("sqrt({0})={1}", element.Original, element.SquareRoot); } This time the type of collection isn’t IEnumerable—it’s IEnumerable , where Something is the anonymous type created by the compiler. You can’t give the collection variable an explicit type other than the nongeneric IEnumerable type or object. Implicit typing (with var) is what allows you to use the Original and SquareRoot properties when writing out the results. The output of listing 10.9 is as follows: sqrt(9)=3 sqrt(7)=2.64575131106459 sqrt(5)=2.23606797749979 sqrt(3)=1.73205080756888 sqrt(1)=1 Of course, a Select method doesn’t have to use an anonymous type at all—you could’ve selected just the square root of the number, discarding the original. In that case, the result would’ve been IEnumerable. Alternatively, you could’ve man- ually written a type to encapsulate an integer and its square root—it was just easiest to use an anonymous type in this case. Listing 10.9 Projection using a lambda expression and an anonymous type 277Extension methods in .NET 3.5 Let’s look at one last method to round off our coverage of Enumerable for the moment: OrderBy. 10.3.5 Sorting using the OrderBy method Sorting is a common requirement when processing data, and in LINQ this is usually performed by using the OrderBy or OrderByDescending methods. The first call is sometimes followed by ThenBy or ThenByDescending if you need to sort by more than one property of the data. This ability to sort on multiple properties has always been available the hard way using a complicated comparison, but it’s much clearer to be able to present a series of simple comparisons. To demonstrate this, let’s make a small change to the operations involved. You’ll start off with the integers –5 to 5 (inclusive, so there are 11 elements in total), and then project to an anonymous type containing the original number and its square (rather than square root). Finally, you’ll sort by the square and then the original num- ber. The following listing shows all of this. var collection = Enumerable.Range(-5, 11) .Select(x => new { Original = x, Square = x * x }) .OrderBy(x => x.Square) .ThenBy(x => x.Original); foreach (var element in collection) { Console.WriteLine(element); } Note how aside from the call to Enumerable.Range, the code reads almost exactly like the textual description. The anonymous type’s ToString implementation does the for- matting this time, and here are the results: { Original = 0, Square = 0 } { Original = -1, Square = 1 } { Original = 1, Square = 1 } { Original = -2, Square = 4 } { Original = 2, Square = 4 } { Original = -3, Square = 9 } { Original = 3, Square = 9 } { Original = -4, Square = 16 } { Original = 4, Square = 16 } { Original = -5, Square = 25 } { Original = 5, Square = 25 } As intended, the main sorting property is Square, but when two values have the same square, the negative original number is always sorted before the positive one. Writing a single comparison to do the same kind of thing (in a general case—there are mathe- matical tricks to cope with this particular example) would’ve been significantly more complicated, to the extent that you wouldn’t want to include the code inline in the lambda expression. Listing 10.10 Ordering a sequence by two properties 278 CHAPTER 10 Extension methods One thing to note is that the ordering doesn’t change an existing collection—it returns a new sequence that yields the same data as the input sequence, except sorted. Contrast this with List.Sort or Array.Sort, which both change the element order within the list or array. LINQ operators are intended to be side-effect free: they don’t affect their input, and they don’t make any other changes to the environment, unless you’re iterating through a naturally stateful sequence (such as reading from a network stream) or a delegate argument has side effects. This is an approach from functional programming, and it leads to code that’s more readable, testable, compos- able, predictable, thread-safe, and robust. We’ve looked at just a few of the many extension methods available in Enumerable, but hopefully you can appreciate how neatly they can be chained together. In the next chapter you’ll see how this can be expressed in a different way using extra syntax pro- vided by C# 3 (query expressions), and we’ll look at some other operations we haven’t covered here. It’s worth remembering that you don’t have to use query expressions— often it can be simpler to make a couple of calls to methods in Enumerable, using extension methods to chain operations together. Now that you’ve seen how all these apply to the collection-of-numbers example, it’s time for me to make good on the promise of showing you some business-related examples. 10.3.6 Business examples involving chaining Much of what we do as developers involves moving data around. In fact, for many appli- cations that’s the only meaningful thing we do—the user interface, web services, data- base, and other components often exist solely to get data from one place to another, or from one form into another. It should come as no surprise that the extension methods we’ve looked at in this section are well suited to many business problems. I’ll just give a couple of examples here. I’m sure you’ll be able to imagine how C# 3 and the Enumerable class can help you solve problems involving your business require- ments more expressively than before. For each example, I’ll only include a sample query—it should be enough to help you understand the purpose of the code, but without all the baggage. Full working code is on the book’s website. AGGREGATION: SUMMING SALARIES The first example involves a company composed of several departments. Each depart- ment has a number of employees, each of whom has a salary. Suppose you want to report on total salary cost by department, with the most expensive department listed first. The query is simply as follows: company.Departments .Select(dept => new { dept.Name, Cost = dept.Employees.Sum(person => person.Salary) }) .OrderByDescending(deptWithCost => deptWithCost.Cost); 279Extension methods in .NET 3.5 This query uses an anonymous type to keep the department name (using a projection initializer) and the sum of the salaries of all the employees within that department. The salary summation uses a self-explanatory Sum extension method, again part of Enumerable. In the result, the department name and total salary can be retrieved as properties. If you wanted the original department reference, you’d just need to change the anon- ymous type used in the Select method. GROUPING: COUNTING BUGS ASSIGNED TO DEVELOPERS If you’re a professional developer, I’m sure you’ve seen many project management tools giving you different metrics. If you have access to the raw data, LINQ can help you transform it in practically any way you choose. As a simple example, let’s look at a list of developers and how many bugs they have assigned to them at the moment: bugs.GroupBy(bug => bug.AssignedTo) .Select(list => new { Developer = list.Key, Count = list.Count() }) .OrderByDescending(x => x.Count); This query uses the GroupBy extension method, which groups the original collection by a projection (the developer assigned to fix the bug, in this case), resulting in an IGrouping. There are many overloads of GroupBy, but this example uses the simplest one and then selects just the key (the name of the developer) and the number of bugs assigned to him. After that you order the result to show the devel- opers with the most bugs first. One of the problems when looking at the Enumerable class can be working out exactly what’s going on; for example, one of the overloads of GroupBy has four type parameters and five normal parameters (three of which are delegates). Don’t panic— just follow the steps shown in chapter 3, assigning different types to different type parameters until you have a concrete example of what the method would look like. That usually makes it a lot easier to understand what’s going on. These examples aren’t particularly involved, but I hope you can see the power of chaining method calls together, where each method takes an original collection and returns another one in some form or other, whether by filtering out some values, ordering values, transforming each element, aggregating some values, or using other options. In many cases, the resulting code can be read aloud and understood immedi- ately, and in other situations it’s still usually a lot simpler than the equivalent code would’ve been in previous versions of C#. We’ll use the example of defect tracking as our sample data when we look at query expressions in the next chapter. Now that you’ve seen some of the extension methods that are provided, let’s consider just how and when it makes sense to write them yourself. 280 CHAPTER 10 Extension methods 10.4 Usage ideas and guidelines Like implicit typing of local variables, extension methods are controversial. It’d be dif- ficult to claim that they make the overall aim of the code harder to understand in many cases, but at the same time they do obscure the details of which method is get- ting called. In the words of one of the lecturers at my university, “I’m hiding the truth in order to show you a bigger truth.” If you believe that the most important aspect of the code is its result, extension methods are great. If the implementation is more important to you, then explicitly calling a static method is more clear. Effectively, it’s the difference between the what and the how. We’ve already looked at using extension methods for utility classes and method chaining, but before we discuss the pros and cons further, it’s worth calling out a cou- ple of aspects that may not be obvious. 10.4.1 “Extending the world” and making interfaces richer Wes Dyer, a former developer on the C# compiler team, has a fantastic blog covering all kinds of subject matter (see http://blogs.msdn.com/b/wesdyer/). One of his posts about extension methods particularly caught my attention (see http://mng.bz/ I4F2). It’s called “Extending the World,” and it talks about how extension methods can make code easier to read by effectively adapting your environment to your needs: Typically for a given problem, a programmer is accustomed to building up a solution until it finally meets the requirements. Now, it is possible to extend the world to meet the solution instead of solely just building up until we get to it. That library doesn’t provide what you need, just extend the library to meet your needs. This has implications beyond situations where you’d use a utility class. Typically devel- opers only start creating utility classes when they’ve seen the same kind of code repro- duced in dozens of places, but extending a library is about clarity of expression as much as avoiding duplication. Extension methods can make the calling code feel like the library is richer than it really is. You’ve already seen this with IEnumerable, where even the simplest implemen- tation appears to have a wide set of operations available, such as sorting, grouping, pro- jection, and filtering. The benefits aren’t limited to interfaces—you can also “extend the world” with enums, abstract classes, and so forth. The .NET Framework also provides a good example of another use for extension methods: fluent interfaces. 10.4.2 Fluent interfaces There used to be a television program in the United Kingdom called Catchphrase. The idea was that contestants would watch a screen where an animation would show some cryptic version of a phrase or saying, which they’d have to guess. The host would often try to help by instructing them: “Say what you see.” That’s pretty much the idea behind fluent interfaces—that if you read the code verbatim, its purpose will leap off 281Usage ideas and guidelines the screen as if it were written in a natural human language. The term “fluent inter- faces” was originally coined by Martin Fowler (see his blog entry at http://mng.bz/ 3T9T) and Eric Evans. If you’re familiar with domain-specific languages (DSLs), you may be wondering what the differences are between a fluent interface and a DSL. A lot has been written on the subject, but the consensus seems to be that a DSL has more freedom to create its own syntax and grammar, whereas a fluent interface is constrained by the host language (C#, in our case). Some good examples of fluent interfaces in the framework are the OrderBy and ThenBy methods: with a bit of interpretation of lambda expressions, the code explains exactly what it does. In the case of listing 10.10 earlier, you could read “order by the square, then by the original number” without much work. Statements end up reading as whole sentences rather than individual noun-verb phrases. Writing fluent interfaces can require a change of mindset. Method names defy the normal descriptive-verb form, with And, Then, and If sometimes being suitable meth- ods in a fluent interface. The methods themselves often do little more than set up con- text for future calls, often returning a type whose sole purpose is to act as a bridge between calls. Figure 10.2 illustrates how this bridging works. It only uses two extension methods (on int and TimeSpan), but they make all the difference in the readability. The grammar of the example in figure 10.2 could have many different forms; you may be able to add additional attendees to an UntimedMeeting or create an Unat- tendedMeeting at a particular time before specifying the attendees, for instance. For a lot more guidance on DSLs, see DSLs in Boo: Domain-Specific Languages in .NET by Ayende Rahien (Manning, 2010). C# 3 only supports extension methods rather than extension properties, which restricts fluent interfaces slightly. It means you can’t have expressions such as 1.week.from.now or 2.days + 10.hours (which are both valid in Groovy with an appropriate package— see Groovy’s Google Data Support: http://groovy.codehaus.org/Google+Data+ Support), but with a few superfluous parentheses you can achieve similar results. At first it looks odd to call a method on a number (such as 2.Dollars() or 3.Meters()), but it’s hard to deny that the meaning is clear. Without extension methods, this sort of clarity isn’t possible when you need to act on types such as numbers that aren’t under your control. Meeting.Between("Jon") .And("Russell") .At(8.OClock().Tomorrow()) Returns SoloMeeting Returns UntimedMeeting Returns Meeting Returns TimeSpan Returns DateTime Figure 10.2 Pulling apart a fluent interface expression to create a meeting. The time of the meeting is specified using extension methods to create a TimeSpan from an int, and a DateTime from a TimeSpan. 282 CHAPTER 10 Extension methods At the time of this writing, the development community is still on the fence about flu- ent interfaces: they’re relatively rare in most fields, although many mocking and unit testing libraries have at least some fluent aspects. They’re certainly not universally applicable, but in the right situations they can radically transform the readability of the calling code. As an example, with appropriate extension methods from my Misc- Util library, I can iterate over every day I’ve been alive in a readable way: foreach (DateTime day in 19.June(1976).To(DateTime.Today) .Step(1.Days())) Although the range-related implementation details are complicated, the extension methods allowing 19.June(1976) and 1.Days() are extremely simple. This is culture- specific code, which you may not want to expose in your production code, but it can make unit tests a great deal more pleasant. These aren’t the only uses available for extension methods, of course. I’ve used them for argument validation, implementing alternative approaches to LINQ, adding my own operators to LINQ to Objects, making composite comparisons easier to build, adding more flag-related functionality to enums, and much more. I’m constantly amazed at how such a simple feature can have such a profound impact on readability when used appropriately. The key word there is “appropriately,” which is easier to say than describe. 10.4.3 Using extension methods sensibly I’m in no position to dictate how you write your code. It may be possible to write tests to objectively measure readability for an average developer, but it only matters for those who’re going to use and maintain your code. You need to consult with the rele- vant people as far as you can, presenting different options and getting appropriate feedback. Extension methods make this particularly easy in many cases, as you can demonstrate both options in working code simultaneously—turning a method into an extension method doesn’t stop you from calling it explicitly in the same way as before. The main question to ask is the one I referred to at the start of this section: is the “what does it do” aspect of the code more important than the “how does it do it” aspect? That varies by person and situation, but here are some guidelines to bear in mind:  Everyone on the development team should be aware of extension methods and where they might be used. Where possible, avoid surprising code maintainers.  By putting extensions in their own namespace, you make it hard to use them accidentally. Even if it’s not obvious when reading the code, the developers writ- ing it should be aware of what they’re doing. Use a project-wide or company- wide convention for naming the namespace. You may choose to take this one step further and use a single namespace for each extended type. For instance, you could create a TypeExtensions namespace for classes that extend System.Type. 283Usage ideas and guidelines  Think carefully before you extend widely used types, such as numbers or object, or before you write a method where the extended type is a type param- eter. Some guidelines go as far as to recommend that you shouldn’t do this at all; I think such extensions have their place, but they should have to really earn their place in your library. In this situation, it’s even more important that the extension method be either internal or in its own namespace; I wouldn’t want IntelliSense to be suggesting the June extension method everywhere I used an integer, for example—only in classes that used at least some extension methods related to date and time.  The decision to write an extension method should always be a conscious one. It shouldn’t become habitual. Not every static method deserves to be an extension method.  Document whether the first parameter (the value your method appears to be called on) is allowed to be null—if it’s not, check the value in the method and throw an ArgumentNullException if necessary.  Be careful not to use a method name that already has a meaning in the extended type. If the extended type is a framework type or comes from a third- party library, check all your extended method names whenever you change ver- sions of the library. If you’re lucky (as I was with Stream.CopyTo), the new meaning is the same as the old, but even so, you may wish to deprecate your extension method.  Question your instincts, but acknowledge that they affect your productivity. Just like with implicit typing, there’s little point in forcing yourself to use a feature you instinctively dislike.  Try to group extension methods into static classes dealing with the same extended type. Sometimes related classes (such as DateTime and TimeSpan) can be sensibly grouped together, but avoid grouping extension methods targeting disparate types such as Stream and string within the same class.  Think really carefully before adding extension methods with the same extended type and same name in two different namespaces, particularly if there are situa- tions where the different methods may both be applicable (they have the same number of parameters). It’s reasonable for adding or removing a using direc- tive to make a program fail to build, but it’s nasty if it still builds but changes the behavior. Few of these guidelines are particularly clear-cut; to some extent you’ll have to feel your own way to the best use or avoidance of extension methods. It’s perfectly reason- able to never write your own extension methods at all, and to use the LINQ-related ones for the readability gains available there. But it’s worth at least thinking about what’s possible. 284 CHAPTER 10 Extension methods 10.5 Summary The mechanical aspect of extension methods is straightforward—the feature is simple to describe and demonstrate. The benefits (and costs) of them are harder to talk about in a definitive manner—it’s a touchy-feely topic, and different people are bound to have different views on the value provided. In this chapter I’ve tried to show a bit of everything. Early on, we looked at what the feature achieves in the language, and then we looked at some of the capabilities available through the framework. In some ways, this was a relatively gentle introduc- tion to LINQ; we’ll revisit some of the extension methods you’ve seen so far, and look at some new ones, when we delve into query expressions in the next chapter. A wide variety of methods is available within the Enumerable class, and we’ve only scratched the surface in this chapter. It’s fun to come up with a scenario of your own devising (whether hypothetical or in a real project) and browse through MSDN to see what’s available to help you. I urge you to use a sandbox project of some sort to play with the extension methods provided—it does feel like play rather than work, and you’re unlikely to want to limit yourself to just the methods you need to achieve your most immediate goal. Appendix A has a list of the standard query operators from LINQ, which covers many of the methods within Enumerable. New patterns and practices keep emerging in software engineering, and ideas from some systems often cross-pollinate to others. That’s one of the things that keeps development exciting. Extension methods allow code to be written in a way that was previously unavailable in C#, creating fluent interfaces and changing the environment to suit your code rather than the other way around. Those are just the techniques we’ve looked at in this chapter—there are bound to be interesting future develop- ments using the new C# features, whether individually or combined. The revolution obviously doesn’t end here. For a few calls, extension methods are fine. In the next chapter, we’ll look at the real power tools: query expressions and full- blown LINQ. 285 Query expressions and LINQ to Objects You may be tired of all the hyperbole around LINQ by now. You’ve already seen some examples in the book, and you’ve almost certainly read a lot about it on the web. This is where we separate myth from reality:  LINQ doesn’t turn the most complicated query into a one-liner.  LINQ doesn’t mean you never need to look at raw SQL again.  LINQ doesn’t magically imbue you with architectural genius. This chapter covers  Streaming sequences of data and deferred execution  Standard query operators and query expression translation  Range variables and transparent identifiers  Projecting, filtering, and sorting  Joining and grouping  Choosing which syntax to use 286 CHAPTER 11 Query expressions and LINQ to Objects Given all that, LINQ is still the best way of expressing queries that I’ve seen within an object-oriented environment. It’s not a silver bullet, but it’s a very powerful tool to have in your development armory. We’ll explore two distinct aspects of LINQ: the framework support and the compiler translation of query expressions. The latter can look odd to start with, but I’m sure you’ll learn to love them. Query expressions are effectively preprocessed by the compiler into “normal” C# 3, which is then compiled in an ordinary way. This is a neat way of integrating que- ries into the language without changing the specification in more than one small sec- tion. Most of this chapter is a list of the preprocessing translations performed by the compiler, as well as the effects achieved when the result uses the Enumerable exten- sion methods. You won’t see any SQL or XML here—that awaits you in chapter 12. But with this chapter as a foundation, you should be able to understand what the more exciting LINQ providers do when you meet them. Call me a spoilsport, but I want to take away some of their magic. Even without the air of mystery, LINQ is still very cool. First let’s consider the basis of LINQ, and how we’ll go about exploring it. 11.1 Introducing LINQ With a topic as large as LINQ, you need a certain amount of background before you’re ready to see it in action. In this section we’ll look at a few of the core principles behind LINQ and at the data model we’ll use for the examples in this chapter and the next. I know you’re likely to be itching to get into the code, so I’ll keep this fairly brief. 11.1.1 Fundamental concepts in LINQ One of the problems with reducing the impedance mismatch between two data mod- els is that it usually involves creating yet another model to act as the bridge. This sec- tion describes the LINQ model, beginning with its most important aspect: sequences. SEQUENCES You’re already familiar with the concept of a sequence: it’s encapsulated by the IEnumerable and IEnumerable interfaces, and we looked at those fairly closely in chapter 6 when we studied iterators. A sequence is like a conveyor belt of items—you fetch them one at a time until either you’re no longer interested or the sequence runs out of data. The key difference between a sequence and other collection data structures, such as lists and arrays, is that when you’re reading from a sequence, you don’t generally know how many more items are waiting, and you don’t have access to arbitrary items—just the current one. Indeed, some sequences could be never-ending; you could easily have an infinite sequence of random numbers, for example. Lists and arrays can act as sequences, just as List implements IEnumerable, but the reverse isn’t always true. You can’t have an infinite array or list, for example. Sequences are LINQ’s bread and butter. When you read a query expression, you should think about the sequences involved; there’s always at least one sequence to 287Introducing LINQ start with, and it’s usually transformed into other sequences along the way, possibly being joined with yet more sequences. LINQ query examples on the web frequently have little explanation, but when you take them apart by looking at each sequence in turn, things make a lot more sense. As well as being an aid to reading code, this approach can also help a lot when writing it. Thinking in sequences can be tricky—it’s a bit of a mental leap sometimes—but if you can get there, it’ll help you immeasurably when you’re working with LINQ. As a simple example, let’s take a query expression running against a list of people. We’ll apply a filter first, and then a projection, so that we end up with a sequence of the names of adults: var adultNames = from person in people where person.Age >= 18 select person.Name; Figure 11.1 shows this query expression graphically, breaking it down into its individ- ual steps. Each arrow represents a sequence—the description is on the left side and some sample data is on the right. Each box is a step in the query expression. Initially, you have the whole family (as Person objects); then, after filtering, the sequence only contains adults (again, as Person objects); and the final result has the names of those adults as strings. Each time, you take one sequence and apply an operation to produce a new sequence. The result isn’t the strings Holly and Jon—instead, it’s an IEnumerable, which, when asked for its elements one by one, will first yield Holly and then Jon. All Person objects in people All Person objects with an age of at least 18 from person in people Name="Holly", Age=36 Name="Tom", Age=9 Name="Jon", Age=36 Name="William", Age=6 Name="Robin", Age=6 (Result of query) Name="Holly", Age=36 Name="Jon", Age=36 Names of people with an age of at least 18 "Holly" "Jon" where person.Age >= 18 select person.Name Figure 11.1 A simple query expression broken down into the sequences and transformations involved 288 CHAPTER 11 Query expressions and LINQ to Objects This example was straightforward to start with, but we’ll apply the same technique later to more complicated query expressions in order to understand them more easily. Some advanced operations involve more than one sequence as input, but it’s still a lot less to worry about than trying to understand the whole query in one go. And why are sequences so important? They’re the basis for a streaming model for data handling—one that allows you to fetch and process data only when you need it. DEFERRED EXECUTION AND STREAMING When the query expression shown in figure 11.1 is created, no data is processed. The original list of people isn’t accessed at all.1 Instead, a representation of the query is built up in memory. Delegate instances are used to represent the predicate testing for adulthood and the conversion from a person to that person’s name. The wheels only start turning when the resulting IEnumerable is asked for its first element. This aspect of LINQ is called deferred execution. When the first element of the result is requested, the Select transformation asks the Where transformation for its first ele- ment. The Where transformation asks the list for its first element, checks whether the predicate matches (which it does, in this case), and returns that element back to Select. That in turn extracts the name and returns it as the result. HAVEN’T WE SEEN THIS BEFORE? You may be getting a sense of déjà vu here, because I did mention all of this in chapter 10. But it’s such an important topic that it’s worth covering a second time, in more detail. That’s a mouthful, but a sequence diagram makes it all much clearer. I’ll collapse the calls to MoveNext and Current to a single fetch operation; it makes the diagram a lot simpler. Just remember that each time the fetch occurs, it’s effectively checking for the end of the sequence as well. Figure 11.2 shows the first few stages of the sample query expression in operation, when you print out each element of the result using a foreach loop. As you can see in figure 11.2, only one element of data is processed at a time. If you decided to stop printing output after Holly, you’d never execute any of the operations on the other elements of the original sequence. Although several stages are involved here, processing data in a streaming manner like this is efficient and flexible. Regard- less of how much source data there is, you don’t need to know about more than one element at any point in time. This is a best-case scenario. There are times where in order to fetch the first result of a query, you have to evaluate all of the data from the source. We’ve already looked at one example of this in the previous chapter: the Enumerable.Reverse method needs to fetch all the data available in order to return the last original element as the first element of the resulting sequence. This makes Reverse a buffering operation, which can have a huge effect on the efficiency (or even feasibility) of your overall 1 The various parameters involved are checked for nullity, though. This is important to bear in mind if you implement your own LINQ operators, as you’ll see in chapter 12. 289Introducing LINQ operation. If you can’t afford to have all the data in memory at one time, you can’t use buffering operations. Just as streaming depends on which operation you perform, some transformations take place as soon as you call them, rather than using deferred execution. This is called immediate execution. Generally speaking, operations that return another sequence (usually an IEnumerable or IQueryable) use deferred execution, whereas operations that return a single value use immediate execution. The operations that are widely available in LINQ are known as the standard query operators—let’s take a brief look at them now. Caller (foreach) Select Where List Fetch Fetch Fetch Return {"Holly", 36} Check: Age >= 18? Yes Transform: {"Holly", 36} => "Holly" Return "Holly" Fetch Fetch Fetch Print "Holly" Return {"Tom", 9} Check: Age >= 18? No Return {"Jon", 36} Check: Age >= 18? Yes Fetch Transform: {"Jon", 36} => "Jon" Return "Jon" Print "Jon" (and so on) Return {"Holly", 36} Return {"Jon", 36} Figure 11.2 Sequence diagram of the execution of a query expression 290 CHAPTER 11 Query expressions and LINQ to Objects STANDARD QUERY OPERATORS LINQ’s standard query operators are a collection of transformations whose meanings are well understood. LINQ providers are encouraged to implement as many of these oper- ators as possible, making the implementation obey the expected behavior. This is cru- cial in providing a consistent query framework across multiple data sources. Of course, some LINQ providers may expose more functionality, and some of the opera- tors may not map appropriately to the target domain of the provider, but at least the opportunity for consistency is there. IMPLEMENTATION-SPECIFIC DETAILS OF STANDARD OPERATORS Just because the standard query operators have common general meanings doesn’t mean they’ll work exactly the same way for every implementation. For example, some LINQ providers may load the data for a whole query as soon as they need the first item—if you’re accessing a web service, that may make perfect sense. Likewise, a query that works in LINQ to Objects may have subtly differ- ent semantics in LINQ to SQL. This doesn’t mean that LINQ has failed, just that you need to consider which data source you’re accessing when you write a query. There’s still a huge advantage in having a single set of query opera- tors and a consistent query syntax, even though it’s not a panacea. C# 3 has support for some of the standard query operators built into the language via query expressions, but you can always choose to call them manually if you find that makes the code clearer. You may be interested to know that VB9 has more of the oper- ators present in the language; as ever, there’s a trade-off between the added complex- ity of including a feature in the language and the benefits that feature brings. Personally, I think the C# team has done an admirable job; I’ve always been a fan of a relatively small language with a large library behind it. OPERATOR OVERLOADING The term operator is used to describe both query operators (methods such as Select and Where) and the familiar operators such as addition, equality, and so on. Usually it should be obvious which one I mean from the context—if I’m talking about LINQ, operator will almost always refer to a method used as part of a query. You’ll see some of these operators in the examples as we go through this chapter and the next, but I don’t aim to give a comprehensive guide to them here; this book is pri- marily about C#, not the whole of LINQ. You don’t need to know all of the operators in order to be productive in LINQ, but your experience is likely to grow over time. Appen- dix A gives a brief description of each of the standard query operators, and MSDN gives more details of each specific overload. When you run into a problem, check the list: if it feels like there ought to be a built-in method to help you, there probably is! That’s not always the case, though—I founded the MoreLINQ open source project to add some extra operators to LINQ to Objects (see http://code.google.com/p/morelinq/). Like- wise the Reactive Extensions package (see http://mng.bz/R7ip) has additions for the 291Introducing LINQ pull model of LINQ to Objects as well as the push model we’ll look at later. If the stan- dard operators fail you, check both projects before building your own solution. It’s not a disaster if you do have to write your own operator, though; it can be a lot of fun. In chapter 12 I’ll give a few tips on this subject. Having mentioned examples, it’s time I introduced the data model that most of the rest of the sample code in this chapter will use. 11.1.2 Defining the sample data model In section 10.3.4 I gave a brief example of defect tracking as a real use for extension methods and lambda expressions. We’ll use the same idea for almost all of the sample code in this chapter—it’s a fairly simple model, but one that can be manipulated in many different ways to give useful information. Defect tracking is also a domain that most professional developers are all too familiar with, unfortunately. Our fictional setting is SkeetySoft, a small software company with big ambitions. The founders have decided to create an office suite, a media player, and an instant messaging application. After all, there are no big players in those markets, are there? The development department of SkeetySoft consists of five people: two developers (Deborah and Darren), two testers (Tara and Tim), and a manager (Mary). There’s currently a single customer: Colin. The aforementioned products are SkeetyOffice, SkeetyMediaPlayer, and SkeetyTalk, respectively.2 We’ll look at defects logged during May 2013, using the data model shown in figure 11.3. As you can see, there’s not a lot of data being recorded here. In particular, there’s no real history to the defects, but there’s enough here to let you work with the query expression features of C# 3. For the purposes of this chapter, all the data is stored in memory. You have a class named SampleData with properties AllDefects, AllUsers, AllProjects, and All- Subscriptions, which each return an appropriate type of IEnumerable. The Start and End properties return DateTime instances for the start and end of May, respectively, and there are nested classes Users and Projects within SampleData to provide easy access to a particular user or project. The one type that may not be immediately obvious is NotificationSubscription; the idea behind this is to send an email to the specified address every time a defect is created or changed in the rel- evant project. There are 41 defects in the sample data, created using C# 3 object initializers. All of the code is available on the book’s website, along with the sample data. Now that the preliminaries are dealt with, let’s get cracking with some queries! 2 The marketing department of SkeetySoft isn’t particularly creative. 292 CHAPTER 11 Query expressions and LINQ to Objects 11.2 Simple beginnings: selecting elements We’ve already discussed some general LINQ concepts—I’ll introduce the concepts that are specific to C# 3 as they arise in the course of the chapter. We’ll start with a sim- ple query (even simpler than the one you saw earlier) and work up to some compli- cated ones, not only building up your understanding of what the C# 3 compiler is doing, but also teaching you how to read LINQ code. All of the examples will follow the pattern of defining a query and then printing the results to the console. We won’t look at binding queries to data grids or anything like that—it’s all important, but not directly relevant to learning C# 3. You can use a simple expression that prints out all the users as the starting point for examining what the compiler is doing behind the scenes and learning about range variables. Figure 11.3 Class diagram of the SkeetySoft defect data model 293Simple beginnings: selecting elements 11.2.1 Starting with a source and ending with a selection Every query expression in C# 3 starts off in the same way—stating the source of a sequence of data: from element in source The element part is just an identifier, with an optional type name before it. Most of the time you won’t need the type name, and we won’t have one for the first example. The source part is a normal expression. Lots of different things can happen after that first clause, but sooner or later you always end with a select clause or a group clause. We’ll start off with a select clause to keep things nice and simple. The syntax for a select clause is also easy: select expression The select clause is known as a projection. Combining the two together and using a trivial expression gives a simple (and practically useless) query, as shown in the following listing. var query = from user in SampleData.AllUsers select user; foreach (var user in query) { Console.WriteLine(user); } The query expression is the part in bold. I’ve overridden ToString for each of the entities in the model, so the results of listing 11.1 are as follows: User: Tim Trotter (Tester) User: Tara Tutu (Tester) User: Deborah Denton (Developer) User: Darren Dahlia (Developer) User: Mary Malcop (Manager) User: Colin Carton (Customer) You may be wondering how useful this is as an example; after all, you could just use SampleData.AllUsers directly in the foreach statement. But we’ll use this query expression—trivial though it is—to introduce two new concepts. First we’ll look at the general nature of the translation process the compiler uses when it encounters a query expression, and then we’ll discuss range variables. 11.2.2 Compiler translations as the basis of query expressions The C# 3 query expression support is based on the compiler translating query expres- sions into normal C# code. It does this in a mechanical manner that doesn’t try to understand the code, apply type inference, check the validity of method calls, or per- form any of the normal business of a compiler. That’s all done later, after the transla- tion. In many ways, this first phase can be regarded as a preprocessor step. Listing 11.1 Trivial query to print the list of users 294 CHAPTER 11 Query expressions and LINQ to Objects The compiler translates listing 11.1 into the following code before doing the real compilation. var query = SampleData.AllUsers.Select(user => user); foreach (var user in query) { Console.WriteLine(user); } The C# 3 compiler translates the query expression into exactly this code before prop- erly compiling it further. In particular, it doesn’t assume that it should use Enumerable .Select, or that List will contain a method called Select. It merely translates the code and then lets the next phase of compilation deal with finding an appropriate method—whether as a straightforward member or as an extension method.3 The parameter can be a suitable delegate type or an Expression for an appropriate type T. This is where it’s important that lambda expressions can be converted into both delegate instances and expression trees. All the examples in this chapter will use dele- gates, but you’ll see how expression trees are used when we look at the other LINQ providers in chapter 12. When I present the signatures for some of the methods called by the compiler later on, remember that these are just the ones called in LINQ to Objects—whenever the parameter is a delegate type (which most of them are), the compiler will use a lambda expression as the argument and then try to find a method with a suitable signature. It’s also important to remember that wherever a normal variable (such as a local variable within the method) appears within a lambda expression after translation has been performed, it’ll become a captured variable in the same way that you saw back in chapter 5. This is normal lambda expression behavior, but unless you understand which variables will be captured, you could easily be confused by the results of your queries. The language specification gives details of the query expression pattern, which must be implemented for all query expressions to work, but this isn’t defined as an inter- face as you might expect. That makes a lot of sense: it allows LINQ to be applied to interfaces such as IEnumerable using extension methods. This chapter tackles each element of the query expression pattern, one at a time. If you want to see exactly how the language specification defines each translation, see section 7.16 (“Query Expressions”). Listing 11.2 The query expression of listing 11.1 translated into a method call 3 It’s even more general than that—the compiler doesn’t require Select to be a method or SampleData .AllUsers to be a property access. So long as the translated code compiles, it’s happy. In almost every sensi- ble case, you’ll access either standard or extension methods, but I have a blog post with some particularly odd queries that the compiler’s perfectly happy with (see http://mng.bz/7E3i). I haven’t found queries like this to be useful in practice, but I do like this example as a way of hammering home how mechanical the transla- tion process is and how it doesn’t care about the meaning of the translated code. 295Simple beginnings: selecting elements Listing 11.3 illustrates how the compiler translation works: it provides a dummy implementation of both Select and Where, with Select as a normal instance method and Where as an extension method. Our original simple query expression only con- tained a select clause, but this one includes a where clause to show both kinds of methods in use. This is a full listing rather than a snippet as extension methods can only be declared in top-level static classes. static class Extensions { public static Dummy Where(this Dummy dummy, Func predicate) { Console.WriteLine("Where called"); return dummy; } } class Dummy { public Dummy Select(Func selector) { Console.WriteLine("Select called"); return new Dummy(); } } class TranslationExample { static void Main() { var source = new Dummy(); var query = from dummy in source where dummy.ToString() == "Ignored" select "Anything"; } } When you run listing 11.3, it prints Where called and then Select called, just as you’d expect, because the query expression has been translated into this code: var query = source.Where(dummy => dummy.ToString() == "Ignored") .Select(dummy => "Anything"); Of course, you’re not doing any querying or transformation here, but it shows how the compiler is translating the query expression. If you’re puzzled as to why the lambda expression in the Select call returns "Anything" instead of just dummy, it’s because a projection of dummy (which is a do-nothing projection) would be removed by the compiler in this particular case. We’ll look at that in section 11.3.2, but for the moment the important idea is the overall type of translation involved. You only need to learn what translations the C# compiler will use, and then you can take any query Listing 11.3 Compiler translation calling methods on a dummy LINQ implementation Declares Where extension methodB Declares Select instance methodC Creates source to be queried D Calls methods via a query expressionE 296 CHAPTER 11 Query expressions and LINQ to Objects expression, convert it into the form that doesn’t use query expressions, and then look at what it’s doing from that point of view. Note that you don’t implement IEnumerable at all in Dummy. The translation from query expressions to normal code doesn’t depend on it, but in practice most LINQ providers will expose data either as IEnumerable or IQueryable (which we’ll look at in chapter 12). The fact that the translation doesn’t depend on any par- ticular types but merely on the method names and parameters is a sort of compile- time form of duck typing. This is similar to the way that the collection initializers pre- sented in chapter 8 find a public method called Add using normal overload resolution rather than using an interface containing an Add method with a particular signature. Query expressions take this idea one step further—the translation occurs early in the compilation process in order to allow the compiler to pick either instance methods or extension methods. You could even consider the translation to be the work of a sepa- rate preprocessing engine. You may think I’m banging on about this a lot, but it’s all part of removing the mist that sometimes shrouds LINQ. If you rewrite a query expression as a series of method calls, effectively doing what the compiler would’ve done, you won’t change the perfor- mance and your query won’t behave any differently. They’re just two different ways of representing the same code. WHY FROM...WHERE...SELECT INSTEAD OF SELECT...FROM...WHERE? Many devel- opers find the order of the clauses in query expressions confusing to start with. It looks just like SQL—except back to front. If you look back to the trans- lation into methods, you’ll see the main reason behind it. The query expres- sion is processed in the same order that it’s written: you start with a source in the from clause, then filter it in the where clause, and then project it in the select clause. Another way of looking at it is to consider the diagrams throughout this chapter. The data flows from top to bottom, and the boxes appear in the diagram in the same order as their corresponding clauses appear in the query expression. Once you get over any initial discomfort due to unfamiliarity, you may find this approach appealing—I know I do. You may even find yourself asking the equivalent question about SQL. You now know that a source level translation is involved, but there’s another crucial concept to understand before we move on any further. 11.2.3 Range variables and nontrivial projections Let’s look back at this chapter’s original query expression in more depth. We haven’t examined the identifier in the from clause or the expression in the select clause. Figure 11.4 shows the query expression again, with each part labeled to explain its purpose. The contextual keywords are easy to explain—they specify to the compiler what you want to do with the data. Likewise, the source expression is a normal C# expression— a property in this case, but it could just as easily have been a method call or a variable. 297Simple beginnings: selecting elements The tricky bits are the range variable declaration and the projection expres- sion. Range variables aren’t like any other type of variable. In some ways they’re not variables at all! They’re only available in query expressions, and they’re effectively present to propagate context from one expression to another. They represent one element of a particular sequence at a time, and they’re used in the compiler translation to allow other expressions to be turned into lambda expressions easily. You’ve already seen that the original query expression was turned into SampleData.AllUsers.Select(user => user) The left side of the lambda expression—the part that provides the parameter name— comes from the range variable declaration. The right side comes from the select clause. The translation is as simple as that (in this case). It all works out okay because the same name is used on both sides. Suppose you’d written the query expression like this: from user in SampleData.AllUsers select person In that case, the translated version would’ve been SampleData.AllUsers.Select(user => person) At that point, the compiler would’ve complained because it wouldn’t have known what person referred to. Now that you know how simple the process is, it becomes easier to understand a query expression that has a slightly more complicated projection. The following list- ing prints out just the names of our users. IEnumerable query = from user in SampleData.AllUsers select user.Name; foreach (string name in query) { Console.WriteLine(name); } This time you’re using user.Name as the projection, and the result is a sequence of strings, not of User objects. (I’ve used an explicitly typed variable to emphasize this point.) The translation of the query expression follows the same rules as before, and becomes Listing 11.4 Query selecting just the names of the users Query expression contextual keywords Range variable declaration Expression using range variable Source expression (normal code) from user in SampleData.AllUsers select user Figure 11.4 A simple query expression broken down into its constituent parts 298 CHAPTER 11 Query expressions and LINQ to Objects SampleData.AllUsers.Select(user => user.Name) The compiler allows this, because the chosen Select extension method from Enumerable has this signature:4 static IEnumerable Select (this IEnumerable source, Func selector) The type inference described in chapter 9 kicks in, converting the lambda expression into a Func. First it infers that TSource is User due to the type of SampleData.AllUsers. At that point, it knows about the parameter type for the lambda expression, so it can resolve user.Name as a property access expression return- ing type string, thus inferring that TResult is string. This is why lambda expressions allow implicitly typed parameters, and why there are such complicated type inference rules; these are the gears and pistons of the LINQ engine. WHY DO YOU NEED TO KNOW ALL THIS? You can almost ignore what’s going on with range variables a lot of the time. You may have seen many, many queries and understood what they achieve without ever knowing about what’s going on behind the scenes. That’s fine when things are working (as they tend to with examples in tutorials), but when things go wrong, it pays to know about the details. If you have a query expression that won’t compile because the compiler is complaining that it doesn’t know about a particular identifier, you should look at the range variables involved. So far we’ve only looked at implicitly typed range variables. What happens when you include a type in the declaration? The answer lies in the Cast and OfType standard query operators. 11.2.4 Cast, OfType, and explicitly typed range variables Most of the time, range variables can be implicitly typed; you’re likely to be working with generic collections where the specified type is all you need. What if that weren’t the case, though? What if you had an ArrayList, or perhaps an object[] that you wanted to perform a query on? It would be a pity if LINQ couldn’t be applied in those situations. Fortunately, there are two standard query operators that come to the res- cue: Cast and OfType. Only Cast is supported directly by the query expression syntax, but we’ll look at both in this section. The two operators are similar: both take an arbitrary untyped sequence (they’re extension methods on the nongeneric IEnumerable type) and return a strongly typed sequence. Cast does this by casting each element to the target type (and failing on any element that isn’t of the right type), and OfType does a test first, skipping any ele- ments of the wrong type. 4 In order to allow all the methods’ signatures in this chapter to fit on the printed page, I’ve omitted the public modifier. In reality they are all public though. 299Simple beginnings: selecting elements The following listing demonstrates both of these operators, used as simple exten- sion methods from Enumerable. For a change, we won’t use the SkeetySoft defect sys- tem for sample data—after all, that’s all strongly typed! Instead, we’ll use two ArrayList objects. ArrayList list = new ArrayList {"First", "Second", "Third" }; IEnumerable strings = list.Cast(); foreach (string item in strings) { Console.WriteLine(item); } list = new ArrayList { 1, "not an int", 2, 3 }; IEnumerable ints = list.OfType(); foreach (int item in ints) { Console.WriteLine(item); } The first list contains only strings, so it’s safe to use Cast to obtain a sequence of strings. The second list has mixed content, so in order to fetch just the integers from it you use OfType. If you’d used Cast on the second list, an exception would’ve been thrown when you tried to cast “not an int” to int. Note that this would only have happened after you’d returned 1—both operators stream their data, converting elements as they fetch them. IDENTITY, REFERENCE, AND UNBOXING CONVERSIONS ONLY The behavior of Cast changed subtly in .NET 3.5 SP1. In the original .NET 3.5, it would perform more conversions, so using Cast on a List would convert each short into a int as it was fetched. In .NET 3.5 service pack 1 and all later releases, this will throw an exception. If you want any conversion other than a reference conversion or an unboxing conversion (or the no-op identity con- version), use a Select projection instead. OfType only performs these conver- sions too, but it doesn’t throw an exception if they fail. When you introduce a range variable with an explicit type, the compiler uses a call to Cast to make sure the sequence used by the rest of the query expression is of the appropriate type. The following listing shows this, with a projection using the Substring method to prove that the sequence generated by the from clause is a sequence of strings. ArrayList list = new ArrayList {"First", "Second", "Third"}; var strings = from string entry in list select entry.Substring(0, 3); foreach (string start in strings) Listing 11.5 Using Cast and OfType to work with weakly typed collections Listing 11.6 Using an explicitly typed range variable to automatically call Cast 300 CHAPTER 11 Query expressions and LINQ to Objects { Console.WriteLine(start); } The output of listing 11.6 is Fir, Sec, Thi, but what’s more interesting is the translated query expression: list.Cast().Select(entry => entry.Substring(0,3)); Without the cast, you wouldn’t be able to call Select at all, because the extension method is only defined for IEnumerable rather than IEnumerable. Even when you’re using a strongly typed collection, you might still want to use an explicitly typed range variable. For instance, you could have a collection that’s defined to be a List but you know that all the elements are instances of MyImplementation. Using a range variable with an explicit type of MyImplementation allows you to access all the members of MyImplementation without manually insert- ing casts all over the code. We’ve covered a lot of important conceptual ground so far, even though we haven’t achieved any impressive results. To recap the most important points briefly:  LINQ is based on sequences of data, which are streamed wherever possible.  Creating a query doesn’t usually execute it; most operations use deferred execution.  Query expressions in C# 3 involve a preprocessing phase that converts the expression into normal C#, which is then compiled properly with all the normal rules of type inference, overloading, lambda expressions, and so forth.  The variables declared in query expressions don’t act like anything else; they’re range variables that allow you to refer to data consistently within the query expression. I know that there’s been a lot of somewhat abstract information to take in. Don’t worry if you’re beginning to wonder if LINQ is worth all this trouble. I promise you that it is. With a lot of the groundwork out of the way, we can start doing genuinely useful things—such as filtering data and then ordering it. 11.3 Filtering and ordering a sequence You may be surprised to learn that filtering and ordering are two of the simplest oper- ations to explain in terms of compiler translations. This is because they always return a sequence with the same element type as their input, which means you don’t need to worry about any new range variables being introduced. It also helps that you’ve seen the corresponding extension methods in chapter 10. 11.3.1 Filtering using a where clause It’s remarkably easy to understand the where clause. The syntax is just where filter-expression 301Filtering and ordering a sequence The compiler translates this into a call to the Where method with a lambda expression, which uses the appropriate range variable as the parameter and the filter expression as the body. The filter expression is applied as a predicate to each element of the incoming stream of data, and only those that return true are present in the resulting sequence. Using multiple where clauses results in multiple chained Where calls—only ele- ments that match all of the predicates are part of the resulting sequence. The following listing demonstrates a query expression that finds all open defects assigned to Tim. User tim = SampleData.Users.TesterTim; var query = from defect in SampleData.AllDefects where defect.Status != Status.Closed where defect.AssignedTo == tim select defect.Summary; foreach (var summary in query) { Console.WriteLine(summary); } The query expression in listing 11.7 is translated into this: SampleData.AllDefects.Where (defect => defect.Status != Status.Closed) .Where(defect => defect.AssignedTo == tim) .Select(defect => defect.Summary) The output of listing 11.7 is as follows: Installation is slow Subtitles only work in Welsh Play button points the wrong way Webcam makes me look bald Network is saturated when playing WAV file Of course, you could write a single where clause that combined the two conditions as an alternative to using multiple where clauses. In some cases this might improve perfor- mance, but it’s worth bearing the readability of the query expression in mind too, and this is likely to be fairly subjective. My personal inclination is to combine conditions that are logically related but keep others separate. In this case both parts of the expression deal directly with a defect (as that’s all our sequence contains), so it’d be reasonable to combine them. As before, it’s worth trying both forms to see which is clearer. In a moment you’ll start applying some ordering rules to the query, but first we should look at a small detail to do with the select clause. 11.3.2 Degenerate query expressions While we have a fairly simple translation to work with, let’s revisit a point I glossed over earlier in section 11.2.2 when I first introduced the compiler translations. So far, all our translated query expressions have included a call to Select. What happens if Listing 11.7 Query expression using multiple where clauses 302 CHAPTER 11 Query expressions and LINQ to Objects the select clause does nothing, effectively returning the same sequence it’s given? The answer is that the compiler removes that call to Select, but only if there are other operations being performed within the query expression. For example, the following query expression just selects all the defects in the system: from defect in SampleData.AllDefects select defect This is known as a degenerate query expression. The compiler deliberately generates a call to Select even though it seems to do nothing: SampleData.AllDefects.Select(defect => defect) There’s a big difference between this and using SampleData.AllDefects as a simple expression, though. The items returned by the two sequences are the same, but the result of the Select method is just the sequence of items, not the source itself. The result of a query expression is never the same object as the source data, unless the LINQ provider has been poorly coded. This can be important from a data integrity point of view—a provider can return a mutable result object, knowing that changes to the returned data sequence won’t affect the master, even in the face of a degenerate query. When other operations are involved, there’s no need for the compiler to keep no-op select clauses. For example, suppose you change the query expression in listing 11.7 to select the whole defect rather than just the name: from defect in SampleData.AllDefects where defect.Status != Status.Closed where defect.AssignedTo == SampleData.Users.TesterTim select defect You now don’t need the final call to Select, so the translated code is just this: SampleData.AllDefects.Where(defect => defect.Status != Status.Closed) .Where(defect => defect.AssignedTo == tim) These rules rarely get in the way when you’re writing query expressions, but they can cause confusion if you decompile the code with a tool such as Reflector—it can be sur- prising to see the Select call go missing for no apparent reason. With that knowledge in hand, it’s time to improve the query so that you know what Tim should work on next. 11.3.3 Ordering using an orderby clause It’s not uncommon for developers and testers to be asked to work on the most critical defects before they tackle more trivial ones. You can use a simple query to tell Tim the order in which he should tackle the open defects assigned to him. The following list- ing does exactly this using an orderby clause, printing out all the details of the defects in descending order of priority. 303Filtering and ordering a sequence User tim = SampleData.Users.TesterTim; var query = from defect in SampleData.AllDefects where defect.Status != Status.Closed where defect.AssignedTo == tim orderby defect.Severity descending select defect; foreach (var defect in query) { Console.WriteLine("{0}: {1}", defect.Severity, defect.Summary); } The output of listing 11.8 shows that you’ve sorted the results appropriately: Showstopper: Webcam makes me look bald Major: Subtitles only work in Welsh Major: Play button points the wrong way Minor: Network is saturated when playing WAV file Trivial: Installation is slow You have two major defects. Which order should those be tackled in? Currently no clear ordering is involved. Let’s change the query so that after sorting by severity in descending order, you sort by last modified time in ascending order. This means that Tim will test the defects that were fixed a long time ago before those addressed more recently. This just requires an extra expression in the orderby clause, as shown in the following listing. User tim = SampleData.Users.TesterTim; var query = from defect in SampleData.AllDefects where defect.Status != Status.Closed where defect.AssignedTo == tim orderby defect.Severity descending, defect.LastModified select defect; foreach (var defect in query) { Console.WriteLine("{0}: {1} ({2:d})", defect.Severity, defect.Summary, defect.LastModified); } The results of listing 11.9 are shown here. Note how the order of the two major defects has been reversed: Showstopper: Webcam makes me look bald (05/27/2013) Major: Play button points the wrong way (05/17/2013) Major: Subtitles only work in Welsh (05/23/2013) Minor: Network is saturated when playing WAV file (05/31/2013) Trivial: Installation is slow (05/15/2013) Listing 11.8 Sorting by the severity of a defect, from high to low priority Listing 11.9 Ordering by severity and then last modified time 304 CHAPTER 11 Query expressions and LINQ to Objects That’s what the query expression looks like, but what does the compiler do? It simply calls the OrderBy and ThenBy methods (or OrderByDescending/ThenByDescending for descending orders). Your query expression is translated into this: SampleData.AllDefects.Where(defect => defect.Status != Status.Closed) .Where(defect => defect.AssignedTo == tim) .OrderByDescending(defect => defect.Severity) .ThenBy(defect => defect.LastModified) Now that you’ve seen an example, we can look at the general syntax of orderby clauses. They’re basically the contextual keyword orderby followed by one or more orderings. An ordering is just an expression (which can use range variables) optionally followed by ascending or descending, which have the obvious meanings. (The default order is ascending.) The translation for the primary ordering is a call to OrderBy or OrderByDescending, followed by as many calls to ThenBy or ThenBy- Descending as you have subsequent orderings. The difference between OrderBy and ThenBy is simple: OrderBy assumes it has pri- mary control over the ordering, whereas ThenBy understands that it’s subservient to one or more previous orderings. For LINQ to Objects, ThenBy is only defined as an extension method for IOrderedEnumerable, which is the type returned by OrderBy (and by ThenBy itself, to allow further chaining). It’s important to note that although you can use multiple orderby clauses, each one will start with its own OrderBy or OrderByDescending clause, which means the last one will effectively win. I’ve yet to see a situation in which this is useful unless you do something else to the query between orderby clauses; you should almost always use a single clause containing multiple orderings instead. As noted in chapter 10, applying an ordering requires all the data to be loaded (at least for LINQ to Objects)—you can’t order an infinite sequence, for example. Hope- fully the reason for this is obvious: you don’t know which element should come at the start of the resulting sequence until you’ve seen all the elements. We’re about halfway through learning about query expressions, and you may be surprised that we haven’t looked at any joins yet. Obviously they’re important in LINQ, just as they’re important in SQL, but they’re also complicated. I promise we’ll get to them in due course, but in order to introduce just one new concept at a time, we’ll detour through let clauses first. That way we can discuss transparent identifiers before we hit joins. 11.4 Let clauses and transparent identifiers Most of the rest of the operators we still need to look at involve transparent identifiers. Just like range variables, you can get along perfectly well without understanding trans- parent identifiers if you only want to have a fairly shallow grasp of query expressions. If you’ve bought this book, though, you probably want to know C# at a deeper level, which will (among other things) enable you to look compilation errors in the face and know what they’re talking about. 305Let clauses and transparent identifiers You don’t need to know everything about transparent identifiers, but I’ll teach you enough so that if you see one in the language specification, you won’t feel like run- ning and hiding. You’ll also understand why they’re needed at all—and that’s where an example will come in handy. The let clause is the simplest transformation avail- able that uses transparent identifiers. 11.4.1 Introducing an intermediate computation with let A let clause introduces a new range variable with a value that can be based on other range variables. The syntax is as easy as pie: let identifier = expression To explain this operator in terms that don’t use any other complicated operators, I’ll resort to a very artificial example. Suspend your disbelief, and imagine that finding the length of a string is a costly operation. Now imagine that you have a completely bizarre system requirement to order your users by the lengths of their names and then display the name and its length. Yes, I know it’s unlikely. The following listing shows one way of doing this without a let clause. var query = from user in SampleData.AllUsers orderby user.Name.Length select user.Name; foreach (var name in query) { Console.WriteLine("{0}: {1}", name.Length, name); } That works fine, but it uses the dreaded Length property twice—once to sort the users, and once in the display side. Surely not even the fastest supercomputer could cope with finding the lengths of six strings twice! No, you need to avoid that redundant computation. You can do so with the let clause, which evaluates an expression and introduces it as a new range variable. The following code achieves the same result as listing 11.10, but only uses the Length property once per user. var query = from user in SampleData.AllUsers let length = user.Name.Length orderby length select new { Name = user.Name, Length = length }; foreach (var entry in query) { Console.WriteLine("{0}: {1}", entry.Length, entry.Name); } Listing 11.10 Sorting by the lengths of user names without a let clause Listing 11.11 Using a let clause to remove redundant calculations 306 CHAPTER 11 Query expressions and LINQ to Objects Listing 11.11 introduces a new range variable called length, which contains the length of the user’s name (for the current user in the original sequence). You then use that new range variable both for sorting and the projection at the end. Have you spotted the problem yet? You need to use two range variables, but the lambda expres- sion passed to Select only takes one parameter! This is where transparent identifiers come on the scene. 11.4.2 Transparent identifiers In listing 11.11 you have two range variables involved in the final projection, but the Select method only acts on a single sequence. How can you combine the range variables? The answer is to create an anonymous type that contains both variables, and then to apply a clever translation to make it look as if you actually have two parameters for the select and orderby clauses. Figure 11.5 shows the sequences involved. The let clause achieves its objectives by using another call to Select, creating an anonymous type for the resulting sequence, and effectively creating a new range from user in SampleData.AllUsers User: { Name="Tim Trotter" ... } User: { Name="Tara Tutu" ... } User: { Name="Dave Denton" ... } ... user=User: { Name="Tim Trotter" ... }, length=11 user=User: { Name="Tara Tutu" ... }, length=9 user=User: { Name="Dave Denton" ... }, length=11 ... user=User: { Name="Tara Tutu" ... }, length=9 user=User: { Name="Tim Trotter" ...}, length=11 user=User: { Name="Dave Denton" ... }, length=11 ... (Result of query) { Name="Tara Tutu", Length=9 } { Name="Tim Trotter", Length=11 } { Name="Dave Denton", Length=11 } ... Two range variables: user and length Same sequence, ordered by length Name and length in an anonymous type let length = user.Name.Length orderby length { Name "Tara Tutu select new { Name=user.Name, Length=length } Figure 11.5 Sequences involved in listing 11.11, where a let clause introduces the length range variable 307Joins variable whose name can never be seen or used in source code. The query expression from listing 11.11 is translated into something like this: SampleData.AllUsers .Select(user => new { user, length = user.Name.Length }) .OrderBy(z => z.length) .Select(z => new { Name = z.user.Name, Length = z.length }) Each part of the query has been adjusted appropriately: where the original query expression referenced user or length directly, if the reference occurs after the let clause, it’s replaced by z.user or z.length. The choice of z as the name here is arbi- trary—it’s all hidden by the compiler. ANONYMOUS TYPES ARE AN IMPLEMENTATION DETAIL Strictly speaking, it’s up to the C# compiler implementation to decide how to group together differ- ent range variables to make transparent identifiers work. The Microsoft implementation uses anonymous types, and the specification shows the trans- lations in those terms as well, so I’ve followed the trend. Even if another com- piler chose a different approach, it shouldn’t affect the results. If you consult the language specification about let clauses (section 7.16.2.4), you’ll see that the translation it describes is from one query expression to another. It uses an asterisk (*) to represent the transparent identifier introduced. The transparent identi- fier is then erased as a final step in translation. I won’t use that notation in this chapter, as it’s hard to come to grips with and unnecessary at the level of detail we’re going into. Hopefully with this background the specification won’t be quite as impenetrable as it might be otherwise, should you need to refer to it. The good news is that we can now look at the rest of the translations making up C# 3’s query expression support. I won’t go into the details of every transparent iden- tifier introduced, but I’ll mention the situations in which they occur. Let’s look at the support for joins first. 11.5 Joins If you’ve ever read anything about SQL, you probably have an idea what a database join is. It takes two tables (or views, or table-valued functions, and so forth) and creates a result by matching one set of rows against another set of rows. A LINQ join is similar, except it works on sequences. Three types of join are available, although not all of them use the join keyword in the query expression. We’ll start with the join that’s closest to a SQL inner join. 11.5.1 Inner joins using join clauses Inner joins involve two sequences. One key selector expression is applied to each ele- ment of the first sequence, and another key selector (which may be totally different) is applied to each element of the second sequence. The result of the join is a sequence of all the pairs of elements where the key from the first element is the same as the key from the second element. 308 CHAPTER 11 Query expressions and LINQ to Objects TERMINOLOGY CLASH! INNER AND OUTER SEQUENCES The MSDN documentation for the Join method used to evaluate inner joins calls the sequences involved inner and outer, and the real method parameters are based on these names too. This has nothing to do with inner joins and outer joins—it’s just a way of differentiating between the sequences. You can think of them as first and sec- ond, left and right, Bert and Ernie—anything you like that helps you. I’ll use left and right for this chapter, so that it’s clear which is which in the diagrams. Usually, outer corresponds with left and inner corresponds with right. The two sequences can be anything you like; the right sequence can even be the same as the left sequence, if that’s useful. (Imagine finding pairs of people who were born on the same day, for example.) The only thing that matters is that the two key selector expressions must result in the same type of key.5 You can’t join a sequence of people to a sequence of cities by saying that the birth date of the person is the same as the population of the city—it doesn’t make any sense. But one important possibility is to use an anonymous type for the key; this works because anonymous types implement equality and hashing appropriately. If you need to effectively create a multicolumn key, anonymous types are the way to go. This is also applicable for the grouping operations we’ll look at later. The syntax for an inner join looks more complicated than it is: [query selecting the left sequence] join right-range-variable in right-sequence on left-key-selector equals right-key-selector Seeing equals as a contextual keyword rather than using symbols can be disconcert- ing, but it makes it easier to distinguish the left key selector from the right key selec- tor. Often (but not always) at least one of the key selectors is a trivial one that just selects the exact element from that sequence. The contextual keyword is used by the compiler to separate the key selectors into different lambda expressions. The query processor’s ability to obtain the keys for each value (on each side of the join) is impor- tant both for performance in LINQ to Objects and for the feasibility of translating the query into other forms, such as SQL. Let’s look at an example from our defect system. Suppose you’ve just added the notification feature and want to send the first batch of emails for all the existing defects. You need to join the list of notifications against the list of defects where their projects match. The following listing performs just such a join. var query = from defect in SampleData.AllDefects join subscription in SampleData.AllSubscriptions on defect.Project equals subscription.Project 5 It’s also valid for there to be two key types involved, with an implicit conversion from one to the other. One of the types must be a better choice than the other, in the same way that the compiler infers the type of an implicitly typed array. In my experience, you rarely need to consciously consider this detail. Listing 11.12 Joining the defects and notification subscriptions based on project 309Joins select new { defect.Summary, subscription.EmailAddress }; foreach (var entry in query) { Console.WriteLine("{0}: {1}", entry.EmailAddress, entry.Summary); } Listing 11.12 will show each of the media player defects twice—once for mediabugs @skeetysoft.com and once for theboss@skeetysoft.com (because the boss really cares about the media player project). In this particular case, you could easily have made the join the other way around, reversing the left and right sequences. The result would’ve been the same entries but in a different order. The implementation in LINQ to Objects returns entries such that all the pairs using the first element of the left sequence are returned (in the order of the right sequence), then all the pairs using the second element of the left sequence, and so on. The right sequence is buffered, but the left sequence is streamed, so if you want to join a massive sequence to a tiny one, it’s worth using the tiny one as the right sequence if you can. The operation is still deferred: it waits until you ask for the first pair before reading any data from either sequence. At that point, it reads the entirety of the right sequence in order to build a lookup from keys to the values producing those keys. After that, it doesn’t need to read from the right sequence again, and can begin to iterate over the left sequence, yielding pairs appropriately. One error that might trip you up is putting the key selectors the wrong way around. In the left key selector, only the left sequence range variable is in scope; in the right key selector, only the right range variable is in scope. If you reverse the left and right sequences, you have to reverse the left and right key selectors too. Fortunately, the compiler knows that this is a common mistake and suggests the appropriate course of action. Just to make it more obvious what’s going on, figure 11.6 shows the sequences as they’re processed. Often you’ll want to filter the sequence, and filtering before the join occurs is more efficient than filtering it afterward. At this stage, the query expression is simpler if the left sequence is the one requiring filtering. For instance, if you wanted to show only defects that are closed, you could use this query expression: from defect in SampleData.AllDefects where defect.Status == Status.Closed join subscription in SampleData.AllSubscriptions on defect.Project equals subscription.Project select new { defect.Summary, subscription.EmailAddress } You can perform the same query with the sequences reversed, but it’s messier: from subscription in SampleData.AllSubscriptions join defect in (from defect in SampleData.AllDefects where defect.Status == Status.Closed select defect) on subscription.Project equals defect.Project select new { defect.Summary, subscription.EmailAddress } 310 CHAPTER 11 Query expressions and LINQ to Objects Note how you can use one query expression inside another—the language specifica- tion describes many of the compiler translations in these terms. Nested query expres- sions are useful but hurt readability as well; it’s often worth looking for an alternative, or using a variable for the sequence on the right in order to make the code clearer. ARE INNER JOINS USEFUL IN LINQ TO OBJECTS? Inner joins are used all the time in SQL. They’re effectively the way that you navigate from one entity to a related one, usually joining a foreign key in one table to the primary key in another. In the object-oriented model, you tend to navigate from one object to another via references. For instance, retrieving the summary of a defect and the name of the user assigned to work on it would require a join in SQL— in C# you often just use a chain of properties. If you’d had a reverse associa- tion from Project to the list of NotificationSubscription objects associ- ated with it in the model, you wouldn’t have needed the join to achieve the Defect: { ID=1, Project=Media player, Summary="MP3 files ..." ... } Defect: { ID=2, Project=Media player, Summary="Text is too big", ...} Defect: { ID=3, Project=Talk, Summary="Sky is wrong ..." ...} ... defect = { ID=1 ... }, subscription = { Media player, "media-bugs@..." } defect = { ID=1 ... }, subscription = { Media player, "theboss@..." } defect = { ID=2 ... }, subscription = { Media player, "media-bugs@..." } defect = { ID=2 ... }, subscription = { Media player, "theboss@..." } defect = { ID=3 ... }, subscription = { Talk, "talk-bugs@..." } { Summary="MP3 files ...", EmailAddress="media-bugs@..." } { Summary="MP3 files ...", EmailAddress="theboss@..." } { Summary="Text is too big", EmailAddress="media-bugs@..." } { Summary="Text is too big", EmailAddress="theboss@... } { Summary="Sky is wrong ...", EmailAddress="talk-bugs@..." } (Result of query) NotificationSubscription sequence: { Media player, "media-bugs@..." }, { Talk, "talk-bugs@..." }, { Office, "office-bugs@..." } { Media player, "theboss@..."} All known defects All subscriptions Defects and subscriptions in two range variables; each can appear multiple times if the join matches more than once The same sequence projected into an anonymous type from defect in SampleData.AllDefects join subscription in SampleData.AllSubscriptions on defect.Project equals subscription.Project from subscription in SampleData.AllSubscriptions select new { defect.Summary, subscription.EmailAddress } Figure 11.6 The join from listing 11.12 in graphical form, showing two different sequences (defects and subscriptions) used as data sources 311Joins goal of this example, either. That’s not to say that inner joins aren’t some- times useful within object-oriented models, but they don’t naturally occur as often as in relational models. Inner joins are translated by the compiler into calls to the Join method, like this: leftSequence.Join(rightSequence, leftKeySelector, rightKeySelector, resultSelector) The signature of the overload used for LINQ to Objects is as follows (this is the real sig- nature, with the real parameter names—hence the inner and outer references): static IEnumerable Join ( this IEnumerable outer, IEnumerable inner, Func outerKeySelector, Func innerKeySelector, Func resultSelector ) The first three parameters are self-explanatory when you’ve remembered to treat inner and outer as right and left, respectively, but the last one is more interesting. It’s a projec- tion from two elements (one from the left sequence and one from the right sequence) into a single element of the resulting sequence. When the join is followed by anything other than a select clause, the C# 3 com- piler introduces a transparent identifier in order to make the range variables used in both sequences available for later clauses, and creates an anonymous type and a sim- ple mapping to use for the resultSelector parameter. But if the next part of the query expression is a select clause, the projection from the select clause is used directly as the resultSelector parameter—there’s no point in creating a pair and then calling Select when you can do the transformation in one step. You can still think about it as a “join” step followed by a “select” step, despite the two being squished into a single method call. This leads to a more consistent mental model in my view, and one that’s easier to reason about. Unless you’re looking at the generated code, just ignore the optimization the compiler is performing for you. The good news is that, having learned about inner joins, you’ll find the next type of join much easier to approach. 11.5.2 Group joins with join...into clauses You’ve seen that the result sequence from a normal join clause consists of pairs of ele- ments, one from each of the input sequences. A group join looks similar in terms of the query expression but has a significantly different outcome. Each element of a group join result consists of an element from the left sequence (using its original range vari- able) and a sequence of all the matching elements of the right sequence, exposed as a new range variable specified by the identifier coming after into in the join clause. 312 CHAPTER 11 Query expressions and LINQ to Objects Let’s change the previous example to use a group join. The following listing again shows all the defects and the notifications required for each of them, but it breaks them out in a per-defect manner. Pay particular attention to how the results are dis- played with a nested foreach loop. var query = from defect in SampleData.AllDefects join subscription in SampleData.AllSubscriptions on defect.Project equals subscription.Project into groupedSubscriptions select new { Defect = defect, Subscriptions = groupedSubscriptions }; foreach (var entry in query) { Console.WriteLine(entry.Defect.Summary); foreach (var subscription in entry.Subscriptions) { Console.WriteLine (" {0}", subscription.EmailAddress); } } The Subscriptions property of each entry is the embedded sequence of subscrip- tions matching that entry’s defect. Figure 11.7 shows how the two initial sequences are combined. One important difference between an inner join and a group join—and between a group join and a normal grouping—is that a group join has a one-to-one correspon- dence between the left sequence and the result sequence, even if some of the ele- ments in the left sequence don’t match any elements of the right sequence. This can be important and is sometimes used to simulate a left outer join from SQL. The embed- ded sequence is empty when the left element doesn’t match any right elements. As with an inner join, a group join buffers the right sequence but streams the left one. Listing 11.14 shows an example of this, counting the number of defects created on each day in May. It uses a DateTimeRange type to generate a sequence of dates in May as the left sequence, and a projection that calls Count() on the embedded sequence in the result of the group join.6 var dates = new DateTimeRange(SampleData.Start, SampleData.End); var query = from date in dates join defect in SampleData.AllDefects on date equals defect.Created.Date into joined select new { Date = date, Count = joined.Count() }; Listing 11.13 Joining defects and subscriptions with a group join Listing 11.14 Counting the number of defects raised on each day in May 6 This is a simple implementation for the sake of the example—not a full-blown, general-purpose range. 313Joins foreach (var grouped in query) { Console.WriteLine("{0:d}: {1}", grouped.Date, grouped.Count); } The Count() method uses immediate execution, iterating through all the elements of the sequence it’s called on—but you’re only calling it in the projection part of the query expression, so it becomes part of a lambda expression. This means you still have deferred execution; nothing is evaluated until you start the foreach loop. Defect: { ID=1, Project=Media player, ... } Defect: { ID=2, Project=Media player, ...} Defect: { ID=3, Project=Talk, ...} ... (Result of query) defect = Defect: { ID=1, Project=Media player ... }, groupedSubscriptions= { Media player, "media-bugs@..." ... } { Media player, "theboss@..." ...} defect = Defect: { ID=2, Project=Media player ... }, groupedSubscriptions= defect = Defect: { ID=3, Project=Talk ... }, groupedSubscriptions= { Talk, "talk-bugs@..." ... } ... (NotificationSubscription sequence) { Media player, "media-bugs@..." ... } { Media player, "theboss@..." ...} { Defect=Defect: { ID=1, Project=Media player ... }, Subscriptions= { Media player, "media-bugs@..." ... } { Media player, "theboss@..." ...} { Defect=Defect: { ID=2, Project=Media player ... }, Subscriptions= { Defect=Defect: { ID=3, Project=Talk ... }, Subscriptions= { Talk, "talk-bugs@..." ... } ... { Media player, "media-bugs@..." ... } { Media player, "theboss@..." ...} df f { 1 join subscription in SampleData.AllSubscriptions on defect.Project equals subscription.Project into groupedSubscriptions from subscription in SampleData.AllSubscriptions from defect in SampleData.AllDefects f f select new { Defect = defect, Subscriptions = groupedSubscriptions } Two range variables: defect is only a single defect per sequence entry, but groupedSubscriptions has all the matching subscriptions The same sequence projected into an anonymous type All known defects Figure 11.7 Sequences involved in the group join from listing 11.13. The short arrows indicate embedded sequences within the result entries. In the output, some entries contain multiple email addresses for the same defect. 314 CHAPTER 11 Query expressions and LINQ to Objects Here’s the first part of the results of listing 11.14, showing the number of defects cre- ated each day in the first week of May: 05/01/2013: 1 05/02/2013: 0 05/03/2013: 2 05/04/2013: 1 05/05/2013: 0 05/06/2013: 1 05/07/2013: 1 The compiler translation involved for a group join is simply a call to the GroupJoin method in the same way that an inner join translates to a call to Join. Here’s the signa- ture for Enumerable.GroupJoin: static IEnumerable GroupJoin( this IEnumerable outer, IEnumerable inner, Func outerKeySelector, Func innerKeySelector, Func,TResult> resultSelector ) This is exactly the same as for inner joins, except that the resultSelector parameter has to work with a sequence of right-hand elements, not just a single one. As with inner joins, if a group join is followed by a select clause, the projection is used as the result selector of the GroupJoin call; otherwise, a transparent identifier is introduced. In this case you have a select clause immediately after the group join, so the trans- lated query looks like this: dates.GroupJoin(SampleData.AllDefects, date => date, defect => defect.Created.Date, (date, joined) => new { Date = date, Count = joined.Count() }) The final type of join is known as a cross join, but it’s not as straightforward as it might initially seem. 11.5.3 Cross joins and flattening sequences using multiple from clauses So far all our joins have been equijoins—a match has been performed between ele- ments of the left and right sequences. Cross joins don’t perform any matching between the sequences; the result contains every possible pair of elements. This is achieved by simply using two (or more) from clauses. For the sake of sanity, we’ll only consider two from clauses for the moment—when there are more, just mentally per- form a cross join on the first two from clauses, then cross join the resulting sequence with the next from clause, and so on. Each extra from clause adds its own range vari- able via a transparent identifier. 315Joins The following listing shows a simple (but useless) cross join in action, producing a sequence where each entry consists of a user and a project. I’ve deliberately picked two completely unrelated initial sequences to show that no matching is performed. var query = from user in SampleData.AllUsers from project in SampleData.AllProjects select new { User = user, Project = project }; foreach (var pair in query) { Console.WriteLine("{0}/{1}", pair.User.Name, pair.Project.Name); } The output of listing 11.15 begins like this: Tim Trotter/Skeety Media Player Tim Trotter/Skeety Talk Tim Trotter/Skeety Office Tara Tutu/Skeety Media Player Tara Tutu/Skeety Talk Tara Tutu/Skeety Office Figure 11.8 shows the sequences involved to get this result. If you’re familiar with SQL, you’re probably comfortable so far—it looks just like a Cartesian product obtained from a query specifying multiple tables. But more power is available when you want it: the right sequence can depend on the current value of the left sequence. In other words, each element of the left sequence is used to gener- ate a right sequence, and then that left element is paired with each element of the new sequence. When this is the case, it’s not a cross join in the normal sense of the term. Instead, it’s effectively flattening a sequence of sequences into one single sequence. The query expression translation is the same whether or not you’re using a true cross join, so you need to understand the more complicated scenario in order to understand the translation process. Before we dive into the details, let’s see the effect it produces. The following listing shows a simple example, using sequences of integers. var query = from left in Enumerable.Range(1, 4) from right in Enumerable.Range(11, left) select new { Left = left, Right = right }; foreach (var pair in query) { Console.WriteLine("Left={0}; Right={1}", pair.Left, pair.Right); } Listing 11.15 Cross joining users against projects Listing 11.16 Cross join where the right sequence depends on the left element 316 CHAPTER 11 Query expressions and LINQ to Objects Listing 11.16 starts with a simple range of integers, 1 to 4. For each of those integers, you create another range, beginning at 11 and having as many elements as the origi- nal integer. By using multiple from clauses, the left sequence is joined with each of the generated right sequences, resulting in this output: Left=1; Right=11 Left=2; Right=11 Left=2; Right=12 Left=3; Right=11 Left=3; Right=12 Left=3; Right=13 Left=4; Right=11 Left=4; Right=12 Left=4; Right=13 Left=4; Right=14 User: { Name="Tim Trotter" ... } User: { Name="Tara Tutu" ... } User: { Name="Dave Denton" ... } User: { Name="Darren Dahlia" ...} User: { Name="Mary Malcop" ...} User: { Name="Colin Carton" ...} user = User (Tim Trotter), project = Project (Media Player) user = User (Tim Trotter), project = Project (Talk) user = User (Tim Trotter), project = Project (Office) user = User (Tara Tutu), project = Project (Media Player) user = User (Tara Tutu), project = Project (Talk) user = User (Tara Tutu), project = Project (Office) ... (Result of query) Project: { Name="Skeety Media Player } Project: { Name="Skeety Talk" } Project: { Name="Skeety Office" } { User = User (Tim Trotter), Project = Project (Media Player) { User = User (Tim Trotter), Project = Project (Talk) } { User = User (Tim Trotter), Project = Project (Office) } { User = User (Tara Tutu), Project = Project (Media Player) } { User = User (Tara Tutu), Project = Project (Talk) } { User = User (Tara Tutu), Project = Project (Office) } ... from user in SampleData.AllUsers from project in SampleData.AllProjects } SampleData.AllProjects select new { User = user, Project = project } All users Two range variables: each project occurs with each user, pair-wise The same sequence projected into an anonymous type Figure 11.8 Sequences from listing 11.15, cross joining users and projects. All possible combinations are returned in the results. 317Joins The method the compiler calls to generate this sequence is SelectMany. It takes a sin- gle input sequence (the left sequence in our terminology), a delegate to generate another sequence from any element of the left sequence, and a delegate to generate a result element given an element of each of the sequences. Here’s the signature of Enumerable.SelectMany: static IEnumerable SelectMany( this IEnumerable source, Func> collectionSelector, Func resultSelector ) As with the other joins, if the part of the query expression following the join is a select clause, that projection is used as the final argument; otherwise, a transparent identifier is introduced to make the range variables of both the left and right sequences available later in the query. Just to make this all a bit more concrete, here’s the query expression of listing 11.16 as the translated source code: Enumerable.Range(1, 4) .SelectMany(left => Enumerable.Range(11, left), (left, right) => new {Left = left, Right = right}) One interesting feature of SelectMany is that the execution is completely streamed— it only needs to process one element of each sequence at a time, because it uses a freshly generated right sequence for each different element of the left sequence. Compare this with inner joins and group joins: they both load the right sequence completely before starting to return any results. The flattening behavior of SelectMany can be very useful. Consider a situation where you want to process a lot of log files, a line at a time. You can process a seamless sequence of lines with barely any work. The following pseudocode is filled in more thoroughly in the downloadable source code, but the overall meaning and usefulness should be clear: var query = from file in Directory.GetFiles(logDirectory, "*.log") from line in ReadLines(file) let entry = new LogEntry(line) where entry.Type == EntryType.Error select entry; In just five lines of code, you can retrieve, parse, and filter a whole collection of log files, returning a sequence of entries representing errors. Crucially, you don’t have to load even a single full log file into memory in one go, let alone all of the files—all the data is streamed. Having tackled joins, the last items we need to look at are slightly easier to under- stand. We’ll look at grouping elements by a key and continuing a query expression after a group ... by or select clause. 318 CHAPTER 11 Query expressions and LINQ to Objects 11.6 Groupings and continuations One common requirement is to group a sequence of elements by one of its proper- ties. LINQ makes this easy with the group ... by clause. In addition to describing this final type of clause in this section, we’ll also revisit select to see a feature called query continuations that can be applied to both groupings and projections. Let’s start with a simple grouping. 11.6.1 Grouping with the group...by clause Grouping is largely intuitive, and LINQ makes it simple. To group a sequence in a query expression, all you need to do is use the group ... by clause, with this syntax: group projection by grouping This clause comes at the end of a query expression in the same way a select clause does. The similarities between these clauses don’t end there: the projection expres- sion is the same kind of projection that select clauses use. The outcome is somewhat different, though. The grouping expression determines what the sequence is grouped by—it’s the key selector of the grouping operation. The overall result is a sequence where each element is a group. Each group is a sequence of projected elements that also has a Key property, which is the key for that group; this combination is encapsulated in the IGrouping interface, which extends IEnumerable. Again, if you want to group by multiple values, you can use an anonymous type for the key. Let’s look at a simple example from the SkeetySoft defect system: grouping defects by their current assignee. The following listing does this with the simplest form of pro- jection, so that the resulting sequence has the assignee as the key and a sequence of defects embedded in each entry. var query = from defect in SampleData.AllDefects where defect.AssignedTo != null group defect by defect.AssignedTo; foreach (var entry in query) { Console.WriteLine(entry.Key.Name); foreach (var defect in entry) { Console.WriteLine(" ({0}) {1}", defect.Severity, defect.Summary); } Console.WriteLine(); } Listing 11.17 Grouping defects by assignee—trivial projection Filters out unassigned defects B Groups by assigneeC Uses key of each entry: the assignee D Iterates over entry’s subsequenceE 319Groupings and continuations Listing 11.17 might be useful in a daily build report, to quickly see what defects each person needs to look at. It filters out all the defects that don’t need any more atten- tion B and then groups using the AssignedTo property. Although this time you’re just using a property, the grouping expression can be anything you like—it’s applied to each entry in the incoming sequence, and the sequence is grouped based on the result of the expression. Note that grouping can’t stream the results; it applies the key selection and projection to each element in the input and buffers the grouped sequences of projected elements. But even though it’s not streamed, execution is still deferred until you start retrieving the results. The projection applied in the grouping C is trivial—it just selects the original ele- ment. As you go through the resulting sequence, each entry has a Key property, which is of type User D, and each entry also implements IEnumerable, which is the sequence of defects assigned to that user E. The results of listing 11.17 start like this: Darren Dahlia (Showstopper) MP3 files crash system (Major) Can't play files more than 200 bytes long (Major) DivX is choppy on Pentium 100 (Trivial) User interface should be more caramelly After all of Darren’s defects have been returned, you’ll see Tara’s, then Tim’s, and so on. The implementation effectively keeps a list of the assignees it’s seen so far, and adds a new one every time it needs to. Figure 11.9 shows the sequences generated throughout the query expression, which may make this ordering more clear. Within each entry’s subsequence, the order of the defects is the same as in the original defect sequence. If you actively care about the ordering, consider explicitly stating it in the query expression, to make it more readable. If you run listing 11.17, you’ll see that Mary Malcop doesn’t appear in the output at all, because she doesn’t have any defects assigned to her. If you wanted to produce a full list of users and defects assigned to each of them, you’d need to use a group join like the one in listing 11.14. The compiler always uses a method called GroupBy for grouping clauses. When the projection in a grouping clause is trivial—when each entry in the original sequence maps directly to the exact same object in a subsequence—the compiler uses a simple method call that only requires the grouping expression, so it knows how to map each element to a key. For instance, the query expression in listing 11.17 is translated into this: SampleData.AllDefects.Where(defect => defect.AssignedTo != null) .GroupBy(defect => defect.AssignedTo) 320 CHAPTER 11 Query expressions and LINQ to Objects When the projection is nontrivial, a slightly more complicated version is used. The fol- lowing listing gives an example of a projection where you only capture the summary of each defect rather than the Defect object itself. var query = from defect in SampleData.AllDefects where defect.AssignedTo != null group defect.Summary by defect.AssignedTo; foreach (var entry in query) { Console.WriteLine(entry.Key.Name); foreach (var summary in entry) { Console.WriteLine(" {0}", summary); } Console.WriteLine(); } Listing 11.18 Grouping defects by assignee—projection retains just the summary Defect: { ID=1, AssignedTo=Darren ...} Defect: { ID=2, AssignedTo=null ...} Defect: { ID=3, AssignedTo=Tara ...} Defect: { ID=4, AssignedTo=Darren ...} Defect: { ID=5, AssignedTo=Tim ...} Defect: { ID=6, AssignedTo=Darren ...} ... (Result of query) Defect: { ID=1, AssignedTo=Darren ...} Defect: { ID=3, AssignedTo=Tara ...} Defect: { ID=4, AssignedTo=Darren ...} Defect: { ID=5, AssignedTo=Tim ...} Defect: { ID=6, AssignedTo=Darren ...} ... Key=Darren Defect: { ID=1, AssignedTo=Darren ...} Defect: { ID=4, AssignedTo=Darren ...} Defect: { ID=6, AssignedTo=Darren ...} ... Key=Tara Defect: { ID=3, AssignedTo=Tara ...} Defect: { ID=13, AssignedTo=Tara ...} ... Key=Tim Defect: { ID=5, AssignedTo=Tim ...} Defect: { ID=8, AssignedTo=Tim ...} ... Df {ID1 A i from defect in SampleData.AllDefects where defect.AssignedTo != null group defect by defect.AssignedTo All known defects Defects assigned to a user Groups of defects, keyed by the user the defects are assigned to Figure 11.9 Sequences used when grouping defects by assignee. Each entry of the result has a Key property and is also a sequence of defect entries. 321Groupings and continuations I’ve highlighted the differences between listings 11.18 and 11.17 in bold. Because each defect is projected to just its summary, the embedded sequence in each entry is just an IEnumerable. In this case the compiler uses an overload of GroupBy with another parameter to represent the projection. The query expression in listing 11.18 is translated into the following expression: SampleData.AllDefects.Where(defect => defect.AssignedTo != null) .GroupBy(defect => defect.AssignedTo, defect => defect.Summary) Grouping clauses are relatively simple but useful. Even in the defect-tracking system, you could easily imagine wanting to group defects by project, creator, severity, or sta- tus, as well as by the assignee used in these examples. So far, you’ve ended each query expression with a select or group ... by clause, and that’s been the end of the expression. But there are times when you’ll want to do more with the results, and that’s when query continuations are used. 11.6.2 Query continuations Query continuations provide a way of using the result of one query expression as the initial sequence of another. They apply to both group ... by and select clauses, and the syntax is the same for both—you simply use the contextual keyword into and then provide the name of a new range variable. That range variable can then be used in the next part of the query expression. The specification explains this in terms of a translation from one query expression to another, changing first-query into identifier second-query-body into from identifier in (first-query) second-query-body An example will make this clearer. Let’s go back to the grouping of defects by assignee, but this time imagine you only want the count of the defects assigned to each person. You can’t do that with the projection in the grouping clause, because that only applies to each individual defect. You want to project each group, which contains an assignee and the sequence of their defects, into a single element consisting of the assignee and the count of defects in the group. This can be achieved with the following code. var query = from defect in SampleData.AllDefects where defect.AssignedTo != null group defect by defect.AssignedTo into grouped select new { Assignee = grouped.Key, Count = grouped.Count() }; foreach (var entry in query) Listing 11.19 Continuing a grouping with another projection 322 CHAPTER 11 Query expressions and LINQ to Objects { Console.WriteLine("{0}: {1}", entry.Assignee.Name, entry.Count); } The changes to the query expression are highlighted in bold. You can use the grouped range variable in the second part of the query, but the defect range variable is no lon- ger available—you can think of it as being out of scope. This projection simply creates an anonymous type with Assignee and Count properties, using the key of each group as the assignee, and counting the sequence of defects associated with each group. The results of listing 11.19 are as follows: Darren Dahlia: 14 Tara Tutu: 5 Tim Trotter: 5 Deborah Denton: 9 Colin Carton: 2 Following the specification, the query expression from listing 11.19 is translated into this one: from grouped in (from defect in SampleData.AllDefects where defect.AssignedTo != null group defect by defect.AssignedTo) select new { Assignee = grouped.Key, Count = grouped.Count() } The rest of the translations are then performed, resulting in the following code: SampleData.AllDefects .Where(defect => defect.AssignedTo != null) .GroupBy(defect => defect.AssignedTo) .Select(grouped => new { Assignee = grouped.Key, Count = grouped.Count() }) An alternative way of understanding continuations is to think of two separate state- ments. This isn’t as accurate in terms of the actual compiler translation, but I find it makes it easier to see what’s going on. In this case, the query expression (and assign- ment to the query variable) can be thought of as the following two statements: var tmp = from defect in SampleData.AllDefects where defect.AssignedTo != null group defect by defect.AssignedTo; var query = from grouped in tmp select new { Assignee = grouped.Key, Count = grouped.Count() }; Of course, if you find this easier to read, there’s nothing to stop you from breaking up the original expression into this form in your source code. Nothing will be evalu- ated until you start trying to step through the query results anyway, due to deferred execution. JOIN...INTO ISN’T A CONTINUATION It’s easy to fall into the trap of thinking that wherever you see the contextual keyword into, you have a continuation. This isn’t true for joins—the join ... into clause (which is used for group joins) 323Groupings and continuations doesn’t form a continuation. The important difference is that with a group join, all the earlier range variables (apart from the one used to name the right side of the join) can still be used. Compare that with the queries we’re looking at in this section, where the continuation wipes the slate clean; the only range variable available afterward is the one declared by the continuation. Let’s extend this example to see how multiple continuations can be used. The results are currently unordered—let’s change that so you can see who has the most defects assigned to them first. You could use a let clause after the first continuation, but the following listing shows an alternative with a second continuation after the current expression. var query = from defect in SampleData.AllDefects where defect.AssignedTo != null group defect by defect.AssignedTo into grouped select new { Assignee = grouped.Key, Count = grouped.Count() } into result orderby result.Count descending select result; foreach (var entry in query) { Console.WriteLine("{0}: {1}", entry.Assignee.Name, entry.Count); } The changes between listings 11.19 and 11.20 are highlighted in bold. You didn’t need to change any of the output code, because you had the same type of sequence— you just needed to apply an ordering to it. This time the translated query expression is as follows: SampleData.AllDefects .Where(defect => defect.AssignedTo != null) .GroupBy(defect => defect.AssignedTo) .Select(grouped => new { Assignee = grouped.Key, Count = grouped.Count() }) .OrderByDescending(result => result.Count); By pure coincidence, this is remarkably similar to the first defect-tracking query we looked at, in section 10.3.6. The final select clause effectively does nothing, so the C# compiler ignores it. It’s required in the query expression, though, as all query expres- sions must end with either a select or a group ... by clause. There’s nothing to stop you from using a different projection or performing other operations with the contin- ued query—joins, further groupings, and so forth. Just keep an eye on the readability of the query expression as it grows. Speaking of readability, there are options to consider when you’re writing LINQ queries. Listing 11.20 Query expression continuations from group and select 324 CHAPTER 11 Query expressions and LINQ to Objects 11.7 Choosing between query expressions and dot notation As you’ve seen throughout this chapter, query expressions are translated into normal C# before being compiled any further. There isn’t an official name for a call to the LINQ query operators written using normal C# rather than as a query expression, but many developers now refer to this as dot notation.7 Every query expression can be written in dot notation, but the reverse isn’t true: many LINQ operators don’t have a query expres- sion equivalent in C#. The big question is this: When should you use which syntax? 11.7.1 Operations that require dot notation The most obvious situation where you’re forced to use dot notation is when you’re calling a method such as Reverse or ToDictionary that isn’t represented in query expression syntax at all. But even when you use a query operator that’s supported by query expressions, it’s quite possible for the overload you want to be unavailable. For example, Enumerable.Where has an overload where the index into the parent sequence is supplied as another argument to the delegate. In such a situation, you could use code like the following to take every other item from a sequence: sequence.Where((item, index) => index % 2 == 0) There’s a similar overload for Select, so if you wanted to be able to get at the original index in a sequence after ordering, you could do something like this: sequence.Select((Item, Index) => new { Item, Index }) .OrderBy(x => x.Item.Name) This example shows another option you might want to consider: if you’re going to use a lambda expression parameter directly in an anonymous type, you could buck the nor- mal convention of starting the parameter name with a lowercase letter, and then use a projection initializer to avoid writing new { Item = item, Index = index }, which can be distracting. Of course, you can ignore the convention about property names instead, and make your anonymous type have properties beginning with a lowercase letter (item and index, for example). All of this is entirely up to you, and it’s worth experi- menting. Although consistency is usually important, it doesn’t matter too much here, as the impact of inconsistency is confined to the method in question; you’re not expos- ing these names in your public API or throughout the rest of your class. Many of the query operators also support custom comparisons—ordering and joining being the most obvious examples. These are unlikely to be required often, in my experience, but they’re occasionally invaluable. For example, if you want to per- form a join on a person’s name in a case-insensitive manner, you can specify String- Comparer.OrdinalIgnoreCase (or a culture-specific comparer) as the final argument to a Join call. Again, if you feel that an operator nearly does what you want but doesn’t quite cut it, check the documentation for other overloads. 7 That’s the term I’ll use from now on, but if you hear others talking about fluent notation, they probably mean the same thing. 325Choosing between query expressions and dot notation When you’re forced to use dot notation, the decision to use it is easy, but what about cases where a query expression could be used? 11.7.2 Query expressions where dot notation may be simpler Some developers use query expressions everywhere they can get away with it; person- ally, I look at what the query is doing and decide which approach will be more readable. For example, take this query expression, which is similar to one near the start of this chapter: var adults = from person in people where person.Age >= 18 select person; This is three lines of code with a lot of baggage, even though all it’s doing is filtering. In this case I’d use dot notation: var adults = people.Where(person => person.Age >= 18); I find that clearer—every part of it mentions something you’re actually interested in. Another area where using dot notation throughout a query expression can give more clarity is when you’re forced to use it for part of the query anyway. For example, suppose you’re going to use the ToList() extension method to end up with a list of the names of adults. (I’m performing a projection as well, in this case, so that it’s a more balanced comparison.) Here’s the query expression: var adultNames = (from person in people where person.Age >= 18 select person.Name).ToList(); Here’s the dot notation equivalent: var adultNames = people.Where(person => person.Age >= 18) .Select(person => person.Name) .ToList(); Something about the need for parentheses around the query expression in the first case makes it seem uglier to me. This is very much a case of personal choice—this sec- tion is really just raising your awareness that there is a choice, and that you can pick and choose. If you’re going to use LINQ to any significant extent, you really should be comfortable with both notations, and there’s no harm in switching style based on the query in question. As you’ve seen, the generated code is absolutely equivalent. None of this is to say that I dislike query expressions, of course. 11.7.3 Where query expressions shine Having explained where you might find dot notation beneficial, I should point out that when it comes to any operations where the query expression would use transpar- ent identifiers—particularly joins—dot notation starts to suffer in terms of readability. The beauty of transparent identifiers is that they’re transparent—so transparent that you can’t see them at all when you only have to look at the query expression. Even a 326 CHAPTER 11 Query expressions and LINQ to Objects simple let clause can be enough to swing the decision in favor of query expressions; introducing a new anonymous type just to propagate context through the query gets annoying quickly. The other area where query expressions win is in situations where multiple lambda expressions would be required, or even multiple method calls. Again, this includes joins, where you have to specify the key selector for each side of the join as well as the result selector. For example, here’s a cut-down version of an earlier query where I introduced inner joins: from defect in SampleData.AllDefects join subscription in SampleData.AllSubscriptions on defect.Project equals subscription.Project select new { defect.Summary, subscription.EmailAddress } In an IDE, it’d be reasonable to put the whole join clause on one line, leading to fairly easy-to-read code. The dot notation equivalent is fairly horrible, though: SampleData.AllDefects.Join(SampleData.AllSubscriptions, defect => defect.Project, subscription => subscription.Project, (defect, subscription) => new { defect.Summary, subscription.EmailAddress }) The last argument could all fit on one line in an IDE, but it’s still pretty ugly because the lambda expressions don’t have much context; you can’t immediately tell which argument means what. Named arguments in C# 4 can help there, but that adds even more bulk to the query. Complex orderings can be similarly unpleasant in dot notation. Consider which you’d rather read—this orderby clause orderby item.Rating descending, item.Price, item.Name or three method calls: .OrderByDescending(item => item.Rating) .ThenBy(item => item.Price) .ThenBy(item => item.Name) Changing the priority of these orderings is simple in the query expression—just switch them around. In dot notation, you may also have to switch from OrderBy to ThenBy or vice versa. To reiterate, I’m not trying to press my own personal preferences onto your code. I simply want you to know what’s available, and to think about the choices you make. Of course, this is only one aspect of writing readable code, but it’s a whole new area to consider in C#. 11.8 Summary In this chapter, we’ve looked at how LINQ to Objects and C# 3 interact, focusing on the way query expressions are first translated into code that doesn’t involve query 327Summary expressions and then are compiled in the usual way. You’ve seen how all query expres- sions form a series of sequences, applying a transformation of some description at each step. In many cases these sequences are evaluated using deferred execution, fetching data only when it’s first required. Compared with all the other features of C# 3, query expressions look somewhat alien—more like SQL than the C# you’re used to. One of the reasons they look so odd is that they’re declarative instead of imperative—a query talks about the features of the end result rather than the exact steps required to achieve it. This goes hand in hand with a more functional way of thinking. It can take a while to click, and it’s not suitable for every situation, but where declarative syntax is appropriate, it can vastly improve readability as well as make code easier to test and parallelize. Don’t be fooled into thinking that LINQ should only be used with databases. Plain in-memory manipulation of collections is common, and you’ve seen how well it’s sup- ported by query expressions and the extension methods in Enumerable. In a real sense, you’ve now seen all the features introduced in C# 3! We haven’t looked at any other LINQ providers yet, but you have a clearer understanding of what the compiler will do for you when you ask it to handle XML and SQL. The compiler itself doesn’t know the difference between LINQ to Objects, LINQ to SQL, or any of the other providers; it just follows the same rules blindly. In the next chapter you’ll see how these rules form the final piece of the LINQ jig- saw puzzle when they convert lambda expressions into expression trees so that the var- ious clauses of query expressions can be executed on different platforms. You’ll also see some other examples of what LINQ can do. 328 LINQ beyond collections Suppose an alien visited you and asked you to describe culture. How could you cap- ture the diversity of human culture in a short space of time? You might decide to spend that time showing him culture rather than describing it in the abstract: a visit to a New Orleans jazz club, opera in La Scala, the Louvre gallery in Paris, a Shake- speare play in Stratford-upon-Avon, and so on. Would this alien know everything about culture afterward? Could he compose a tune, write a book, dance a ballet, craft a sculpture? Absolutely not. But he’d hope- fully leave with a sense of culture—its richness and variety, its ability to light up peo- ple’s lives. So it is with this chapter. You’ve now seen all of the features of C# 3, but without seeing more of LINQ, you don’t have enough context to really appreciate them. This chapter covers  LINQ to SQL  IQueryable and expression tree queries  LINQ to XML  Parallel LINQ  Reactive Extensions for .NET  Writing your own operators 329Querying a database with LINQ to SQL When the first edition of this book was published, not many LINQ technologies were available, but now there’s a glut of them, both from Microsoft and from third parties. That in itself hasn’t surprised me, but I’ve been fascinated to see the different natures of these technologies. We’ll look at various ways in which LINQ manifests itself, with an example of each. I’ve chosen to demonstrate Microsoft technologies in the main, because they’re the most typical ones. This isn’t meant to imply that third parties aren’t welcome in the LINQ ecosystem: there are a number of projects, both commercial and open source, providing access to varied data sources and building extra features on top of existing providers. In contrast to the rest of this book, we’ll only skim the surface of each of the topics here—the point isn’t to learn the details, but to immerse yourself in the spirit of LINQ. To investigate any of these technologies further, I recommend that you get a book dedicated to the subject or read the relevant documentation carefully. I’ve resisted the temptation to say, “There’s more to LINQ to [xxx] than this” at the end of each sec- tion, but please take it as read. Each technology has many capabilities beyond query- ing, but I’ve focused here on the areas that are directly related to LINQ. Let’s start off with the provider that generally got the most attention when LINQ was first introduced: LINQ to SQL. 12.1 Querying a database with LINQ to SQL I’m sure by now you’ve absorbed the message that LINQ to SQL converts query expres- sions into SQL, which is then executed on the database. It’s more than that—it’s a full ORM solution—but I’ll concentrate on the query side of LINQ to SQL rather than go into concurrency handling and the other details that an ORM has to deal with. I’ll show you just enough so that you can experiment with it yourself—the database and code are available on the book’s website (http://csharpindepth.com). The database is in SQL Server 2005 format to make it easy to play with, even if you don’t have the latest version of SQL Server installed, although obviously Microsoft has made sure that LINQ to SQL works against newer versions too. WHY LINQ TO SQL RATHER THAN THE ENTITY FRAMEWORK? Speaking of “newer versions,” you may be wondering why I’ve chosen to demonstrate LINQ to SQL instead of the Entity Framework, which is now Microsoft’s preferred solution (and which also supports LINQ). The answer is merely simplicity; the Entity Framework is undoubtedly more powerful than LINQ to SQL in various ways, but it requires extra concepts that would take too much space to explain here. I’m trying to give you a sense of the consistency (and occasional incon- sistency) that LINQ provides, and that’s as applicable to LINQ to SQL as to the Entity Framework. Before you start writing any queries, you need a database and a model to represent it in code. 330 CHAPTER 12 LINQ beyond collections 12.1.1 Getting started: the database and model LINQ to SQL needs metadata about the database to know which classes correspond to which database tables, and so on. There are various ways of representing that meta- data, and this section will use the LINQ to SQL designer built into Visual Studio. You can design the entities first and ask LINQ to create the database, or design your data- base and let Visual Studio work out what the entities should look like. Personally, I favor the second approach, but there are pros and cons for both ways. CREATING THE DATABASE SCHEMA The mapping from the classes in chapter 11 to database tables is straightforward. Each table has an auto-incrementing integer ID column with an appropriate name: ProjectID, DefectID, and so on. The references between tables simply use the same name, so the Defect table has a ProjectID column, for instance, with a foreign key constraint. There are a few exceptions to this simple set of rules:  User is a reserved word in T-SQL, so the User class is mapped to the DefectUser table.  The enumerations (status, severity, and user type) don’t have tables. Their val- ues are mapped to tinyint columns in the Defect and DefectUser tables.  The Defect table has two links to the DefectUser table: one for the user who created the defect and one for the current assignee. These are represented with the CreatedByUserId and AssignedToUserId columns, respectively. CREATING THE ENTITY CLASSES Once your tables are created, generating the entity classes from Visual Studio is easy. Simply open Server Explorer (View > Server Explorer) and add a data source to the SkeetySoftDefects database (right-click on Data Connections and select Add Con- nection). You should be able to see four tables: Defect, DefectUser, Project, and NotificationSubscription. You can then add a new item of type “LINQ to SQL classes” to the project. This name will be the basis for a generated class representing the overall database model; I’ve used the name DefectModel, which leads to a class called DefectModelData- Context. The designer will open when you’ve created the new item. You can then drag the four tables from Server Explorer into the designer, and it’ll figure out all the associations. After that, you can rearrange the diagram and adjust various properties of the entities. Here’s a list of what I changed:  I renamed the DefectID property to ID to match the previous model.  I renamed DefectUser to User (so although the table is still called DefectUser, it’ll generate a class called User, just like before).  I changed the type of the Severity, Status, and UserType properties to their enum equivalents (having copied those enumerations into the project). 331Querying a database with LINQ to SQL  I renamed the parent and child properties used for the associations between Defect and DefectUser—the designer guessed suitable names for the other associations but had trouble here because there were two associations between the same pair of tables. I named the relationships AssignedTo/Assigned- Defects and CreatedBy/CreatedDefects. Figure 12.1 shows the designer diagram after all of these changes. As you can see, it looks much like the class diagram in figure 11.3, but without the enumerations. If you look in the C# code generated by the designer (DefectModel.designer.cs), you’ll find five partial classes: one for each of the entities, and the DefectModelData- Context class I mentioned earlier. The fact that they’re partial is useful; in this case I added extra constructors to match the ones from the original in-memory classes, so the code from chapter 11 to create the sample data can be reused without much extra work. For the sake of brevity, I didn’t include the insertion code here, but if you look at PopulateDatabase.cs in the source code, you should be able to follow it easily enough. Of course, you don’t have to run this yourself—the downloadable database is already populated. Now that you have a schema in SQL, an entity model in C#, and some sample data, it’s time to get querying. Figure 12.1 The LINQ to SQL classes designer showing the rearranged and modified entities 332 CHAPTER 12 LINQ beyond collections 12.1.2 Initial queries I’m sure you’ve guessed what’s coming, but hopefully that won’t make it any less impressive. We’ll execute query expressions against the data source, watching LINQ to SQL convert the query into SQL on the fly. For the sake of familiarity, we’ll use some of the same queries we executed against the in-memory collections in chapter 11. FIRST QUERY: FINDING DEFECTS ASSIGNED TO TIM I’ll skip over the trivial examples from early in chapter 11 and start instead with the query from listing 11.7 that checks for open defects assigned to Tim. Here’s the query part of listing 11.7, for the sake of comparison: User tim = SampleData.Users.TesterTim; var query = from defect in SampleData.AllDefects where defect.Status != Status.Closed where defect.AssignedTo == tim select defect.Summary; The full LINQ to SQL equivalent of listing 11.7 is as follows. using (var context = new DefectModelDataContext()) { context.Log = Console.Out; User tim = context.Users .Where(user => user.Name == "Tim Trotter") .Single(); var query = from defect in context.Defects where defect.Status != Status.Closed where defect.AssignedTo == tim select defect.Summary; foreach (var summary in query) { Console.WriteLine(summary); } } Listing 12.1 requires a certain amount of explanation, because it’s all new. First you create a new data context to work with B. Data contexts are pretty multifunctional, tak- ing responsibility for connection and transaction management, query translation, tracking changes in entities, and dealing with identity. For the purposes of this chapter, you can regard a data context as your point of contact with the database. I don’t show the more advanced features here, but you do take advantage of one useful capability here: you tell the data context to write out all the SQL commands it executes to the console C. The model-related properties used in the code for this section (Defects, Users, and so on) are all of type Table for the relevant entity type. They act as the data sources for your queries. Listing 12.1 Querying the database to find all Tim’s open defects Creates context to work withB Enables console loggingCQueries database to find Tim D Queries database to find Tim’s open defectsE 333Querying a database with LINQ to SQL You can’t use SampleData.Users.TesterTim to identify Tim in the main query because that object doesn’t know the ID of the relevant row in the DefectUser table. Instead, you use a separate query to load Tim’s user entity D. I used dot notation for this, but a query expression would’ve worked just as well. The Single method just returns a single result from a query, throwing an exception if there isn’t exactly one element. In a real-life situation, you may have the entity as a product of other opera- tions, such as logging in, and if you don’t have the full entity, you may have its ID, which can be used equally well within the main query. As an alternative in this case, you could’ve change the open defects query to filter based on the assignee’s name. That wouldn’t have quite been in the spirit of the original query, though. Within the query expression E, the only difference between the in-memory query and the LINQ to SQL query is the data source—instead of using SampleData.All- Defects, you use context.Defects. The final results are the same (although the ordering isn’t guaranteed), but the work has been done on the database. Because you asked the data context to log the generated SQL, you can see exactly what’s going on when you run the code. The console output shows both of the queries executed on the database, along with the query parameter values:1 SELECT [t0].[UserID], [t0].[Name], [t0].[UserType] FROM [dbo].[DefectUser] AS [t0] WHERE [t0].[Name] = @p0 -- @p0: Input String (Size = 11; Prec = 0; Scale = 0) [Tim Trotter] SELECT [t0].[Summary] FROM [dbo].[Defect] AS [t0] WHERE ([t0].[AssignedToUserID] = @p0) AND ([t0].[Status] <> @p1) -- @p0: Input Int32 (Size = 0; Prec = 0; Scale = 0) [2] -- @p1: Input Int32 (Size = 0; Prec = 0; Scale = 0) [4] Note how the first query fetches all of the properties of the user because you’re popu- lating a whole entity, but the second query only fetches the summary, as that’s all you need. LINQ to SQL has also converted the two separate where clauses in the second query into a single filter on the database. LINQ to SQL is capable of translating a wide range of expressions. Let’s try a slightly more complicated query from chapter 11, just to see what SQL is generated. SQL GENERATION FOR A MORE COMPLEX QUERY: A LET CLAUSE The next query shows what happens when you introduce a sort of temporary variable with a let clause. In chapter 11 we considered a bizarre situation, if you remember— pretending that calculating the length of a string took a long time. Again, the query expression here is exactly the same as in listing 11.11, with the exception of the data source. The following listing shows the LINQ to SQL code. 1 Additional log output is generated showing some details of the data context, which I’ve omitted here to avoid distracting from the SQL. The console output also contains the summaries printed by the foreach loop, of course. 334 CHAPTER 12 LINQ beyond collections using (var context = new DefectModelDataContext()) { context.Log = Console.Out; var query = from user in context.Users let length = user.Name.Length orderby length select new { Name = user.Name, Length = length }; foreach (var entry in query) { Console.WriteLine("{0}: {1}", entry.Length, entry.Name); } } The generated SQL is close to the spirit of the sequences we saw in figure 11.5. The innermost sequence (the first one in that diagram) is the list of users; that’s trans- formed into a sequence of name/length pairs (as the nested select), and then the no-op projection is applied, with an ordering by length: SELECT [t1].[Name], [t1].[value] FROM ( SELECT LEN([t0].[Name]) AS [value], [t0].[Name] FROM [dbo].[DefectUser] AS [t0] ) AS [t1] ORDER BY [t1].[value] This is a good example of where the generated SQL is wordier than it needs to be. Although you couldn’t reference the elements of the final output sequence when per- forming an ordering on the query expression, you can in SQL. This simpler query would’ve worked fine: SELECT LEN([t0].[Name]) AS [value], [t0].[Name] FROM [dbo].[DefectUser] AS [t0] ORDER BY [value] Of course, what’s important is what the query optimizer does on the database—the execution plan displayed in SQL Server Management Studio Express is the same for both queries, so it doesn’t look like you’re losing out. The final set of LINQ to SQL queries we’ll look at are all joins. 12.1.3 Queries involving joins We’ll try both inner joins and group joins, using the examples of joining notification subscriptions against projects. I suspect you’re used to the drill now—the pattern of the code is the same for each query, so from here on I’ll just show the query expres- sion and the generated SQL, unless something else is going on. EXPLICIT JOINS: MATCHING DEFECTS WITH NOTIFICATION SUBSCRIPTIONS The first query is the simplest kind of join—an inner equijoin using a LINQ join clause: Listing 12.2 Using a let clause in LINQ to SQL 335Querying a database with LINQ to SQL // Query expression (modified from listing 11.12) from defect in context.Defects join subscription in context.NotificationSubscriptions on defect.Project equals subscription.Project select new { defect.Summary, subscription.EmailAddress } -- Generated SQL SELECT [t0].[Summary], [t1].[EmailAddress] FROM [dbo].[Defect] AS [t0] INNER JOIN [dbo].[NotificationSubscription] AS [t1] ON [t0].[ProjectID] = [t1].[ProjectID] Unsurprisingly, it uses an inner join in SQL. It’d be easy to guess at the generated SQL in this case. How about a group join, though? This is where things get slightly more hectic: // Query expression (modified from listing 11.13) from defect in context.Defects join subscription in context.NotificationSubscriptions on defect.Project equals subscription.Project into groupedSubscriptions select new { Defect = defect, Subscriptions = groupedSubscriptions } -- Generated SQL SELECT [t0].[DefectID] AS [ID], [t0].[Created], [t0].[LastModified], [t0].[Summary], [t0].[Severity], [t0].[Status], [t0].[AssignedToUserID], [t0].[CreatedByUserID], [t0].[ProjectID], [t1].[NotificationSubscriptionID], [t1].[ProjectID] AS [ProjectID2], [t1].[EmailAddress], (SELECT COUNT(*) FROM [dbo].[NotificationSubscription] AS [t2] WHERE [t0].[ProjectID] = [t2].[ProjectID]) AS [count] FROM [dbo].[Defect] AS [t0] LEFT OUTER JOIN [dbo].[NotificationSubscription] AS [t1] ON [t0].[ProjectID] = [t1].[ProjectID] ORDER BY [t0].[DefectID], [t1].[NotificationSubscriptionID] That’s a major change in the amount of SQL generated! There are two important things to notice. First, it uses a left outer join instead of an inner join, so you’d still see a defect even if it didn’t have anyone subscribing to its project. If you want a left outer join but without the grouping, the conventional way of expressing this is to use a group join and then an extra from clause, using the DefaultIfEmpty extension method on the embedded sequence. It looks odd, but it works well. The second odd thing about the previous query is that it calculates the count for each group within the database. This is effectively a trick performed by LINQ to SQL to make sure that all the processing can be done on the server. A naive implementation would have to perform the grouping in memory after fetching all the results. In some cases, the provider could do tricks to avoid needing the count, simply spotting when the grouping ID changes, but there are issues with this approach for some queries. It’s possible that a later implementation of LINQ to SQL will be able to switch courses of action depending on the exact query. 336 CHAPTER 12 LINQ beyond collections You don’t need to explicitly write a join in the query expression to see one in the SQL. Our final queries will show joins implicitly created through property access expressions. IMPLICIT JOINS: SHOWING DEFECT SUMMARIES AND PROJECT NAMES Let’s take a simple example. Suppose you want to list each defect, showing its sum- mary and the name of the project it’s part of. The query expression is just a matter of a projection: // Query expression from defect in context.Defects select new { defect.Summary, ProjectName = defect.Project.Name } -- Generated SQL SELECT [t0].[Summary], [t1].[Name] FROM [dbo].[Defect] AS [t0] INNER JOIN [dbo].[Project] AS [t1] ON [t1].[ProjectID] = [t0].[ProjectID] Note how you navigate from the defect to the project via a property—LINQ to SQL has converted that navigation into an inner join. It can use an inner join here because the schema has a non-nullable constraint on the ProjectID column of the Defect table— every defect has a project. Not every defect has an assignee, though, because the AssignedToUserID field is nullable, so if you use the assignee in a projection instead, a left outer join is generated: // Query expression from defect in context.Defects select new { defect.Summary, Assignee = defect.AssignedTo.Name } -- Generated SQL SELECT [t0].[Summary], [t1].[Name] FROM [dbo].[Defect] AS [t0] LEFT OUTER JOIN [dbo].[DefectUser] AS [t1] ON [t1].[UserID] = [t0].[AssignedToUserID] Of course, if you navigate via more properties, the joins get more and more compli- cated. I’m not going into the details here—the important thing is that LINQ to SQL has to do a lot of analysis of the query expression to work out what SQL is required. In order to perform that analysis, it clearly needs to be able to look at the query you’ve specified. Let’s move away from LINQ to SQL specifically, and think in general terms about what LINQ providers of this kind need to do. This will apply to any provider that needs to introspect the query, rather than just being handed a delegate. At long last, it’s time to see why expression trees were added as a feature of C# 3. 12.2 Translations using IQueryable and IQueryProvider In this section I’ll show you the basics of how LINQ to SQL manages to convert your query expressions into SQL. This is the starting point for implementing your own 337Translations using IQueryable and IQueryProvider LINQ provider, should you wish to. (Please don’t underestimate the technical difficul- ties involved in doing so—but if you like a challenge, implementing a LINQ provider is certainly interesting.) This is the most theoretical section in the chapter, but it’s useful to have some insight as to how LINQ decides whether to use in-memory processing, a database, or some other query engine. In all the query expressions we’ve looked at in LINQ to SQL, the source has been a Table. But if you look at Table, you’ll see it doesn’t have a Where method, or Select, or Join, or any of the other standard query operators. Instead, it uses the same trick that LINQ to Objects does; just as the source in LINQ to Objects always implements IEnumerable (possibly after a call to Cast or OfType) and then uses the extension methods in Enumerable, so Table implements IQueryable and then uses the extension methods in Queryable. You’ll see how LINQ builds up an expression tree and then allows a provider to execute it at the appropriate time. Let’s start by looking at what IQueryable consists of. 12.2.1 Introducing IQueryable and related interfaces If you look up IQueryable in the documentation and see what members it con- tains directly (rather than inheriting), you may be disappointed. There aren’t any. Instead, it inherits from IEnumerable and the nongeneric IQueryable, which in turn inherits from the nongeneric IEnumerable. So IQueryable is where the new and exciting members are, right? Well, nearly. In fact, IQueryable just has three proper- ties: QueryProvider, ElementType, and Expression. The QueryProvider property is of type IQueryProvider—yet another new interface to consider. Lost? Perhaps figure 12.2 will help out—it’s a class diagram of all the interfaces directly involved. The easiest way of thinking about IQueryable is that it represents a query that’ll yield a sequence of results when you execute it. The details of the query in LINQ terms are held in an expression tree, as returned by the Expression property of the IQueryable. The query is executed by iterating through the IQueryable (in other words, calling the GetEnumerator method and then MoveNext on the result) or by call- ing the Execute method on an IQueryProvider, passing in an expression tree. Now that you have at least some grasp of what IQueryable is for, what’s IQuery- Provider? You can do more with a query than just execute it; you can also use it to build a bigger query, which is the purpose of the standard query operators in LINQ.2 To build up a query, you need to use the CreateQuery method on the relevant IQueryProvider.3 Think of a data source as a simple query (SELECT * FROM SomeTable in SQL, for instance)—calling Where, Select, OrderBy, and similar methods results in a different 2 Well, the ones that keep deferring execution, such as Where and Join. You’ll see what happens with the aggregations such as Count in a while. 3 Both Execute and CreateQuery have generic and nongeneric overloads. The nongeneric versions make it easier to create queries dynamically in code. Compile-time query expressions use the generic version. 338 CHAPTER 12 LINQ beyond collections query based on the first one. Given any IQueryable query, you can create a new query by performing the following steps: 1 Ask the existing query for its query expression tree (using the Expression property). 2 Build a new expression tree that contains the original expression and the extra functionality you want (a filter, projection, or ordering, for instance). 3 Ask the existing query for its query provider (using the Provider property). 4 Call CreateQuery on the provider, passing in the new expression tree. Of those steps, the only tricky one is creating the new expression tree. Fortunately, there are a bunch of extension methods on the static Queryable class that can do all that for you. Enough theory—let’s start implementing the interfaces so we can see all this in action. 12.2.2 Faking it: interface implementations to log calls Before you get too excited, you’re not going to build a full-fledged query provider in this chapter. But if you understand everything in this section, you’ll be in a much bet- ter position to build one if you ever need to—and possibly more important, you’ll understand what’s going on when you issue LINQ to SQL queries. Most of the hard work of query providers goes on at the point of execution, where they need to parse an expression tree and convert it into the appropriate form for the target platform. Figure 12.2 Class diagram of the interfaces involved in IQueryable 339Translations using IQueryable and IQueryProvider We’ll concentrate on the work that happens before that—how LINQ prepares to exe- cute a query. We’ll look at implementations of IQueryable and IQueryProvider and then try to run a few queries against them. The interesting part isn’t the results—the queries won’t do anything useful—but the series of calls made up to the point of execution. We’ll focus on two types: FakeQueryProvider and FakeQuery. The implementation of each interface method writes out the current expression involved, using a simple log- ging method (not shown here). Let’s look first at FakeQuery, as shown in the following listing. class FakeQuery : IQueryable { public Expression Expression { get; private set; } public IQueryProvider Provider { get; private set; } public Type ElementType { get; private set; } internal FakeQuery(IQueryProvider provider, Expression expression) { Expression = expression; Provider = provider; ElementType = typeof(T); } internal FakeQuery() : this(new FakeQueryProvider(), null) { Expression = Expression.Constant(this); } public IEnumerator GetEnumerator() { Logger.Log(this, Expression); return Enumerable.Empty().GetEnumerator(); } IEnumerator IEnumerable.GetEnumerator() { Logger.Log(this, Expression); return Enumerable.Empty().GetEnumerator(); } public override string ToString() { return "FakeQuery"; } } The property members of IQueryable are implemented in FakeQuery with automatic properties B, which are set by the constructors. There are two constructors: a param- eterless one that’s used by the main program to create a plain source for the query, and one that’s called by FakeQueryProvider with the current query expression. Listing 12.3 A simple implementation of IQueryable that logs method calls Declares simple automatic propertiesB Uses this query as initial expressionC Returns empty result sequence D Overrides ToString for the sake of logging E 340 CHAPTER 12 LINQ beyond collections The use of Expression.Constant(this) as the initial source expression C is just a way of showing that the query initially represents the original object. (Imagine an implementation representing a table, for example—until you apply any query opera- tors, the query would just return the whole table.) When the constant expression is logged, it uses the overridden ToString method, which is why you give a short, con- stant description e. This makes the final expression much cleaner than it would’ve been without the override. When you’re asked to iterate over the results of the query, you always return an empty sequence D to make life easy. Production implementa- tions would parse the expression here, or (more likely) call Execute on their query provider and return the result. As you can see, not a lot is going on in FakeQuery, and the following listing shows that FakeQueryProvider is simple too. class FakeQueryProvider : IQueryProvider { public IQueryable CreateQuery(Expression expression) { Logger.Log(this, expression); return new FakeQuery(this, expression); } public IQueryable CreateQuery(Expression expression) { Type queryType = typeof(FakeQuery<>).MakeGenericType(expression.Type); object[] constructorArgs = new object[] { this, expression }; return (IQueryable)Activator.CreateInstance(queryType, constructorArgs); } public T Execute(Expression expression) { Logger.Log(this, expression); return default(T); } public object Execute(Expression expression) { Logger.Log(this, expression); return null; } } There’s even less to say about the implementation of FakeQueryProvider than there was for FakeQuery. The CreateQuery methods do no real processing but act as fac- tory methods for the query. The only tricky bit is that the nongeneric overload still needs to provide the right type argument for FakeQuery based on the Type prop- erty of the given expression. The Execute method overloads return empty results after logging the call. This is where a lot of analysis would normally be done, along with the actual call to the web service, database, or other target platform. Listing 12.4 An implementation of IQueryProvider that uses FakeQuery 341Translations using IQueryable and IQueryProvider Even though you’ve done no real work, interesting things start to happen when you start to use FakeQuery as the source in a query expression. I’ve already let slip that you’re able to write query expressions without explicitly writing methods to handle the standard query operators: it’s all about extension methods—this time the ones in the Queryable class. 12.2.3 Gluing expressions together: the Queryable extension methods Just as the Enumerable type contains extension methods on IEnumerable to imple- ment the LINQ standard query operators, the Queryable type contains extension methods on IQueryable. There are two big differences between the implementa- tions in Enumerable and those in Queryable. First, the Enumerable methods all use delegates as their parameters—the Select method takes a Func, for example. That’s fine for in-memory manipulation, but for LINQ providers that execute the query elsewhere, you need a format you can examine more closely—expression trees. For example, the correspond- ing overload of Select in Queryable takes a parameter of type Expression>. The compiler doesn’t mind at all—after query translation, it has a lambda expression that it needs to pass as an argument to the method, and lambda expressions can be converted to either delegate instances or expression trees. This is how LINQ to SQL can work so seamlessly. The four key elements involved are all new features of C# 3: lambda expressions, the translation of query expressions into normal expressions that use lambda expressions, extension methods, and expres- sion trees. Without all four, there’d be problems. If query expressions were always translated into delegates, for instance, they couldn’t be used with a provider such as LINQ to SQL, which requires expression trees. Figure 12.3 shows two possible paths taken by query expressions; they differ only in what interfaces their data source implements. Note how in figure 12.3 the early parts of the compilation process are independent of the data source. The same query expression is used, and it’s translated in exactly the same way. It’s only when the compiler looks at the translated query to find the appropriate Select and Where methods to use that the data source is truly important. At that point, the lambda expressions can be converted to either delegate instances or expression trees, potentially giving radically different implementations: typically in- memory for the left path, and SQL executing against a database in the right path. Just to hammer home a familiar point, the decision in figure 12.3 of whether to use Enumerable or Queryable has no explicit support in the C# compiler. These aren’t the only two possible paths, as you’ll see later with Parallel LINQ and Reactive LINQ. You can create your own interface and implement extension methods following the query pattern, or even create a type with appropriate instance methods. The second big difference between Enumerable and Queryable is that the Enumerable extension methods do the actual work associated with the corresponding query operator (or at least they build iterators that do the work). There’s code in 342 CHAPTER 12 LINQ beyond collections Enumerable.Where to execute the specified filter and only yield appropriate elements as the result sequence, for example. By contrast, the query operator implementations in Queryable do little: they just create a new query based on the parameters or they call Execute on the query provider, as described at the end of section 12.2.1. In other words, they’re only used to build up queries and request that they be executed—they don’t contain the logic behind the operators. This means they’re suitable for any LINQ provider that uses expression trees, but they’re useless on their own. They’re the glue between your code and the details of the provider. With the Queryable extension methods available and ready to use the IQueryable and IQueryProvider implementations, it’s finally time to see what happens when you use a query expression with your custom provider. 12.2.4 The fake query provider in action Listing 12.5 shows a simple query expression, which (supposedly) finds all the strings in the fake source, beginning with abc, and projects the results into a sequence listing the lengths of the matching strings. You iterate through the results but don’t do any- thing with them, as you know already that they’ll be empty. That’s because you have Overload resolution Query expression translation IQueryable implementationPlain IEnumerable implementation from user in users where user.Name.StartsWith("D") select user.Name users.Where(user => user.Name.StartsWith("D")) .Select(user => user.Name) IL to create expression trees, with calls to Queryable.Where and Queryable.Select Extension methods on Queryable are chosen, which use expression trees as parameters Extension methods on Enumerable are chosen, which use delegates as parameters IL to create delegate instances, with calls to Enumerable.Where and Enumerable.Select Figure 12.3 A query taking two paths, depending on whether the data source implements IQueryable or only IEnumerable 343Translations using IQueryable and IQueryProvider no source data, and you haven’t written any code to do any real filtering—you’re just logging which calls are made by LINQ in the course of creating the query expression, and iterating through the results. var query = from x in new FakeQuery() where x.StartsWith("abc") select x.Length; foreach (int i in query) {} What would you expect the results of running listing 12.5 to be? In particular, what would you like to be logged last, at the point where you’d normally expect to do some real work with the expression tree? Here are the results, reformatted slightly for clarity: FakeQueryProvider.CreateQuery Expression=FakeQuery.Where(x => x.StartsWith("abc")) FakeQueryProvider.CreateQuery Expression=FakeQuery.Where(x => x.StartsWith("abc")) .Select(x => x.Length) FakeQuery.GetEnumerator Expression=FakeQuery.Where(x => x.StartsWith("abc")) .Select(x => x.Length) The two important things to note are that GetEnumerator is only called at the end, not on any intermediate queries; by the time GetEnumerator is called, you have all the information present in the original query expression. You haven’t manually had to keep track of earlier parts of the expression in each step—a single expression tree cap- tures all the information so far. Don’t be fooled by the concise output, by the way—the actual expression tree is deep and complicated, particularly due to the where clause including an extra method call. This expression tree is what LINQ to SQL will examine to work out what query to execute. LINQ providers could build up their own queries (in whatever form they may need) when calls to CreateQuery are made, but usually looking at the final tree when GetEnumerator is called is simpler, because all the necessary information is available in one place. The final call logged by listing 12.5 was to FakeQuery.GetEnumerator, and you may be wondering why you also need an Execute method on IQueryProvider. Well, not all query expressions generate sequences. If you use an aggregation operator such as Sum, Count, or Average, you’re no longer really creating a source—you’re evaluating a result immediately. That’s when Execute is called, as shown by the following listing and its output. var query = from x in new FakeQuery() where x.StartsWith("abc") select x.Length; Listing 12.5 A simple query expression using the fake query classes Listing 12.6 IQueryProvider.Execute 344 CHAPTER 12 LINQ beyond collections double mean = query.Average(); // Output FakeQueryProvider.CreateQuery Expression=FakeQuery.Where(x => x.StartsWith("abc")) FakeQueryProvider.CreateQuery Expression=FakeQuery.Where(x => x.StartsWith("abc")) .Select(x => x.Length) FakeQueryProvider.Execute Expression=FakeQuery.Where(x => x.StartsWith("abc")) .Select(x => x.Length) .Average() The FakeQueryProvider can be quite useful when it comes to understanding what the C# compiler is doing behind the scenes with query expressions. It’ll show the trans- parent identifiers introduced within a query expression, along with the translated calls to SelectMany, GroupJoin, and the like. 12.2.5 Wrapping up IQueryable You haven’t written any of the significant code that a real query provider would need in order to get useful work done, but hopefully this fake provider has given you insight into how LINQ providers get their information from query expressions. It’s all built up by the Queryable extension methods, given an appropriate implementation of IQueryable and IQueryProvider. We’ve gone into a bit more detail in this section than we will for the rest of the chapter, as it involved the foundations that underpin the LINQ to SQL code we saw earlier. Even though you’re unlikely to need to implement query interfaces yourself, the steps involved in taking a C# query expression and (at execution time) running some SQL on a database are quite profound and lie at the heart of the big features of C# 3. Understanding why C# has gained these features will help keep you more in tune with the language. This is the end of our coverage of LINQ using expression trees. The rest of the chap- ter involves in-process queries using delegates, but as you’ll see, there can still be a great deal of variety and innovation in how LINQ can be used. Our first port of call is LINQ to XML, which is “merely” an XML API designed to integrate well with LINQ to Objects. 12.3 LINQ-friendly APIs and LINQ to XML LINQ to XML is by far the most pleasant XML API I’ve ever used. Whether you’re con- suming existing XML, generating a new document, or a bit of both, it’s easy to use and understand. Part of that is completely independent of LINQ, but a lot of it’s due to how well it interacts with the rest of LINQ. As in section 12.1, I’ll give you just enough introductory information to understand the examples, and then you’ll see how LINQ to XML blends its own query operators with those in LINQ to Objects. By the end of the section, you may have some ideas about how you can make your own APIs work in harmony with the framework. 345LINQ-friendly APIs and LINQ to XML 12.3.1 Core types in LINQ to XML LINQ to XML lives in the System.Xml.Linq assembly, and most of the types are in the System.Xml.Linq namespace too.4 Almost all of the types in that namespace have a prefix of X, so whereas the normal DOM API has an XmlElement type, the LINQ to XML equivalent is XElement. This makes it easy to spot when code is using LINQ to XML, even if you’re not immediately familiar with the exact type involved. Figure 12.4 shows the types you’ll use most often. Here’s a brief rundown of the types shown:  XName is used for names of elements and attributes. Instances are usually cre- ated using an implicit conversion from a string (in which case no namespace is used) or via the +(XNamespace, string) overloaded operator.  XNamespace represents an XML namespace—a URI, basically. Instances are usu- ally created by the implicit conversion from string.  XObject is the common ancestor of both XNode and XAttribute; unlike in the DOM API, an attribute isn’t a node in LINQ to XML. Methods returning child nodes don’t include attributes, for example.  XNode represents a node in the XML tree. It defines various members to manip- ulate and query the tree. There are several other classes derived from XNode that aren’t shown in figure 12.4, such as XComment and XDeclaration. These are 4 I used to forget whether it was System.Xml.Linq or System.Linq.Xml. If you remember that it’s an XML API first and foremost, you should be okay. Figure 12.4 Class diagram for LINQ to XML, showing the most commonly used types 346 CHAPTER 12 LINQ beyond collections used relatively infrequently—the most common node types are documents, ele- ments, and text.  XAttribute is an attribute with a name and a value. The value is intrinsically text, but there are explicit conversions to many other data types, such as int and DateTime.  XContainer is a node in the XML tree that can have child content—it’s an ele- ment or a document, basically.  XText is a text node, and a further derived type XCData is used to represent CDATA text nodes. (A CDATA node is roughly equivalent to a verbatim string lit- eral—less escaping is required.) XText is rarely instantiated directly in user code; instead, when a string is used as the content of an element or document, that’s converted into an XText instance.  XElement is an element. This is the most commonly used class in LINQ to XML, along with XAttribute. Unlike in the DOM API, you can create an XElement without creating a document to contain it. Unless you really need a document object (for a custom XML declaration, perhaps), you can often just use elements.  XDocument is a document. Its root element is accessed using the Root prop- erty—this is the equivalent to XmlDocument.DocumentElement. As noted earlier, this often isn’t required. More types are available even within the document model, and there are a few other types for things such as loading and saving options—but this list covers the most important ones. Of the preceding types, the only ones you regularly need to reference explicitly are XElement and XAttribute. If you use namespaces, you’ll use XNamespace as well, but most of the rest of the types can be ignored the rest of the time. It’s amaz- ing how much you can do with so few types. Speaking of amazing, I can’t resist showing you how the namespace support works in LINQ to XML. We won’t use namespaces anywhere else, but it’s a good example of how a well-designed set of conversions and operators can make life easier. It’ll also ease us into our next topic: constructing elements. If you only need to specify the name of an element or attribute without a namespace, you can use a string. You won’t find any constructors for either type with parameters of type string though—they all accept an XName. An implicit conversion exists from string to XName, and also from string to XNamespace. Adding together a namespace and a string also gives you an XName. There’s a fine line between operator abuse and genius, but in this case LINQ to XML really makes it work. Here’s some code to create two elements—one within a namespace and one not: XElement noNamespace = new XElement("no-namespace"); XNamespace ns = "http://csharpindepth.com/sample/namespace"; XElement withNamespace = new XElement(ns + "in-namespace"); This makes for readable code even when namespaces are involved, which comes as a welcome relief from some other APIs. But this just creates two empty elements. How do you give them some content? 347LINQ-friendly APIs and LINQ to XML 12.3.2 Declarative construction Normally in the DOM API, you create an element and then add content to it. You can do that in LINQ to XML via the Add method inherited from XContainer, but that’s not the idiomatic LINQ to XML way of doing things.5 It’s still worth looking at the signa- ture of XContainer.Add though, because it introduces the content model. You might’ve expected a signature of Add(XNode) or perhaps Add(XObject), but it’s just Add(object). The same pattern is used for the XElement (and XDocument) construc- tor signatures. The XElement constructors all have one parameter for the name of the element, but after that you can specify nothing (to create an empty element), a single object (to create an element with a single child node), or an array of objects to create multiple child nodes. In the multiple children case, a parameter array is used (the params keyword in C#), which means the compiler will create the array for you—you can just keep listing arguments. The use of plain object for the content type may sound crazy, but it’s incredibly useful. When you add content—whether it’s through a constructor or the Add method—the following points are considered:  Null references are ignored.  XNode and XAttribute instances are added in a relatively straightforward man- ner; they’re cloned if they already have parents, but otherwise no conversion is required. (Some other sanity checks are performed, such as making sure you don’t have duplicate attributes in a single element.)  Strings, numbers, dates, times, and so on are added by converting them into XText nodes using standard XML formatting.  If the argument implements IEnumerable (and isn’t covered by anything else), Add will iterate over its contents and add each value in turn, recursing where necessary.  Anything that doesn’t have special-case handling is converted into text by just calling ToString(). This means that you often don’t need to prepare your content in a special way before adding it to an element—LINQ to XML does the right thing for you. The details are explicitly documented, so you don’t need to worry about it being too magical—but it really works. Constructing nested elements leads to code that naturally resembles the hierarchi- cal structure of the tree. This is best shown with an example. Here’s a snippet of LINQ to XML code: new XElement("root", new XElement("child", new XElement("grandchild", "text")), new XElement("other-child")); 5 In some ways, it’s a shame that XElement doesn’t implement IEnumerable, as otherwise collection initializ- ers would be another approach to construction. But using the constructor works neatly anyway. 348 CHAPTER 12 LINQ beyond collections And here’s the XML of the created element—note the visual similarity between the code and the output: text So far, so good, but the important part is the fourth bullet in the earlier list, where sequences are processed recursively, because that lets you build an XML structure out of a LINQ query in a natural way. For example, the book’s website has some code to generate an RSS feed from its database. The statement to construct the XML docu- ment is 28 lines long—which I’d normally expect to be an abomination—but it’s remarkably pleasant to read.6 That statement contains two LINQ queries—one to pop- ulate an attribute value, and the other to provide a sequence of elements, each repre- senting a news item. As you read the code, it’s obvious what the resulting XML will look like. To make this more concrete, let’s take two simple examples from the defect- tracking system. I’ll demonstrate using the LINQ to Objects sample data, but you could use almost identical queries to work with another LINQ provider instead. First, you need to build an element containing all the users in the system. In this case, you just need a projection, so the following listing uses dot notation. var users = new XElement("users", SampleData.AllUsers.Select(user => new XElement("user", new XAttribute("name", user.Name), new XAttribute("type", user.UserType))) ); Console.WriteLine(users); // Output If you want to make a slightly more complex query, it’s probably worth using a query expression. The following listing creates another list of users, but this time only the 6 One contributing factor to the readability is an extension method I created to convert anonymous types into elements, using the properties for child elements. If you’re interested, the code is freely available as part of my MiscUtil project (see http://mng.bz/xDMt). It only helps when the XML structure you need fits a certain pattern, but in that case it can reduce the clutter of XElement constructor calls significantly. Listing 12.7 Creating elements from the sample users 349LINQ-friendly APIs and LINQ to XML developers within SkeetySoft. For a bit of variety, this time each developer’s name is a text node within an element instead of an attribute value. var developers = new XElement("developers", from user in SampleData.AllUsers where user.UserType == UserType.Developer select new XElement("developer", user.Name) ); Console.WriteLine(developers); // Output Deborah Denton Darren Dahlia This sort of thing can be applied to all the sample data, producing a document struc- ture like this: You can see the code to generate all of this in XmlSampleData.cs in the downloadable solution. It demonstrates an alternative to the one-huge-statement approach: each of the elements under the top level is created separately, and then glued together like this: XElement root = new XElement("defect-system", projects, users, defects); We’ll use this XML to demonstrate the next LINQ integration point: queries. Let’s start with the query methods available on a single node. 12.3.3 Queries on single nodes You may be expecting me to reveal that XElement implements IEnumerable and that LINQ queries come for free. It’s not quite that simple, because there are so many differ- ent things that an XElement could iterate through. XElement contains a number of axis methods that are used as query sources. If you’re familiar with XPath, the idea of an axis will no doubt be familiar to you. Listing 12.8 Creating elements with text nodes 350 CHAPTER 12 LINQ beyond collections Here are the axis methods used directly for querying a single node, each of which returns an appropriate IEnumerable:  Ancestors  DescendantNodes  Annotations  Elements  Descendants  ElementsBeforeSelf  AncestorsAndSelf  DescendantNodesAndSelf  Attributes  ElementsAfterSelf  DescendantsAndSelf  Nodes All of these are fairly self-explanatory (and the MSDN documentation provides more details). There are useful overloads to retrieve only nodes with an appropriate name; calling Descendants("user") on an XElement will return all user elements under- neath the element you call it on, for instance. In addition to these calls returning sequences, some methods return a single result—Attribute and Element are the most important, returning the named attri- bute and the first child element with the specified name, respectively. Additionally, there are explicit conversions from an XAttribute or XElement to any number of other types, such as int, string, and DateTime. These are important for both filtering and projecting results. Each conversion to a non-nullable value type also has a conver- sion to its nullable equivalent—these (and the conversion to string) return a null value if you invoke them on a null reference. This null propagation means you don’t have to check for the presence or absence of attributes or elements within the query— you can use the query results instead. What does this have to do with LINQ? Well, the fact that multiple search results are returned in terms of IEnumerable means you can use the normal LINQ to Objects methods after finding some elements. The following listing shows an example of find- ing the names and types of the users, this time starting off with the sample data in XML. XElement root = XmlSampleData.GetElement(); var query = root.Element("users").Elements().Select(user => new { Name = (string) user.Attribute("name"), UserType = (string) user.Attribute("type") }); foreach (var user in query) { Console.WriteLine ("{0}: {1}", user.Name, user.UserType); } After creating the data at the start, you navigate down to the users element and ask it for its direct child elements. This two-step fetch could be shortened to just root.Descendants("user"), but it’s good to know about the more rigid navigation so you can use it where necessary. It’s also more robust in the face of changes to the Listing 12.9 Displaying the users within an XML structure 351LINQ-friendly APIs and LINQ to XML document structure, such as another (unrelated) user element being added else- where in the document. The rest of the query expression is merely a projection of an XElement into an anonymous type. I’ll admit that this is slightly cheating with the user type: it’s kept as a string instead of calling Enum.Parse to convert it into a proper UserType value. The latter approach works perfectly well, but it’s quite long-winded when you only need the string form, and the code becomes hard to format sensibly within the strict limits of the printed page. There’s nothing particularly special here—returning query results as sequences is fairly common, after all. It’s worth noting how seamlessly you can go from domain- specific query operators to general-purpose ones. That’s not the end of the story, though. LINQ to XML has some extra extension methods to add as well. 12.3.4 Flattened query operators You’ve seen how the result of one part of a query is often a sequence, and in LINQ to XML it’s often a sequence of elements. What if you wanted to then perform an XML- specific query on each of those elements? To present a somewhat contrived example, you can find all the projects in the sample data with root.Element("projects") .Elements(), but how can you find the subscription elements within them? You need to apply another query to each element and then flatten the results. (Again, you could use root.Descendants("subscription"), but imagine a more complex docu- ment model where that wouldn’t work.) This may sound familiar, and it is—LINQ to Objects already provides the Select- Many operator (represented by multiple from clauses in a query expression) to do this. You could write the query as follows: from project in root.Element("projects").Elements() from subscription in project.Elements("subscription") select subscription As there are no elements within a project other than subscription, you could use the overload of Elements that doesn’t specify a name. I find it clearer to specify the element name in this case, but it’s often just a matter of taste. (The same argument could be made for calling Element("projects").Elements("project") to start with, admittedly.) Here’s the same query written using dot notation and an overload of SelectMany that only returns the flattened sequence, without performing any further projections: root.Element("projects").Elements() .SelectMany(project => project.Elements("subscription")) Neither of these queries are completely unreadable by any means, but they’re not ideal. LINQ to XML provides a number of extension methods (in the System .Xml.Linq.Extensions class), which either act on a specific sequence type or are generic with a constrained type argument, to cope with the lack of generic interface covariance prior to C# 4. There’s InDocumentOrder, which does exactly what it sounds 352 CHAPTER 12 LINQ beyond collections like, and most of the axis methods mentioned in section 12.4.3 are also available as extension methods. This means that you can convert the previous query into this sim- pler form: root.Element("projects").Elements().Elements("subscription") This sort of construction makes it easy to write XPath-like queries in LINQ to XML without everything being a string. If you want to use XPath, that’s available too via more extension methods, but the query methods have served me well more often than not. You can also mix the axis methods with the operators of LINQ to Objects. For example, to find all the subscriptions for projects with a name including Media, you could use this query: root.Element("projects").Elements() .Where(project => ((string) project.Attribute("name")) .Contains("Media")) .Elements("subscription") Before we move on to Parallel LINQ, let’s think about how the design of LINQ to XML merits the “LINQ” part of its title, and how you could potentially apply the same tech- niques to your own API. 12.3.5 Working in harmony with LINQ Some of the design decisions in LINQ to XML seem odd if you take them in isolation as part of an XML API, but in the context of LINQ they make perfect sense. The design- ers clearly imagined how their types could be used within LINQ queries, and how they could interact with other data sources. If you’re writing your own data access API, in whatever context that might be, it’s worth taking the same things into account. If someone uses your methods in the middle of a query expression, are they left with something useful? Will they be able to use some of your query methods, then some from LINQ to Objects, and then some more of yours in one fluent expression? We’ve seen three ways in which LINQ to XML has accommodated the rest of LINQ:  It’s good at consuming sequences with its approach to construction. LINQ is deliberately declarative, and LINQ to XML supports this with a declarative way of creating XML structures.  It returns sequences from its query methods. This is probably the most obvious step that data access APIs would already take: returning query results as IEnumerable or a class implementing it is pretty much a no-brainer.  It extends the set of queries you can perform on sequences of XML types; this makes it feel like a unified querying API, even though some of it is XML-specific. You may be able to think of other ways in which your own libraries can play nicely with LINQ; these aren’t the only options you should consider, but they’re a good starting point. Above all, I’d urge you to put yourself in the shoes of a developer wanting to use your API within code that’s already using LINQ. What might such a developer want 353Replacing LINQ to Objects with Parallel LINQ to achieve? Can LINQ and your API be mixed easily, or are they really aiming for differ- ent goals? We’re roughly halfway through our whirlwind tour of different approaches to LINQ. Our next stop is in some ways reassuring and in some ways terrifying: we’re back to querying simple sequences, but this time in parallel… 12.4 Replacing LINQ to Objects with Parallel LINQ I’ve been following Parallel LINQ for a long time. I first came across it when Joe Duffy introduced it in his blog on September 2006 (see http://mng.bz/vYCO). The first Community Technology Preview (CTP) was released in November 2007, and the over- all feature set has evolved over time too. It’s now part of a wider effort called Parallel Extensions, which is part of .NET 4, aiming to provide higher-level building blocks for concurrent programming than the relatively small set of primitives we’ve had to work with until now. There’s a lot more to Parallel Extensions than Parallel LINQ—or PLINQ, as it’s often known—but we’ll only look at the LINQ aspect here. The idea behind Parallel LINQ is that you should be able to take a LINQ to Objects query that’s taking a long time and make it run faster by using multiple threads to take advantage of multiple cores—with as few changes to the query as possible. As with any- thing to do with concurrency, it’s not quite as simple as that, but you may be surprised at what can be achieved. Of course, we’re still trying to think bigger than individual LINQ technologies—we’re thinking about the different models of interaction involved, rather than the precise details. But if you’re interested in concurrency, I heartily recommend that you dive into Parallel Extensions—it’s one of the most prom- ising approaches to parallelism that I’ve come across recently. I’ll use a single example for this section: rendering a Mandelbrot set image (see Wikipedia for an explanation of Mandelbrot sets: http://en.wikipedia.org/wiki/ Mandelbrot_set). Let’s start off by trying to get it right with a single thread before moving into trickier territory. 12.4.1 Plotting the Mandelbrot set with a single thread Before any mathematicians attack me, I should point out that I’m using the term Man- delbrot set loosely here. The details aren’t really important, but these aspects are:  You’ll create a rectangular image, given various options such as width, height, origin, and search depth.  For each pixel in the image, you’ll calculate a byte value that will end up as the index into a 256-entry palette.  The calculation of one pixel value doesn’t rely on any other results. The last point is absolutely crucial—it means this task is embarrassingly parallel. In other words, there’s nothing in the task itself that makes it hard to parallelize. You still need a mechanism for distributing the workload across threads and then gather- ing the results together, but the rest should be easy. PLINQ will be responsible for the 354 CHAPTER 12 LINQ beyond collections distribution and collation (with a little help and care); you just need to express the range of pixels, and how each pixel’s color should be computed. For the purposes of demonstrating multiple approaches, I’ve put together an abstract base class that’s responsible for setting things up, running the query, and dis- playing the results; it also has a method to compute the color of an individual pixel. An abstract method is responsible for creating a byte array of values, which are then converted into the image. The first row of pixels comes first, left to right, then the sec- ond row, and so on. Each example here is just an implementation of this method. I should note that using LINQ really isn’t an ideal solution here—there are various inefficiencies in this approach. Don’t focus on that side of things: concentrate on the idea that we have an embarrassingly parallel query, and we want to execute it across multiple cores. The following listing shows the single-threaded version of the method in all its sim- ple glory. var query = from row in Enumerable.Range(0, Height) from column in Enumerable.Range(0, Width) select ComputeIndex(row, column); return query.ToArray(); You iterate over every row and every column within each row, computing the index of the relevant pixel. Calling ToArray() evaluates the resulting sequence, converting it into an array. Figure 12.5 shows the beautiful results. This took about 5.5 seconds to gener- ate on my old dual-core laptop. The ComputeIndex method performs more iterations than you really need, but it makes the timing differences more obvi- ous.7 Now that you have a benchmark in terms of both timing and what the results should look like, it’s time to parallelize the query. 12.4.2 Introducing ParallelEnumerable, ParallelQuery, and AsParallel Parallel LINQ brings with it several new types, but in many cases you’ll never see their names mentioned. They live in the System.Linq namespace, so you don’t Listing 12.10 Single-threaded Mandelbrot generation query 7 Proper benchmarking is hard, particularly when threading is involved. I haven’t attempted to do rigorous measurements here. The timings given are just meant to indicate faster and slower; please take the numbers with a pinch of salt. Figure 12.5 Mandelbrot image generated on a single thread 355Replacing LINQ to Objects with Parallel LINQ even need to change using directives. ParallelEnumerable is a static class, similar to Enumerable—it mostly contains exten- sion methods, the majority of which extend a new ParallelQuery type. This latter type has both nongeneric and generic forms (ParallelQuery and ParallelQuery), but most of the time you’ll use the generic form, just as IEnumerable is more widely used than IEnumerable. Additionally, there’s Ordered- ParallelQuery, which is the par- allel equivalent of IOrderedEnumerable. The relationships between all of these types are shown in figure 12.6. As you can see, ParallelQuery implements IEnumerable, so once you’ve constructed a query appropriately, you can iterate through the results in the nor- mal way. When you have a parallel query, the extension methods in Parallel- Enumerable take precedence over the ones in Enumerable (because ParallelQuery is more specific than IEnumerable; see section 10.2.3 if you need a reminder of the rules); this is how the parallelism is maintained throughout a query. There’s a parallel equivalent to all the LINQ standard query operators, but you should be careful if you’ve created any of your own extension methods. You’ll still be able to call them, but they’ll force the query to be single-threaded from that point onward. How do you get a parallel query to start with? By calling AsParallel, an extension method in ParallelEnumerable that extends IEnumerable. This means you can parallelize the Mandelbrot query incredibly simply, as shown in the following listing. var query = from row in Enumerable.Range(0, Height) .AsParallel() from column in Enumerable.Range(0, Width) select ComputeIndex(row, column); return query.ToArray(); Job done? Well, not quite. This query does run in parallel, but the results aren’t quite what you need: it doesn’t maintain the order in which you process the rows. Instead of the beautiful Mandelbrot image, we get something like figure 12.7, but the exact details change every time, of course. Oops. On the bright side, this rendered in about 3.2 seconds, so my machine was clearly making use of its second core. But getting the right answer is pretty important. Listing 12.11 First attempt at a multithreaded Mandelbrot generation query Figure 12.6 Class diagram for Parallel LINQ, including relationship to normal LINQ interfaces 356 CHAPTER 12 LINQ beyond collections You may be surprised to hear that this is a deliberate feature of Parallel LINQ. Ordering a parallel query requires more coordination between the threads, and the whole purpose of parallelization is to improve performance, so PLINQ defaults to an unordered query. It’s a bit of a nui- sance in this case, though. 12.4.3 Tweaking parallel queries Fortunately, there’s a way out of this— you just need to force the query to be treated as ordered, which can be done with the AsOrdered extension method. The following listing shows the fixed code, which produces the original image. It’s slightly slower than the unordered query, but still significantly faster than the sin- gle-threaded version. var query = from row in Enumerable.Range(0, Height) .AsParallel().AsOrdered() from column in Enumerable.Range(0, Width) select ComputeIndex(row, column); return query.ToArray(); The nuances of ordering are beyond the scope of this book, but I recommend that you read the “PLINQ Ordering” MSDN blog post (http://mng.bz/9x9U), which goes into the gory details. A number of other methods can be used to alter how the query behaves:  AsUnordered—Makes an ordered query unordered; if you only need results to be ordered for the first part of a query, this allows later stages to be executed more efficiently.  WithCancellation—Specifies a cancellation token to be used with this query. Cancellation tokens are used throughout Parallel Extensions to allow tasks to be canceled in a safe, controlled manner.  WithDegreeOfParallelism—Allows you to specify the maximum number of concurrent tasks used to execute the query. You could use this to limit the num- ber of threads used if you wanted to avoid swamping the machine, or to increase the number of threads used for a query that wasn’t CPU-bound.  WithExecutionMode—Can be used to force the query to execute in parallel, even if Parallel LINQ thinks it’d execute faster as a single-threaded query. Listing 12.12 Multithreaded Mandelbrot query maintaining ordering Figure 12.7 Mandelbrot image generated using an unordered query, resulting in some sections being incorrectly placed 357Inverting the query model with LINQ to Rx  WithMergeOptions—Allows you to tweak how the results are buffered. Dis- abling buffering gives the shortest time before the first result is returned, but also lower throughput; full buffering gives the highest throughput, but no results are returned until the query has executed completely. The default is a compromise between the two. The important point is that aside from ordering, these methods shouldn’t affect the results of the query. You can design your query and test it in LINQ to Objects, then par- allelize it, work out your ordering requirements, and tweak it if necessary to perform just how you want it to. If you showed the final query to someone who knew LINQ but not PLINQ, you’d only have to explain the PLINQ-specific method calls—the rest of the query would be familiar. Have you ever seen such an easy way to achieve concurrency? (The rest of Parallel Extensions is aimed at achieving simplicity where possible too.) PLAY WITH THE CODE YOURSELF A couple of further points are demonstrated in the downloadable source code. If you parallelize across the whole query of pixels rather than just the rows, then an unordered query looks even weirder, and there’s a ParallelEnumerable.Range method that gives PLINQ a bit more information than calling Enumerable.Range(...).AsParallel(). I used AsParallel() in this section, because that’s the more general way of parallelizing a query; most queries don’t start with a range. Changing the in-process query model from single-threaded to parallel is quite a small conceptual leap, really. In the next section we’ll turn the model on its head. 12.5 Inverting the query model with LINQ to Rx All of the LINQ libraries you’ve seen so far have one thing in common: you pull data from them using IEnumerable. At first sight, that seems so obvious that it’s not worth saying—what would be the alternative? Well, how about if you push the data instead of pulling it? Instead of the data consumer being in control, the provider can be in the driving seat, letting the data consumer react when new data is available. Don’t worry too much if this sounds dauntingly different; you actually know about the fundamental concept already, in the form of events. If you’re comfortable with the idea of subscribing to an event, reacting to it, and unsubscribing later, that’s a good starting point. Reactive Extensions for .NET is a Microsoft project (http://mng.bz/R7ip); there are multiple versions available, including one targeting JavaScript. These days, the simplest way of obtaining the latest version is via NuGet. You may hear Reactive Extensions going by various names, but Rx and LINQ to Rx are the most common abbreviations, and they’re the ones I’ll use here. Even more so than for the other technologies covered in this chapter, we’ll barely scratch the surface here. Not only is there a lot to learn about the library itself, but it’s a whole different way of thinking. There are loads of videos on Channel 9 (see http://channel9.msdn.com/tags/Rx/)—some are based on the math- ematical aspects, whereas others are more practical. In this section I’ll emphasize the way that the LINQ concepts can be applied to this push model for data flow. 358 CHAPTER 12 LINQ beyond collections Enough of the introduction…let’s meet the two interfaces that form the basis of LINQ to Rx. 12.5.1 IObservable and IObserver The data model of LINQ to Rx is the mathematical dual of the normal IEnumerable model.8 When you iterate over a pull collection, you effectively start off by saying, “Please give me an iterator” (the call to GetEnumerator) and then repeatedly ask, “Is there another item? If so, I’d like it now” (via calls to MoveNext and Current). LINQ to Rx reverses this. Instead of requesting an iterator, you provide an observer. Then, instead of requesting the next item, your code is told when one is ready—or when an error occurs or the end of the data is reached. Here are the declarations of the two interfaces involved: public interface IObservable { IDisposable Subscribe(IObserver observer); } public interface IObserver { void OnNext(T value); void OnCompleted(); void OnException(Exception error); } These interfaces are actually part of .NET 4 (in the System namespace), even though the rest of LINQ to Rx is in a separate download. In fact, they’re IObservable and IObserver in .NET 4, expressing the covariance of IObservable and the contravariance of IObserver. You’ll learn more about generic variance in the next chapter, but I’m presenting the interfaces here as if they were invariant for the sake of simplicity. One concept at a time! Figure 12.8 shows the duality in terms of how data flows in each model. I suspect I’m not alone in finding the push model harder to think about, as it has the natural ability to work asynchronously. But look at how much simpler it is than the pull model, in terms of the flow diagram. This is partly due to the multiple method approach of the pull model; if IEnumerator just had a method with a signature of bool TryGetNext(out T item), it’d be somewhat simpler. Earlier I mentioned that LINQ to Rx is similar to the events you’re already familiar with. Calling Subscribe on an observable is like using += with an event to register a handler. The disposable value returned by Subscribe remembers the observer you passed in; disposing of it is like using -= with the same handler. In many cases, you don’t need to unsubscribe from the observable; this is just available in case you need to unsubscribe halfway through a sequence—the equivalent of breaking out of a foreach loop early. Failing to dispose of an IDisposable value may feel like anathema 8 For a more detailed examination of this duality—and the essence of LINQ itself—I recommend Bart de Smet’s “The Essence of LINQ—MINLINQ” blog post at http://mng.bz/96Wh. 359Inverting the query model with LINQ to Rx to you, but it’s often safe in LINQ to Rx. None of the examples in this chapter use the return value of Subscribe. That’s all there is to IObservable, but what about the observer itself? Why does it have three methods? Consider the normal pull model where for any MoveNext/ Current pair of calls, three things can happen:  You may be at the end of the sequence, in which case MoveNext returns false.  You may not have reached the end of the sequence, in which case MoveNext returns true and Current returns the new value.  An error may occur—reading the next line from a network connection could fail, for example. In this case, an exception would be thrown. The IObserver interface represents each of these options as a separate method. Typically an observer will have its OnNext method called repeatedly, and then finally OnCompleted, unless there’s an error of some kind, in which case OnError will be called instead. After the sequence has completed or encountered an error, no further method calls will be made. You rarely need to implement IObserver directly, though. There are many extension methods on IObservable, including overloads for Subscribe, and these allow you to subscribe to an observable by just providing appropriate delegates. Usually you provide a delegate to be executed for each item, and then optionally one to be executed on completion, on error, or both. With that bit of theory out of the way, let’s look at some actual code using LINQ to Rx. Pull model Caller IEnumerable IEnumerator GetEnumerator() Return: IEnumerator MoveNext() Return: true Current Return: first value MoveNext() Return: false Caller IObservable IObserver Subscribe(observer) Return: IDisposable ...(more items)... OnNext(first value) ...(more items)... OnCompleted() Push model Figure 12.8 Sequence diagram showing the duality of IEnumerable and IObservable 360 CHAPTER 12 LINQ beyond collections 12.5.2 Starting simply (again) We’ll demonstrate LINQ to Rx in the same way we started off with LINQ to Objects— using a range. Instead of Enumerable.Range, we’ll use Observable.Range, which cre- ates an observable range. Each time an observer subscribes to the range, the numbers are emitted to that observer using OnNext, followed by OnCompleted. We’ll start off as simply as we can, just printing out each value as it’s received, and printing a confirma- tion message at the end or if an error occurs. The following listing shows that this involves less code than you’d need for the pull model. var observable = Observable.Range(0, 10); observable.Subscribe(x => Console.WriteLine("Received {0}", x), e => Console.WriteLine("Error: {0}", e), () => Console.WriteLine("Finished")); In this case, it’s hard to see how you could get an error, but I’ve included the error notification delegate for completeness. The results are as you’d expect: Received 0 Received 1 ... Received 9 Finished The observable returned by the Range method is known as a cold observable: it lies dor- mant until an observer subscribes to it, at which point it’ll emit the values to that indi- vidual observer. If you subscribe with another observer, that will see another copy of the range. This isn’t quite the same as a normal event such as a button click, where several observers could be subscribed to the same actual sequence of values, and the values may be effectively yielded whether there are any observers or not. (You can click a button even if there aren’t any event handlers attached, after all.) Sequences like this are known as hot observables. It’s important to know which type you’re dealing with, even though the same set of operations applies to both kinds. Now that you’ve done the simplest thing possible, let’s try some familiar LINQ operators. 12.5.3 Querying observables By now I’m sure you’re familiar with the pattern—there are various extension meth- ods in a static class (called Observable, somewhat predictably) that perform appropri- ate transformations. We’ll look at just a few of the available operators and think a little about what’s not available, and why it’s not. FILTERING AND PROJECTING Let’s jump straight into a query expression that takes a sequence of numbers, filters out the odd ones, and squares anything that’s left. Then we’ll subscribe Console Listing 12.13 First contact with IObservable 361Inverting the query model with LINQ to Rx .WriteLine to the final result of the query, so that any items produced will be dis- played. The following listing shows the code—look at how the query expression could easily be a LINQ to Objects query. var numbers = Observable.Range(0, 10); var query = from number in numbers where number % 2 == 0 select number * number; query.Subscribe(Console.WriteLine); For simplicity’s sake, I haven’t added handlers for completion or error, and using the conversion from the Console.WriteLine method group to an Action keeps the code nice and short. This produces the same results it would in LINQ to Objects: 0, 4, 16 and so on. Let’s move on to grouping. GROUPING A group by query expression in LINQ to Rx produces a new IGroupedObservable for each group, although what you then do with the grouping isn’t always obvious. For example, it’s not uncommon to have a nested subscription so that each time a new group is produced, you subscribe an observer to that group. The results within each group are produced as they’re received by the grouping construct—effectively it acts as a sort of redirection choice, like an usher at a play examining each person’s ticket as they arrive, and directing them to the relevant section of the theater. By contrast, LINQ to Objects collects a whole group together before returning it, which means it has to read to the end of the sequence, buffering all the results. The following listing shows an example of this nested subscription, and also dem- onstrates how group results are emitted. var numbers = Observable.Range(0, 10); var query = from number in numbers group number by number % 3; query.Subscribe(group => group.Subscribe (x => Console.WriteLine("Value: {0}; Group: {1}", x, group.Key))); The best way to understand this is probably to remember that dealing with groups in LINQ to Objects often involves having a nested foreach loop—so you have nested sub- scriptions in LINQ to Rx. When in doubt, try to find the duality between the two data models. In LINQ to Objects, you’d normally process each whole group in turn, whereas the order in LINQ to Rx means the output of listing 12.15 looks like this: Value: 0; Group: 0 Value: 1; Group: 1 Value: 2; Group: 2 Value: 3; Group: 0 Listing 12.14 Filtering and projecting in LINQ to Rx Listing 12.15 Grouping numbers mod 3 362 CHAPTER 12 LINQ beyond collections Value: 4; Group: 1 Value: 5; Group: 2 Value: 6; Group: 0 Value: 7; Group: 1 Value: 8; Group: 2 Value: 9; Group: 0 This makes perfect sense when you think of the push model, and in some cases it means that operations that would’ve required a lot of data buffering in LINQ to Objects can be implemented in LINQ to Rx much more efficiently. As a final example, let’s look at another operator that uses multiple sequences. FLATTENING LINQ to Rx supplies a few overloads of SelectMany, and the idea is still the same as in LINQ to Objects: each item in the original sequence produces a new sequence, and the result is the combination of all these new sequences, flattened. The following list- ing shows this in action—it’s a little like listing 11.16, when we first discussed Select- Many in LINQ to Objects. var query = from x in Observable.Range(1, 3) from y in Observable.Range(1, x) select new { x, y }; query.Subscribe(Console.WriteLine); Here are the results, which should be reasonably predictable: { x = 1, y = 1 } { x = 2, y = 1 } { x = 2, y = 2 } { x = 3, y = 1 } { x = 3, y = 2 } { x = 3, y = 3 } In this case, the results are deterministic, but that’s only because, by default, Observable.Range emits items on the current thread. It’s entirely possible to have multiple sequences being produced on multiple threads. For fun, you might want to change the second call to Observable.Range to specify Scheduler.ThreadPool as a third argument. In that case, each of the inner sequences comes out in order with respect to itself, but those separate sequences can be mixed up among each other. Imagine a sports stadium with one official firing a starting pistol for several different races in quick succession—even if you know the winner of each race, you don’t know which race will finish first. Apologies if this makes you want to go and lie down. If it’s any consolation, it gives me the same feeling. I do find it fascinating at the same time though. WHAT’S IN AND WHAT’S OUT? You already know that a let clause works by just calling Select, so that naturally works in LINQ to Rx, but not all LINQ to Objects operators are implemented in LINQ to Rx. Listing 12.16 SelectMany producing multiple ranges 363Inverting the query model with LINQ to Rx The missing operators are generally the ones that would have to buffer all their out- put and return a new observable. For example, there’s no Reverse method, and no OrderBy. C# is quite happy with that—it just won’t let you use an orderby clause in a query expression based on observables. There’s a Join method, but that doesn’t deal with observables directly—it handles join plans. This is part of the Rx implementation of the join-calculus, and it’s well beyond the scope of this book. Likewise there’s no GroupJoin method, so join...into isn’t supported. For the various LINQ standard query operators that aren’t covered by the query expression syntax—and to see the range of extra methods it makes available—see the System.Reactive documentation. Although you may start off being disappointed about the familiar functionality from LINQ to Objects that’s missing in LINQ to Rx (usually because it just doesn’t make sense), you may be surprised by how rich the set of available methods really is. Many of the new methods are then ported to LINQ to Objects in the System.Interactive assembly. 12.5.4 What’s the point? I’m well aware that I haven’t provided any compelling reasons to use LINQ to Rx yet. This is deliberate, as I don’t intend to show a full, useful example—it’s incidental to the point of this chapter, and would take too much space. But Rx provides an elegant way of thinking about all kinds of asynchronous processes, such as normal .NET events (which can be viewed as an observable using Observable.FromEvent), asynchronous I/O, and calls to web services. It provides a way of managing the complexity and con- currency in an efficient manner. There’s no doubt that it is harder to get your head around than LINQ to Objects, but if you’re in the kind of situation where it’d be use- ful, you’re already facing a mountain of complexity. The reason I wanted to cover Rx in this book, despite not being able to do it any sort of justice, is because it shows why LINQ was designed the way it was. Although there are conversion methods available between IEnumerable and IObservable, there’s no inheritance relationship. If the language had made any requirement that the types involved in LINQ had to be pull sequences, there would’ve been no query expression support for Rx at all. It would’ve been even more disastrous if extension methods had been limited to IEnumerable in some way. Likewise, you’ve seen that not all the normal LINQ operators are applicable to Rx, which is why it’s important that the language specifies query translations in terms of a pattern that should be sup- ported as far as it makes sense for the given provider. I hope you have a sense that even though the push and pull models are very different to work with, LINQ acts as a sort of unifying force where possible. You may be relieved to hear that our last topic is a lot simpler—it’s back on the home ground of LINQ to Objects, but this time we’re writing our own extension methods. 364 CHAPTER 12 LINQ beyond collections 12.6 Extending LINQ to Objects One of the nice things about LINQ is that it’s extensible. Not only can you come up with your own query providers and data models, you can also add to existing ones. In my experience, the most common situation where this is useful is with LINQ to Objects. If you need a particular type of query that isn’t directly supported (or is awk- ward or inefficient with the standard query operators), you can write your own. Of course, writing a general-purpose generic method can be more challenging than just solving your immediate problem, but if you find yourself writing similar code a few times, it’s worth considering whether you could refactor it into a new operator. Personally, I enjoy writing query operators. There are interesting technical chal- lenges, but it rarely requires a huge amount of code, and the results can be elegant. In this section, we’ll look at some of the ways you can make your custom operators behave efficiently and predictably, followed by a full sample for selecting a random element from a sequence. 12.6.1 Design and implementation guidelines Most of these guidelines may seem fairly obvious, but this section can form a useful checklist when you write an operator. UNIT TESTS It’s generally pretty easy to write a good set of unit tests for operators, although you may be surprised at how many you end up with for what originally appeared to be sim- ple code. Don’t forget to test corner cases, such as empty sequences, as well as invalid arguments. MoreLINQ (http://code.google.com/p/morelinq/) has some helper methods in its unit test project that you might want to use for your own tests. ARGUMENT CHECKING Good methods check their arguments, but there’s a problem when it comes to LINQ operators. Many operators return another sequence, as you’ve already seen, and itera- tor blocks are the easiest way to implement this functionality. But you should really perform the argument checking as soon as your method is called, rather than waiting until the caller decides to iterate over the results. If you’re going to use an iterator block, split your method into two: perform argument checking in a public method and then call a private method to do the iteration. OPTIMIZATION IEnumerable itself is fairly weak in terms of the operations it supports, but the execution-time type of a sequence you’re working on may have considerably more functionality. For example, the Count() operator will always work, but it’ll generally be an O(n) operation. If you call it on an implementation of ICollection, though, it can use the Count property directly, which will generally be O(1). In .NET 4, this optimization is extended to cover ICollection as well. Likewise, retriev- ing a specific element by index is slow in the general case, but can be efficient if the sequence implements IList. 365Extending LINQ to Objects If your operator can benefit from these optimizations, you can have different exe- cution paths depending on the execution-time type. To test the slow path in unit tests, you can always call Select(x => x) on a List to retrieve a nonlist sequence. LinkedList can test the case where you want an ICollection that doesn’t implement IList. DOCUMENTATION It’s important to document what your code will do with its inputs, and also the expected performance of the operator. This is particularly important if your method needs to work with multiple sequences: which one will be evaluated first, and how far? Does your code stream its data, buffer it, or is it a mixture? Does it use deferred or immediate execution? Can any parameters be null, and if so, does that have a special meaning? ITERATE ONCE WHERE POSSIBLE At an interface level, IEnumerable will let you iterate over the same sequence mul- tiple times—you can have multiple iterators active at the same time over the same sequence, potentially. But this is rarely a good idea within an operator. Wherever pos- sible, it’s wise to iterate over your input sequences just once. This means your code will work even for nonrepeatable sequences, such as lines read from a network stream. If you do need to read the sequence multiple times (and you don’t want to buffer the whole sequence, like Reverse does), you should draw particular attention to this in the documentation. REMEMBER TO DISPOSE OF ITERATORS In most cases, you can use a foreach statement to iterate over your data source. But it’s sometimes useful to treat the first item differently, in which case using an iterator directly can lead to the simplest code. In that situation, remember to include a using block for the iterator. You probably aren’t used to disposing of iterators yourself because normally foreach does it for you, which can make it hard to spot the bug. SUPPORT CUSTOM COMPARISONS Many LINQ operators have overloads that allow you to specify an appropriate IEqualityComparer or IComparer. If you’re building a general-purpose library for others (potentially developers who you aren’t in contact with), it may be worth pro- viding similar overloads yourself. On the other hand, if you’re the sole user, or it’s just going to be members of your team using it, you can do this on a need-to-implement basis. It’s easy, though: typically the simpler overloads just call a more complex one, passing EqualityComparer.Default or Comparer.Default as the comparison. Now that I’ve talked the talk, let’s check whether I can actually walk the walk. 12.6.2 Sample extension: selecting a random element The aim of the extension method we’ll look at here is simple: given a sequence and an instance of Random, return a random element from the sequence. You could add an overload that doesn’t require the instance of Random, but I prefer to make the 366 CHAPTER 12 LINQ beyond collections dependency on a random number generator explicit. Randomness is a tricky topic for various reasons, and rather than discuss it here, I’ve included an article on the book’s website (see http://mng.bz/h483). Also for reasons of space, I haven’t included the XML documentation or unit tests in the following listing, but of course they’re in the downloadable code. public static T RandomElement(this IEnumerable source, Random random) { if (source == null) { throw new ArgumentNullException("source"); } if (random == null) { throw new ArgumentNullException("random"); } ICollection collection = source as ICollection; if (collection != null) { int count = collection.Count; if (count == 0) { throw new InvalidOperationException("Sequence was empty."); } int index = random.Next(count); return source.ElementAt(index); } using (IEnumerator iterator = source.GetEnumerator()) { if (!iterator.MoveNext()) { throw new InvalidOperationException("Sequence was empty."); } int countSoFar = 1; T current = iterator.Current; while (iterator.MoveNext()) { countSoFar++; if (random.Next(countSoFar) == 0) { current = iterator.Current; } } return current; } } Listing 12.17 doesn’t show the technique of splitting an extension method into argu- ment validation and then implementation, because it doesn’t use an iterator block. Look back at the implementation of the Where operator in section 10.3.3 for an Listing 12.17 Extension method to choose a random element from a sequence Validates argumentsB Optimizes for collectionsC ElementAt optimizes further Handles slow caseD Replaces current guess with appropriate probability E 367Summary example of this. No custom comparisons are required either, but apart from that, every item on the checklist is appropriate. First you validate your arguments in the obvious way B. Things get more interest- ing where the source sequence implements ICollection C.9 This allows you to take the count cheaply and then generate a single random number to work out which ele- ment to pick. You don’t explicitly handle the case where the source sequence imple- ments IList; instead, you rely on ElementAt to do that for you (as it’s documented to do). If you’re dealing with a noncollection sequence (such as the result of another query operator), you want to avoid taking the count and then picking an element; that would require you to either buffer the contents of the sequence or iterate over it twice. Instead you step through it once, explicitly fetching the iterator D so that you can test for an empty sequence easily. The clever bit10 is at E—you replace your cur- rent idea of a random element with the element from the iterator with a probability of 1/n, where n is the number of elements you’ve seen so far. There’s a one-half chance of replacing the first element with the second, a one-third chance of replacing the result after two elements with the third element, and so on. The final result is that each element in the sequence has an equal chance of being picked, and you’ve man- aged to iterate just once. Of course, the important point isn’t what this particular method does—it’s the potential issues that had to be considered as you implemented it. Once you know what to look for, it really doesn’t take much effort to implement a robust method like this, and your personal toolbox will grow over time. 12.7 Summary Phew! This chapter has been the exact opposite of most of the rest of the book. Instead of focusing on a single topic in great detail, we’ve covered a range of LINQ technologies, but at a shallow level. I wouldn’t expect you to feel particularly familiar with any one of the specific tech- nologies we’ve looked at here, but I hope you have a deeper understanding of why LINQ is important. It’s not about XML, or in-memory queries, SQL queries, observ- ables, or enumerators—it’s about consistency of expression, and giving the C# com- piler the opportunity to validate your queries to at least some extent, regardless of their final execution platform. You should now appreciate why expression trees are so important that they’re among the few framework elements that the C# compiler has direct intimate knowledge of (along with strings, IDisposable, IEnumerable, and Nullable, for example). They act as passports, allowing behavior to cross the border of the local machine, expressing logic in whatever foreign tongue is catered for by a LINQ provider. 9 The downloadable code contains the same test for implementations of ICollection, just like Count() does in .NET 4. It’s exactly the same block of code, just with a different type and a different variable name. 10 I’m comfortable claiming this is clever because, even though it’s my implementation, it’s not my idea. 368 CHAPTER 12 LINQ beyond collections It’s not just expression trees—we’ve also relied on the query expression translation employed by the compiler, and the way that lambda expressions can be converted to both delegates and expression trees. But extension methods are also important, as without them each provider would have to give implementations of all the relevant methods. If you look back at all the new features of C#, you’ll find few that don’t con- tribute significantly to LINQ in some way or another. That’s part of the reason for this chapter’s existence: to show the connections between all the features. I shouldn’t wax lyrical for too long, though. As well as the upsides of LINQ, we’ve seen a few gotchas. LINQ won’t always allow you to express everything you need in a query, nor does it hide all the details of the underlying data source. When it comes to database LINQ providers, the impedance mismatches that have caused developers so much trouble in the past are still with us: you can reduce their impact with ORM sys- tems and the like, but without a proper understanding of the query being executed on your behalf, you’re likely to run into significant issues. In particular, don’t think of LINQ as a way of removing your need to understand SQL—think of it as a way of hiding the SQL when you’re not interested in the details. Likewise, in order to plan an effec- tive parallel query, you’ve got to know where ordering matters and where it doesn’t, and perhaps help the framework along a bit by giving it more tuning information. Since .NET 3.5 came out, I’ve been delighted to see how wholeheartedly the com- munity has embraced LINQ. Likewise, there have been plenty of interesting uses of the features of C# 4, which you’ll see in the next part of the book. Part 4 C# 4: Playing nicely with others C# 4 is a funny beast. It doesn’t have the “several, almost unrelated, major new features” feeling of C# 2, nor the “all in the cause of LINQ” feeling of C# 3. Instead, the new features of C# 4 fall somewhere between the two. Interoperabil- ity is a major theme, but many of the features are equally useful even if you never need to work with other environments. My personal favorite features from C# 4 are optional parameters and named arguments. They’re relatively simple but can be put to good use in many places, improving the readability of code and generally making life more pleasant. Do you waste time working out which argument means what? Put some names on them. Are you tired of writing endless overloads to avoid callers having to specify everything? Make some parameters optional. If you work with COM, C# 4 will be a breath of fresh air for you. To start with, the features I just described make some APIs much simpler to work with, where the component designers have pretty much assumed that you’ll be working with a language supporting optional parameters and named arguments. Beyond that, there’s a better deployment story, support for named indexers, and a helpful shortcut to avoid having to pass arguments by reference everywhere. The biggest feature of C# 4—dynamic typing—also makes COM integration easier. We’ll look at all of these areas in chapter 13, along with the brain-busting topic of generic variance applied to interfaces and delegates. Don’t worry; we’ll take that reasonably slowly, and the best part is that most of the time you don’t need to know the details—it just makes code work where you might’ve expected it to in C# 3 anyway! Chapter 14 covers dynamic typing and the Dynamic Language Runtime (DLR). This is an enormous topic. I’ve concentrated on how the C# language implements dynamic typing, but we’ll also look at a few examples of interoperating with dynamic languages such as IronPython, and see examples of how a type can dynamically respond to method calls, property accesses, and so on. It’s worth applying a little per- spective here: the fact that this is a major feature doesn’t mean that you should expect to see dynamic expressions cropping up all over your code base. This won’t be as per- vasive as LINQ, for example, but when you do want dynamic typing, you’ll find it well implemented in C# 4. 371 Minor changes to simplify code Just as with previous versions, C# 4 has a few minor features that don’t merit indi- vidual chapters to themselves. In fact, there’s only one really big feature in C# 4— dynamic typing—which we’ll cover in the next chapter. The changes we’ll cover here just make C# that little bit more pleasant to work with, particularly if you work with COM on a regular basis. These features generally make code clearer, remove drudgery from COM calls, or simplify deployment. This chapter covers  Optional parameters  Named arguments  Streamlining ref parameters in COM  Embedding COM Primary Interop Assemblies  Calling named indexers declared in COM  Generic variance for interfaces and delegates  Changes in locking and field-like events 372 CHAPTER 13 Minor changes to simplify code Will any of these features make your heart race with excitement? It’s unlikely. But they’re nice features all the same, and some of them may be widely applicable. Let’s start by looking at how we call methods. 13.1 Optional parameters and named arguments These are perhaps the Batman and Robin1 features of C# 4. They’re distinct, but usu- ally seen together. I’ll keep them apart for the moment so we can examine each in turn, but then we’ll use them together for some more interesting examples. We’ll start by looking at optional parameters. 13.1.1 Optional parameters Visual Basic has had optional parameters for ages, and they’ve been in the CLR from .NET 1.0. The concept is as obvious as it sounds: some parameters are optional, so their values don’t have to be explicitly specified by the caller. Any parameter that hasn’t been specified as an argument by the caller is given a default value. MOTIVATION Optional parameters are usually used when several values are required for an opera- tion, and the same values are used a lot of the time. For example, suppose you wanted to read a text file; you might want to provide a method that allows the caller to specify the name of the file and the encoding to use. The encoding is almost always UTF-8, though, so it’s nice to be able to use that automatically if it’s all you need. Historically the idiomatic way of allowing this in C# has been to use method over- loading: declare one method with all the possible parameters, and others that call that 1 Or Cavalleria rusticana and Pagliacci if you’re feeling more highly cultured. Parameters and arguments This section obviously talks about parameters and arguments a lot. In casual conver- sation, the two terms are often used interchangeably, but I’ll use them in line with their formal definitions. Just to remind you, a parameter (also known as a formal parameter) is the variable that’s part of the method or indexer declaration. An argu- ment is an expression used when calling the method or indexer. For example, con- sider this snippet: void Foo(int x, int y) { // Do something with x and y } ... int a = 10; Foo(a, 20); Here the parameters are x and y, and the arguments are a and 20. 373Optional parameters and named arguments method, passing in default values where appropriate. For instance, you might create methods like this: public IList LoadCustomers(string filename, Encoding encoding) { ... } public IList LoadCustomers(string filename) { return LoadCustomers(filename, Encoding.UTF8); } This works fine for a single parameter, but it becomes complicated when there are multiple options, as each extra option doubles the number of possible overloads. If two of them are of the same type, this approach would naturally lead to multiple methods with the same signature, which is invalid. Often the same set of overloads is also required for multiple parameter types. For example, the XmlReader.Create() method can create an XmlReader from a Stream, a TextReader, or a string, but it also provides the option of specifying an XmlReaderSettings and other arguments. Due to this duplication, there are 12 overloads for the method. This could be significantly reduced with optional parameters. Let’s see how it’s done. DECLARING OPTIONAL PARAMETERS AND OMITTING THEM WHEN SUPPLYING ARGUMENTS Making a parameter optional is as simple as supplying a default value for it, using what looks like a variable initializer. Figure 13.1 shows a method with three parameters: two are optional, one is required. All this method does is print out the arguments, but that’s enough to see what’s going on. The following listing gives the full code and calls the method three times, spec- ifying a different number of arguments for each call. static void Dump(int x, int y = 20, int z = 30) { Console.WriteLine("x={0} y={1} z={2}", x, y, z); } ... Dump(1, 2, 3); Dump(1, 2); Dump(1); Listing 13.1 Declaring and calling a method with optional parameters Do real work here Default to UTF-8 Declares method with optional parametersB Calls method with all arguments C Omits one argumentD Omits two argumentsE void Dump(int x, int y = 20, int z = 30) Required parameter Default values Optional parameters Figure 13.1 Declaring optional parameters 374 CHAPTER 13 Minor changes to simplify code The optional parameters are the ones with default values specified B. If the caller doesn’t specify y, its initial value will be 20, and likewise z has a default value of 30. The first call C explicitly specifies all the arguments; the remaining calls (D and E) omit one or two arguments, respectively, so the default values are used. When there’s one argument missing, the compiler assumes that the final parameter has been omit- ted, then the penultimate one, and so on. The output is as follows: x=1 y=2 z=3 x=1 y=2 z=30 x=1 y=20 z=30 Note that although the compiler could use some clever analysis of the types of the optional parameters and the arguments in order to work out what’s been left out, it doesn’t: it assumes that you’re supplying arguments in the same order as the parame- ters.2 This means that the following code is invalid: static void TwoOptionalParameters(int x = 10, string y = "default") { Console.WriteLine("x={0} y={1}", x, y); } ... TwoOptionalParameters("second parameter"); This tries to call the TwoOptionalParameters method specifying a string for the first argument. There’s no overload with a first parameter that’s convertible from a string, so the compiler issues an error. This is a good thing—overload resolution is tricky enough (particularly when generic type inference gets involved) without the compiler trying all kinds of different permutations to find something you might be trying to call. If you want to omit the value for one optional parameter but specify a later one, you need to use named arguments. RESTRICTIONS ON OPTIONAL PARAMETERS There are a few rules for optional parameters. All optional parameters must come after required parameters. The exception to this is a parameter array (as declared with the params modifier), which still has to come at the end of a parameter list, but can come after optional parameters. A parameter array can’t be declared as an optional parameter—if the caller doesn’t specify any values for it, an empty array will be used instead. Optional parameters can’t have ref or out modifiers either. An optional parameter can be of any type, but there are restrictions on the default value specified. You can always use constants: numeric and string literals, null, const members, enum members, and the default(T) operator. Additionally, for value types, you can call the parameterless constructor, although this is equivalent to using the default (...) operator anyway. There has to be an implicit conversion from the specified value to the parameter type, but this must not be a user-defined conversion. Table 13.1 shows some examples of valid parameter lists. 2 Unless you’re using named arguments, of course—you’ll learn about those soon. INVALID Error! 375Optional parameters and named arguments In contrast, table 13.2 shows some invalid parameter lists and explains why they’re not allowed. The fact that the default value has to be constant is a pain in two different ways. One of them is familiar from a slightly different context, as we’ll look at now. VERSIONING AND OPTIONAL PARAMETERS The restrictions on default values for optional parameters may remind you of the restrictions on const fields or attribute values, and they behave very similarly. In both cases, when the compiler references the value, it copies it directly into the output. The generated IL acts exactly as if your original source code had contained the default value. This means if you ever change the default value without recompiling everything that references it, the old callers will still be using the old default value. To make this concrete, imagine this set of steps: Table 13.1 Valid method parameter lists using optional parameters Declaration Notes Foo(int x, int y = 10) Numeric literal used for default value Foo(decimal x = 10) Implicit built-in conversion from int to decimal Foo(string name = "default") String literal used for default value Foo(DateTime dt = new DateTime()) Zero value of DateTime Foo(DateTime dt = default(DateTime)) Alternative syntax for the zero value Foo(T value = default(T)) Default value operator works with type param- eters Foo(int? x = null) Nullable conversion Foo(int x, int y = 10, params int[] z) Parameter array after optional parameters Table 13.2 Invalid method parameter lists using optional parameters Declaration (invalid) Notes Foo(int x = 0, int y) Required non-params parameter after optional parameter Foo(DateTime dt = DateTime.Now) Default values must be constants Foo(XName name = "default") Conversion from string to XName is user-defined Foo(params string[] names = null) Parameter arrays can’t be optional Foo(ref string name = "default") ref/out parameters can’t be optional 376 CHAPTER 13 Minor changes to simplify code 1 Create a class library (Library.dll) with a class like this: public class LibraryDemo { public static void PrintValue(int value = 10) { System.Console.WriteLine(value); } } 2 Create a console application (Application.exe) that references the class library: public class Program { static void Main() { LibraryDemo.PrintValue(); } } 3 Run the application—it’ll print 10, predictably. 4 Change the declaration of PrintValue as follows, and then recompile just the class library: public static void PrintValue(int value = 20) 5 Rerun the application—it’ll still print 10. The value has been compiled directly into the executable. 6 Recompile the application and rerun it—this time it’ll print 20. This versioning issue can cause bugs that are hard to track down, because all the code looks correct. Essentially, you’re restricted to using genuine constants that should never change as default values for optional parameters.3 There’s one benefit of this setup: it gives the caller a guarantee that the value it knew about at compile time is the one that’ll be used. Developers may feel more comfortable with that than with a dynamically computed value, or one that depends on the version of the library used at execution time. Of course, this also means you can’t use any values that can’t be expressed as con- stants anyway. You can’t create a method with a default value of “the current time,” for example. MAKING DEFAULTS MORE FLEXIBLE WITH NULLITY Fortunately, there’s a way around the restriction that default values must be constants. Essentially, you introduce a magic value to represent the default, and then replace that magic value with the real default within the method itself. If the phrase magic value bothers you, I’m not surprised, but we’ll use null for the magic value, which already represents the absence of a normal value. If the parameter type would normally be a value type, we’ll simply make it the corresponding nullable value type, at which point we can still specify that the default value is null. 3 Or you could just accept that you’ll need to recompile everything if you change the value. In many contexts that’s a reasonable trade-off. 377Optional parameters and named arguments As an example of this, let’s look at a situation similar to the one I used to introduce the whole topic: allowing the caller to supply an appropriate text encoding to a method, but defaulting to UTF-8. You can’t specify the default encoding as Encoding.UTF8 as that’s not a constant value, but you can treat a null parameter value as “use the default.” To demonstrate how you can handle value types, let’s make the method append a timestamp to a text file with a message. We’ll default the encoding to UTF-8 and the timestamp to the current time. The following listing shows the com- plete code and a few examples of using it. static void AppendTimestamp(string filename, string message, Encoding encoding = null, DateTime? timestamp = null) { Encoding realEncoding = encoding ?? Encoding.UTF8; DateTime realTimestamp = timestamp ?? DateTime.Now; using (TextWriter writer = new StreamWriter(filename, true, realEncoding)) { writer.WriteLine("{0:s}: {1}", realTimestamp, message); } } ... AppendTimestamp("utf8.txt", "First message"); AppendTimestamp("ascii.txt", "ASCII", Encoding.ASCII); AppendTimestamp("utf8.txt", "Message in the future", null, new DateTime(2030, 1, 1)); Listing 13.2 shows a few nice features of this approach. First, it solves the versioning problem. The default values for the optional parameters are null B, but the effective val- ues are “the UTF-8 encoding” and “the current date and time.” Neither of these could be expressed as constants, and should you ever want to change the effective default— for example, to use the current UTC time instead of the local time—you could do so without having to recompile everything that called AppendTimestamp. Of course, changing the effective default changes the behavior of the method; you need to take the same sort of care with this as you would with any other code change. At this point, you (as the library author) are in charge of the versioning story—you’re taking respon- sibility for not breaking clients, effectively. At least it’s more familiar territory; you know that all callers will experience the same behavior, regardless of recompilation. This listing also introduces an extra level of flexibility. Not only do optional param- eters mean you can make the calls shorter, but having a specific “use the default” value means that should you ever wish to, you can explicitly make a call allowing the method to choose the appropriate value. At the moment, this is the only way you know to spec- ify the timestamp explicitly without also providing an encoding D, but that’ll change when we look at named arguments. Listing 13.2 Using null default values to handle nonconstant situations Two required parametersTwo optional parameters B Null coalescing operator for convenienceC Explicit use of null D 378 CHAPTER 13 Minor changes to simplify code The optional parameter values are simple to deal with, thanks to the null coalesc- ing operator C. This example uses separate variables for the sake of printed format- ting, but in real code you’d probably use the same expressions directly in the calls to the StreamWriter constructor and the WriteLine method. There are two downsides to this approach: first, it means that if a caller accidentally passes in null due to a bug, it’ll get the default value instead of an exception. In cases where you’re using a nullable value type and callers will either explicitly use null or have a non-nullable argument, that’s not much of a problem, but for reference types it could be an issue. On a related note, it requires that you don’t want to use null as a “real” value.4 There are occasions where you want null to mean null, and if you don’t want that to be the default value, you’ll have to find a different constant or just leave the parameter as a required one. But in other cases, where there isn’t an obvious constant value that’ll clearly always be the right default, I recommend this approach to optional parameters as one that’s easy to follow consistently and removes some of the normal difficulties. We’ll need to look at how optional parameters affect overload resolution, but it makes sense to wait until you’ve seen how named arguments work. Speaking of which… 13.1.2 Named arguments The basic idea of named arguments is that when you specify an argument value, you can also specify the name of the parameter it’s supplying the value for. The compiler then makes sure that there is a parameter of the right name and uses the value for that parameter. Even on its own, this can increase readability in some cases. In practice, named arguments are most useful in cases where optional parameters are also likely to appear, but we’ll look at the simple situation first. INDEXERS, OPTIONAL PARAMETERS, AND NAMED ARGUMENTS You can use optional parameters and named arguments with indexers as well as methods. But this is only useful for indexers with more than one parameter: you can’t access an indexer without specifying at least one argument anyway. Given this limitation, I don’t expect to see the feature used much with indexers, and I haven’t demonstrated it in the book. It works exactly as you’d expect it to, though. I’m sure you’ve seen code that looks something like this: MessageBox.Show("Please do not press this button again", // text "Ouch!"); // title I’ve chosen a pretty tame example; it can get a lot worse when there are loads of argu- ments, especially if a lot of them are the same type. But this is still realistic; even with 4 We almost need a second null-like special value, meaning “please use the default value for this parameter,” and then we could allow that special value to be supplied either automatically for missing arguments or explic- itly in the argument list. I’m sure this would cause dozens of problems, but it’s an interesting thought exper- iment. 379Optional parameters and named arguments just two parameters, I’d find myself guessing which argument meant what based on the text when reading this code, unless it had comments like the ones I have here. There’s a problem though: comments can lie about the code they describe. Nothing is checking them at all. In contrast, named arguments ask the compiler to help. SYNTAX All you need to do to the previous example to make the code clearer is prefix each argument with the name of the corresponding parameter and a colon: MessageBox.Show(text: "Please do not press this button again", caption: "Ouch!"); Admittedly, you now don’t get to choose the name you find most meaningful (I prefer title to caption), but at least you’ll know if you get something wrong. Of course, the most common way in which you could get something wrong here is to get the arguments the wrong way around. Without named arguments, this would be a problem: you’d end up with the pieces of text switched in the message box. With named arguments, the ordering becomes largely irrelevant. You can rewrite the previ- ous code like this: MessageBox.Show(caption: "Ouch!", text: "Please do not press this button again"); You’d still have the right text in the right place, because the compiler would work out what you meant based on the names. For another example, look at the StreamWriter constructor call in listing 13.2. The second argument is just true—what does this mean? Is it going to force a stream flush after every write? Include a byte order mark? Append to an existing file instead of creating a new one? Here’s the equivalent call using named arguments: new StreamWriter(path: filename, append: true, encoding: realEncoding) In both of these examples, you’ve seen how named arguments effectively attach semantic meaning to values. In the never-ending quest to make code communicate bet- ter with humans as well as computers, this is a definite step forward. I’m not suggesting that named arguments should be used when the meaning is already obvious, of course. Like all features, it should be used with discretion and thought. Named arguments with out and ref If you want to specify the name of an argument for a ref or out parameter, you put the ref or out modifier after the name and before the argument. Using int.Try- Parse as an example, you might have code like this: int number; bool success = int.TryParse("10", result: out number); 380 CHAPTER 13 Minor changes to simplify code To explore some other aspects of the syntax, the following listing shows a method with three integer parameters, just like the one we used to start looking at optional parameters. static void Dump(int x, int y, int z) { Console.WriteLine("x={0} y={1} z={2}", x, y, z); } ... Dump(1, 2, 3); Dump(x: 1, y: 2, z: 3); Dump(z: 3, y: 2, x: 1); Dump(1, y: 2, z: 3); Dump(1, z: 3, y: 2); The output is the same for each call in listing 13.3: x=1, y=2, z=3. This code effec- tively makes the same method call in five different ways. It’s worth noting that there are no tricks in the method declaration B; you can use named arguments with any method that has parameters. First, you call the method in the normal way, without using any new features C. This is a sort of control point to make sure that the other calls really are equivalent. You then make two calls to the method using just named arguments D. The second of these calls reverses the order of the arguments, but the result is still the same, because the arguments are matched up with the parameters by name, not position. Finally, there are two calls using a mixture of named arguments and positional arguments E. A positional argument is one that isn’t named, so every argument in valid C# 3 code is a positional argument from the point of view of C# 4. Figure 13.2 shows how the final line of code works. All named arguments have to come after positional arguments—you can’t switch between the styles. Positional arguments always refer to the corresponding parameter in the method declaration—you can’t make positional arguments skip a parameter by specifying it later with a named argument. This means that these method calls would both be invalid:  Dump(z: 3, 1, y: 2)—Positional argu- ments must come before named ones.  Dump(2, x: 1, z: 3)—x has already been specified by the first positional argument, so you can’t specify it again with a named argument. Now, although in this particular case the method calls are equivalent, that’s not always the case. Let’s look at why reordering arguments might change behavior. Listing 13.3 Simple examples of using named arguments Declares method as normalB Calls method as normal C Specifies names for all argumentsD Specifies names for some argumentsE static void Dump(int x, int y, int z) Dump(1, z: 3, y: 2) Positional argument Named arguments Figure 13.2 Positional and named arguments in the same call 381Optional parameters and named arguments ARGUMENT EVALUATION ORDER You’re used to C# evaluating its arguments in the order they’re specified, which, until C# 4, has always been the order in which the parameters have been declared too. In C# 4, only the first part is still true: the arguments are still evaluated in the order they’re written, even if that’s not the same as the order in which they’re declared as parameters. This matters if evaluating the arguments has side effects. It’s usually worth trying to avoid having side effects in arguments, but there are cases where it can make the code clearer. A more realistic rule is to try to avoid side effects that might interfere with each other. For the sake of demonstrating execution order, we’ll break both of these rules. Please don’t treat this as a recommendation that you do the same thing. First we’ll create a relatively harmless example, introducing a method that logs its input and returns it—a sort of logging echo. We’ll use the return values of three calls to this to call the Dump method (which isn’t shown, as it hasn’t changed). The follow- ing listing shows two calls to Dump that result in slightly different output. static int Log(int value) { Console.WriteLine("Log: {0}", value); return value; } ... Dump(x: Log(1), y: Log(2), z: Log(3)); Dump(z: Log(3), x: Log(1), y: Log(2)); The results of running listing 13.4 show what happens: Log: 1 Log: 2 Log: 3 x=1 y=2 z=3 Log: 3 Log: 1 Log: 2 x=1 y=2 z=3 In both cases, the parameters x, y, and z in the Dump method still have the values 1, 2, and 3, respectively. But you can see that although they were evaluated in that order in the first call (which was equivalent to using positional arguments), the second call evaluated the value used for the z parameter first. You can make the effect even more significant by using side effects that change the results of the argument evaluation, as shown in the following listing, again using the same Dump method. Listing 13.4 Logging argument evaluation 382 CHAPTER 13 Minor changes to simplify code int i = 0; Dump(x: ++i, y: ++i, z: ++i); i = 0; Dump(z: ++i, x: ++i, y: ++i); The results of listing 13.5 may be best expressed in terms of the blood spatter pattern at a murder scene, after someone maintaining code like this has gone after the origi- nal author with an axe. Yes, technically speaking the last line results are x=2 y=3 z=1, but I’m sure you see what I’m getting at. Just say No to code like this. By all means, reorder your arguments for the sake of readability. You may think that laying out a call to MessageBox.Show with the title coming above the text in the code itself reflects the onscreen layout more closely, for example. If you want to rely on a particular evaluation order for the arguments, though, introduce some local vari- ables to execute the relevant code in separate statements. The compiler won’t care either way—it’ll follow the rules of the spec—but this reduces the risk of a “harmless refactoring” that inadvertently introduces a subtle bug. To return to cheerier matters, let’s combine the two features (optional parameters and named arguments) and see how much tidier the code can be. 13.1.3 Putting the two together Optional parameters and named arguments work in tandem with no extra effort required on your part. It’s not uncommon to have a bunch of parameters where there are obvious defaults, but where it’s hard to predict which ones a caller will want to specify explicitly. Figure 13.3 shows just about every combination: a required parame- ter, two optional parameters, a positional argument, a named argument, and a miss- ing argument for an optional parameter. Going back to an earlier example, in listing 13.2 you wanted to append a timestamp to a file using the default encoding of UTF-8, but with a particu- lar timestamp. That code used null for the encoding argument, but now you can write the same code more sim- ply, as shown in the following listing. static void AppendTimestamp(string filename, string message, Encoding encoding = null, DateTime? timestamp = null) { } ... Listing 13.5 Abusing argument evaluation order Listing 13.6 Combining named arguments and optional parameters Same implementation as before static void Dump(int x, int y = 20, int z = 30) Dump(10, z: 3) 20 Figure 13.3 Mixing named arguments and optional parameters 383Optional parameters and named arguments AppendTimestamp("utf8.txt", "Message in the future", timestamp: new DateTime(2030, 1, 1)); In this fairly simple situation, the benefit isn’t particularly huge, but in cases where you want to omit three or four arguments but specify the final one, it’s a real blessing. You’ve seen how optional parameters reduce the need for huge long lists of over- loads, but one specific pattern where this is worth mentioning is with respect to immutability. IMMUTABILITY AND OBJECT INITIALIZATION One aspect of C# 4 that disappoints me somewhat is that it hasn’t done much explicitly to make immutability easier. Immutable types are a core part of functional program- ming, and C# has been gradually supporting the functional style more and more… except for immutability. Object and collection initializers make it easy to work with mutable types, but immutable types have been left out in the cold. (Automatically implemented properties fall into this category too.) Fortunately, though they’re not particularly designed to aid immutability, named arguments and optional parameters allow you to write object initializer–like code that calls a constructor or other factory method. For instance, suppose you were creating a Message class, which required a from address, a to address, and a body, with the subject and attachment being optional. (We’ll stick with single recipients in order to keep the example as simple as possible.) You could create a mutable type with appropriate writable properties, and construct instances like this: Message message = new Message { From = "skeet@pobox.com", To = "csharp-in-depth-readers@everywhere.com", Body = "Hope you like the third edition", Subject = "A quick message" }; That has two problems: first, it doesn’t enforce the required data to be provided. You could force those to be supplied to the constructor, but then (before C# 4) it wouldn’t be obvious which argument meant what: Message message = new Message( "skeet@pobox.com", "csharp-in-depth-readers@everywhere.com", "Hope you like the third edition") { Subject = "A quick message" }; The second problem is that this initialization pattern simply doesn’t work for immuta- ble types. The compiler has to call a property setter after it has initialized the object. But you can use optional parameters and named arguments to come up with some- thing that has the nice features of the first form (only specifying what you’re interested Encoding is omitted Named timestamp argument 384 CHAPTER 13 Minor changes to simplify code in and supplying names) without losing the validation of which aspects of the message are required or the benefits of immutability. The following listing shows a possible con- structor signature and the construction step for the same message as before. public Message(string from, string to, string body, string subject = null, byte[] attachment = null) { } ... Message message = new Message( from: "skeet@pobox.com", to: "csharp-in-depth-readers@everywhere.com", body: "I hope you like the third edition", subject: "A quick message" ); I really like this in terms of readability and general cleanliness. You don’t need hun- dreds of constructor overloads—just one with some of the parameters being optional. The same syntax will also work with static creation methods, unlike object initializers. The only downside is that it really relies on your code being consumed by a language that supports optional parameters and named arguments; otherwise callers will be forced to write ugly code to specify values for all the optional parameters. Obviously there’s more to immutability than getting values to the initialization code, but this is a welcome step in the right direction nonetheless. There are a couple of final points to make about these features before we move on to COM—points concerning the details of how the compiler handles your code and the difficulty of good API design. OVERLOAD RESOLUTION Clearly both named arguments and optional parameters affect how the compiler resolves overloads—if there are multiple method signatures available with the same name, which one should it pick? Optional parameters can increase the number of applicable methods (if some methods have more parameters than the number of specified arguments) and named arguments can decrease the number of applicable methods (by ruling out methods that don’t have the appropriate parameter names). For the most part, the changes are intuitive: to check whether any particular method is applicable, the compiler tries to build a list of the arguments it would pass in, using the positional arguments in order, and then matching the named arguments up with the remaining parameters. If a required parameter hasn’t been specified or if a named argument doesn’t match any remaining parameters, the method isn’t appli- cable. The specification gives more detail about this in section 7.5.3, but there are two situations I’d like to draw particular attention to. Listing 13.7 Using optional parameters and named arguments for immutability Normal initialization code goes here 385Optional parameters and named arguments First, if two methods are both applicable and one of them has been given all of its arguments explicitly, whereas the other uses an optional parameter filled in with a default value, the method that doesn’t use any default values will win. But this doesn’t extend to just comparing the number of default values used—it’s a strict “does it use default values or not” divide. For example, consider the following: static void Foo(int x = 10) {} static void Foo(int x = 10, int y = 20) {} ... Foo(); Foo(1); Foo(y: 2); Foo(1, 2); In the first call B, both methods are applicable because of their optional parameters. But the compiler can’t work out which one you meant to call; it’ll raise an error. In the second call C, both methods are still applicable, but the first overload is used because it can be applied without using any default values, whereas the second overload uses the default value for y. For both the third and fourth calls, only the second overload is applicable. The third call D names the y argument, and the fourth call E has two arguments; both of these mean the first overload isn’t applicable. OVERLOADS AND INHERITANCE DON’T ALWAYS MIX NICELY All of this assumes that the compiler has gone as far as finding multiple overloads to choose between. If some methods are declared in a base type, but there are applica- ble methods in a more derived type, the latter will win. This has always been the case, and it can cause some surprising results (see the book’s website for more details and examples: http://mng.bz/aEmE), but now optional param- eters mean there may be more applicable methods than you’d expect. I advise you to avoid overloading a base class method within a derived class unless you get a huge benefit. The second point is that sometimes named arguments can be an alternative to casting in order to help the compiler resolve overloads. Sometimes a call can be ambiguous because the arguments can be converted to the parameter types in two different meth- ods, but neither method is better than the other in all respects. For instance, consider the following method signatures and call: void Method(int x, object y) {... } void Method(object a, int b) {... } ... Method(10, 10); Both methods are applicable, and neither is better than the other. There are two ways to resolve this, assuming you can’t change the method names to make them unam- biguous that way. (That’s my preferred approach. Make each method name more informative and specific, which will improve the general readability of the code.) You Error: ambiguousB Calls first overloadC Calls second overloadD Calls second overloadE Ambiguous call 386 CHAPTER 13 Minor changes to simplify code can either cast one of the arguments explicitly, or use named arguments to resolve the ambiguity: void Method(int x, object y) {... } void Method(object a, int b) {... } ... Method(10, (object) 10); Method(x: 10, y: 10); Of course, this only works if the parameters have different names in the different methods, but it’s a handy trick to know. Sometimes the cast will give more readable code; sometimes the name will. It’s just an extra weapon in the fight for clear code. Unfortunately, it does have a downside, along with named arguments in general: it’s another thing to be careful about when you change parameter names. THE SILENT HORROR OF CHANGING NAMES In the past, parameter names haven’t mattered much if you’ve only been using C#. Other languages may have cared, but in C# the only times that parameter names were important were when you were looking at IntelliSense and when you were looking at the method code itself. Now the parameter names of a method are effectively part of the API even if you’re only using C#. If you change them at a later date, code can break—anything that was using a named argument to refer to one of your parameters will fail to compile if you decide to change it. This may not be much of an issue if your code is only consumed by itself, but if you’re writing a public API, be aware that chang- ing a parameter name is a big deal. It always has been, really, but if everything calling the code was written in C#, you’ve been able to ignore that until now. Renaming parameters is bad; switching the names around is worse. The calling code may still compile, but with a different meaning. A particularly evil form of this is to override a method and switch the parameter names in the overridden version. The compiler will always look at the deepest override it knows about, based on the static type of the expression used as the target of the method call. You don’t want to get into a situation where calling the same method implementation with the same argument list results in different behavior based on the static type of a variable. IN CONCLUSION… Named arguments and optional parameters are possibly two of the simplest-sounding features of C# 4, and yet they have a fair amount of complexity, as you’ve seen. The basic ideas are easily expressed and understood, and the good news is that most of the time that’s all you need to care about. You can take advantage of optional parameters to reduce the number of overloads you write, and named arguments can make code much more readable when several easily confusable arguments are used. The trickiest bit is probably deciding which default values to use, bearing in mind potential versioning issues. Likewise, it’s now more obvious than before that parame- ter names matter, and you need to be careful when overriding existing methods, to avoid being evil to your callers. Casting to resolve ambiguity Naming to resolve ambiguity 387Improvements for COM interoperability Speaking of evil, let’s move on to the new features relating to COM. I’m only kid- ding…mostly, anyway. 13.2 Improvements for COM interoperability I readily admit to being far from a COM expert. When I tried to use it before .NET came along, I always ran into issues that were no doubt partially caused by my lack of knowledge and partially caused by the components I was working with being poorly designed or implemented. My overall impression of COM as a sort of black magic has lingered, though. I’ve been reliably informed that there’s a lot to like about it, but unfortunately I haven’t found myself going back to learn it in detail—and there seems to be a lot of detail to study. THIS SECTION IS MICROSOFT-SPECIFIC The changes for COM interoperability won’t make sense for all C# compilers, and a compiler can still be deemed compliant with the specification without implementing these features. .NET has made COM somewhat friendlier in general, but until now there have been distinct advantages to using it from Visual Basic instead of C#. The playing field has been leveled significantly by C# 4, as you’ll see in this section. For the sake of familiar- it