data structures and algorithm analysis

Data Structures and Algorithm Analysis Edition 3.2 (C++ Version) Clifford A. Shaffer Department of Computer Science Virginia Tech Blacksburg, VA 24061 January 2, 2012 Update For a list of changes, see˜shaffer/Book/errata.html Copyright © 2009-2012 by Clifford A. Shaffer. This document is made freely available in PDF form for educational and other non-commercial use. You may make copies of this file and redistribute in electronic form without charge. You may extract portions of this document provided that the front page, including the title, author, and this notice are included. Any commercial use of this document requires the written consent of the author. The author can be reached at If you wish to have a printed version of this document, print copies are published by Dover Publications (see Further information about this text is available at˜shaffer/Book/. Contents Preface xiii I Preliminaries 1 1 Data Structures and Algorithms 3 1.1 A Philosophy of Data Structures 4 1.1.1 The Need for Data Structures 4 1.1.2 Costs and Benefits 6 1.2 Abstract Data Types and Data Structures 8 1.3 Design Patterns 12 1.3.1 Flyweight 13 1.3.2 Visitor 13 1.3.3 Composite 14 1.3.4 Strategy 15 1.4 Problems, Algorithms, and Programs 16 1.5 Further Reading 18 1.6 Exercises 20 2 Mathematical Preliminaries 25 2.1 Sets and Relations 25 2.2 Miscellaneous Notation 29 2.3 Logarithms 31 2.4 Summations and Recurrences 32 2.5 Recursion 36 2.6 Mathematical Proof Techniques 38 iii iv Contents 2.6.1 Direct Proof 39 2.6.2 Proof by Contradiction 39 2.6.3 Proof by Mathematical Induction 40 2.7 Estimation 46 2.8 Further Reading 47 2.9 Exercises 48 3 Algorithm Analysis 55 3.1 Introduction 55 3.2 Best, Worst, and Average Cases 61 3.3 A Faster Computer, or a Faster Algorithm? 62 3.4 Asymptotic Analysis 65 3.4.1 Upper Bounds 65 3.4.2 Lower Bounds 67 3.4.3 ⇥ Notation 68 3.4.4 Simplifying Rules 69 3.4.5 Classifying Functions 70 3.5 Calculating the Running Time for a Program 71 3.6 Analyzing Problems 76 3.7 Common Misunderstandings 77 3.8 Multiple Parameters 79 3.9 Space Bounds 80 3.10 Speeding Up Your Programs 82 3.11 Empirical Analysis 85 3.12 Further Reading 86 3.13 Exercises 86 3.14 Projects 90 II Fundamental Data Structures 93 4 Lists, Stacks, and Queues 95 4.1 Lists 96 4.1.1 Array-Based List Implementation 100 4.1.2 Linked Lists 103 4.1.3 Comparison of List Implementations 112 Contents v 4.1.4 Element Implementations 114 4.1.5 Doubly Linked Lists 115 4.2 Stacks 120 4.2.1 Array-Based Stacks 121 4.2.2 Linked Stacks 124 4.2.3 Comparison of Array-Based and Linked Stacks 125 4.2.4 Implementing Recursion 125 4.3 Queues 129 4.3.1 Array-Based Queues 129 4.3.2 Linked Queues 134 4.3.3 Comparison of Array-Based and Linked Queues 134 4.4 Dictionaries 134 4.5 Further Reading 145 4.6 Exercises 145 4.7 Projects 149 5 Binary Trees 151 5.1 Definitions and Properties 151 5.1.1 The Full Binary Tree Theorem 153 5.1.2 A Binary Tree Node ADT 155 5.2 Binary Tree Traversals 155 5.3 Binary Tree Node Implementations 160 5.3.1 Pointer-Based Node Implementations 160 5.3.2 Space Requirements 166 5.3.3 Array Implementation for Complete Binary Trees 168 5.4 Binary Search Trees 168 5.5 Heaps and Priority Queues 178 5.6 Huffman Coding Trees 185 5.6.1 Building Huffman Coding Trees 186 5.6.2 Assigning and Using Huffman Codes 192 5.6.3 Search in Huffman Trees 195 5.7 Further Reading 196 5.8 Exercises 196 5.9 Projects 200 6 Non-Binary Trees 203 vi Contents 6.1 General Tree Definitions and Terminology 203 6.1.1 An ADT for General Tree Nodes 204 6.1.2 General Tree Traversals 205 6.2 The Parent Pointer Implementation 207 6.3 General Tree Implementations 213 6.3.1 List of Children 214 6.3.2 The Left-Child/Right-Sibling Implementation 215 6.3.3 Dynamic Node Implementations 215 6.3.4 Dynamic “Left-Child/Right-Sibling” Implementation 218 6.4 K-ary Trees 218 6.5 Sequential Tree Implementations 219 6.6 Further Reading 223 6.7 Exercises 223 6.8 Projects 226 III Sorting and Searching 229 7 Internal Sorting 231 7.1 Sorting Terminology and Notation 232 7.2 Three ⇥(n2) Sorting Algorithms 233 7.2.1 Insertion Sort 233 7.2.2 Bubble Sort 235 7.2.3 Selection Sort 237 7.2.4 The Cost of Exchange Sorting 238 7.3 Shellsort 239 7.4 Mergesort 241 7.5 Quicksort 244 7.6 Heapsort 251 7.7 Binsort and Radix Sort 252 7.8 An Empirical Comparison of Sorting Algorithms 259 7.9 Lower Bounds for Sorting 261 7.10 Further Reading 265 7.11 Exercises 265 7.12 Projects 269 Contents vii 8 File Processing and External Sorting 273 8.1 Primary versus Secondary Storage 273 8.2 Disk Drives 276 8.2.1 Disk Drive Architecture 276 8.2.2 Disk Access Costs 280 8.3 Buffers and Buffer Pools 282 8.4 The Programmer’s View of Files 290 8.5 External Sorting 291 8.5.1 Simple Approaches to External Sorting 294 8.5.2 Replacement Selection 296 8.5.3 Multiway Merging 300 8.6 Further Reading 303 8.7 Exercises 304 8.8 Projects 307 9 Searching 311 9.1 Searching Unsorted and Sorted Arrays 312 9.2 Self-Organizing Lists 317 9.3 Bit Vectors for Representing Sets 323 9.4 Hashing 324 9.4.1 Hash Functions 325 9.4.2 Open Hashing 330 9.4.3 Closed Hashing 331 9.4.4 Analysis of Closed Hashing 339 9.4.5 Deletion 344 9.5 Further Reading 345 9.6 Exercises 345 9.7 Projects 348 10 Indexing 351 10.1 Linear Indexing 353 10.2 ISAM 356 10.3 Tree-based Indexing 358 10.4 2-3 Trees 360 10.5 B-Trees 364 10.5.1 B+-Trees 368 viii Contents 10.5.2 B-Tree Analysis 374 10.6 Further Reading 375 10.7 Exercises 375 10.8 Projects 377 IV Advanced Data Structures 379 11 Graphs 381 11.1 Terminology and Representations 382 11.2 Graph Implementations 386 11.3 Graph Traversals 390 11.3.1 Depth-First Search 393 11.3.2 Breadth-First Search 394 11.3.3 Topological Sort 394 11.4 Shortest-Paths Problems 399 11.4.1 Single-Source Shortest Paths 400 11.5 Minimum-Cost Spanning Trees 402 11.5.1 Prim’s Algorithm 404 11.5.2 Kruskal’s Algorithm 407 11.6 Further Reading 409 11.7 Exercises 409 11.8 Projects 411 12 Lists and Arrays Revisited 413 12.1 Multilists 413 12.2 Matrix Representations 416 12.3 Memory Management 420 12.3.1 Dynamic Storage Allocation 422 12.3.2 Failure Policies and Garbage Collection 429 12.4 Further Reading 433 12.5 Exercises 434 12.6 Projects 435 13 Advanced Tree Structures 437 13.1 Tries 437 Contents ix 13.2 Balanced Trees 442 13.2.1 The AVL Tree 443 13.2.2 The Splay Tree 445 13.3 Spatial Data Structures 448 13.3.1 The K-D Tree 450 13.3.2 The PR quadtree 455 13.3.3 Other Point Data Structures 459 13.3.4 Other Spatial Data Structures 461 13.4 Further Reading 461 13.5 Exercises 462 13.6 Projects 463 V Theory of Algorithms 467 14 Analysis Techniques 469 14.1 Summation Techniques 470 14.2 Recurrence Relations 475 14.2.1 Estimating Upper and Lower Bounds 475 14.2.2 Expanding Recurrences 478 14.2.3 Divide and Conquer Recurrences 480 14.2.4 Average-Case Analysis of Quicksort 482 14.3 Amortized Analysis 484 14.4 Further Reading 487 14.5 Exercises 487 14.6 Projects 491 15 Lower Bounds 493 15.1 Introduction to Lower Bounds Proofs 494 15.2 Lower Bounds on Searching Lists 496 15.2.1 Searching in Unsorted Lists 496 15.2.2 Searching in Sorted Lists 498 15.3 Finding the Maximum Value 499 15.4 Adversarial Lower Bounds Proofs 501 15.5 State Space Lower Bounds Proofs 504 15.6 Finding the ith Best Element 507 x Contents 15.7 Optimal Sorting 509 15.8 Further Reading 512 15.9 Exercises 512 15.10Projects 515 16 Patterns of Algorithms 517 16.1 Dynamic Programming 517 16.1.1 The Knapsack Problem 519 16.1.2 All-Pairs Shortest Paths 521 16.2 Randomized Algorithms 523 16.2.1 Randomized algorithms for finding large values 523 16.2.2 Skip Lists 524 16.3 Numerical Algorithms 530 16.3.1 Exponentiation 531 16.3.2 Largest Common Factor 531 16.3.3 Matrix Multiplication 532 16.3.4 Random Numbers 534 16.3.5 The Fast Fourier Transform 535 16.4 Further Reading 540 16.5 Exercises 540 16.6 Projects 541 17 Limits to Computation 543 17.1 Reductions 544 17.2 Hard Problems 549 17.2.1 The Theory of NP-Completeness 551 17.2.2 NP-Completeness Proofs 555 17.2.3 Coping with NP-Complete Problems 560 17.3 Impossible Problems 563 17.3.1 Uncountability 564 17.3.2 The Halting Problem Is Unsolvable 567 17.4 Further Reading 569 17.5 Exercises 570 17.6 Projects 572 Contents xi VI APPENDIX 575 A Utility Functions 577 Bibliography 579 Index 585 Preface We study data structures so that we can learn to write more efficient programs. But why must programs be efficient when new computers are faster every year? The reason is that our ambitions grow with our capabilities. Instead of rendering efficiency needs obsolete, the modern revolution in computing power and storage capability merely raises the efficiency stakes as we attempt more complex tasks. The quest for program efficiency need not and should not conflict with sound design and clear coding. Creating efficient programs has little to do with “program- ming tricks” but rather is based on good organization of information and good al- gorithms. A programmer who has not mastered the basic principles of clear design is not likely to write efficient programs. Conversely, concerns related to develop- ment costs and maintainability should not be used as an excuse to justify inefficient performance. Generality in design can and should be achieved without sacrificing performance, but this can only be done if the designer understands how to measure performance and does so as an integral part of the design and implementation pro- cess. Most computer science curricula recognize that good programming skills be- gin with a strong emphasis on fundamental software engineering principles. Then, once a programmer has learned the principles of clear program design and imple- mentation, the next step is to study the effects of data organization and algorithms on program efficiency. Approach: This book describes many techniques for representing data. These techniques are presented within the context of the following principles: 1. Each data structure and each algorithm has costs and benefits. Practitioners need a thorough understanding of how to assess costs and benefits to be able to adapt to new design challenges. This requires an understanding of the principles of algorithm analysis, and also an appreciation for the significant effects of the physical medium employed (e.g., data stored on disk versus main memory). 2. Related to costs and benefits is the notion of tradeoffs. For example, it is quite common to reduce time requirements at the expense of an increase in space requirements, or vice versa. Programmers face tradeoff issues regularly in all xiii xiv Preface phases of software design and implementation, so the concept must become deeply ingrained. 3. Programmers should know enough about common practice to avoid rein- venting the wheel. Thus, programmers need to learn the commonly used data structures, their related algorithms, and the most frequently encountered design patterns found in programming. 4. Data structures follow needs. Programmers must learn to assess application needs first, then find a data structure with matching capabilities. To do this requires competence in Principles 1, 2, and 3. As I have taught data structures through the years, I have found that design issues have played an ever greater role in my courses. This can be traced through the various editions of this textbook by the increasing coverage for design patterns and generic interfaces. The first edition had no mention of design patterns. The second edition had limited coverage of a few example patterns, and introduced the dictionary ADT and comparator classes. With the third edition, there is explicit coverage of some design patterns that are encountered when programming the basic data structures and algorithms covered in the book. Using the Book in Class: Data structures and algorithms textbooks tend to fall into one of two categories: teaching texts or encyclopedias. Books that attempt to do both usually fail at both. This book is intended as a teaching text. I believe it is more important for a practitioner to understand the principles required to select or design the data structure that will best solve some problem than it is to memorize a lot of textbook implementations. Hence, I have designed this as a teaching text that covers most standard data structures, but not all. A few data structures that are not widely adopted are included to illustrate important principles. Some relatively new data structures that should become widely used in the future are included. Within an undergraduate program, this textbook is designed for use in either an advanced lower division (sophomore or junior level) data structures course, or for a senior level algorithms course. New material has been added in the third edition to support its use in an algorithms course. Normally, this text would be used in a course beyond the standard freshman level “CS2” course that often serves as the initial introduction to data structures. Readers of this book should typically have two semesters of the equivalent of programming experience, including at least some exposure to C++. Readers who are already familiar with recursion will have an advantage. Students of data structures will also benefit from having first completed a good course in Discrete Mathematics. Nonetheless, Chapter 2 attempts to give a reasonably complete survey of the prerequisite mathematical topics at the level necessary to understand their use in this book. Readers may wish to refer back to the appropriate sections as needed when encountering unfamiliar mathematical material. Preface xv A sophomore-level class where students have only a little background in basic data structures or analysis (that is, background equivalent to what would be had from a traditional CS2 course) might cover Chapters 1-11 in detail, as well as se- lected topics from Chapter 13. That is how I use the book for my own sophomore- level class. Students with greater background might cover Chapter 1, skip most of Chapter 2 except for reference, briefly cover Chapters 3 and 4, and then cover chapters 5-12 in detail. Again, only certain topics from Chapter 13 might be cov- ered, depending on the programming assignments selected by the instructor. A senior-level algorithms course would focus on Chapters 11 and 14-17. Chapter 13 is intended in part as a source for larger programming exercises. I recommend that all students taking a data structures course be required to im- plement some advanced tree structure, or another dynamic structure of comparable difficulty such as the skip list or sparse matrix representations of Chapter 12. None of these data structures are significantly more difficult to implement than the binary search tree, and any of them should be within a student’s ability after completing Chapter 5. While I have attempted to arrange the presentation in an order that makes sense, instructors should feel free to rearrange the topics as they see fit. The book has been written so that once the reader has mastered Chapters 1-6, the remaining material has relatively few dependencies. Clearly, external sorting depends on understand- ing internal sorting and disk files. Section 6.2 on the UNION/FIND algorithm is used in Kruskal’s Minimum-Cost Spanning Tree algorithm. Section 9.2 on self- organizing lists mentions the buffer replacement schemes covered in Section 8.3. Chapter 14 draws on examples from throughout the book. Section 17.2 relies on knowledge of graphs. Otherwise, most topics depend only on material presented earlier within the same chapter. Most chapters end with a section entitled “Further Reading.” These sections are not comprehensive lists of references on the topics presented. Rather, I include books and articles that, in my opinion, may prove exceptionally informative or entertaining to the reader. In some cases I include references to works that should become familiar to any well-rounded computer scientist. Use of C++: The programming examples are written in C++, but I do not wish to discourage those unfamiliar with C++ from reading this book. I have attempted to make the examples as clear as possible while maintaining the advantages of C++. C++ is used here strictly as a tool to illustrate data structures concepts. In particu- lar, I make use of C++’s support for hiding implementation details, including fea- tures such as classes, private class members, constructors, and destructors. These features of the language support the crucial concept of separating logical design, as embodied in the abstract data type, from physical implementation as embodied in the data structure. xvi Preface To keep the presentation as clear as possible, some important features of C++ are avoided here. I deliberately minimize use of certain features commonly used by experienced C++ programmers such as class hierarchy, inheritance, and virtual functions. Operator and function overloading is used sparingly. C-like initialization syntax is preferred to some of the alternatives offered by C++. While the C++ features mentioned above have valid design rationale in real programs, they tend to obscure rather than enlighten the principles espoused in this book. For example, inheritance is an important tool that helps programmers avoid duplication, and thus minimize bugs. From a pedagogical standpoint, how- ever, inheritance often makes code examples harder to understand since it tends to spread the description for one logical unit among several classes. Thus, my class definitions only use inheritance where inheritance is explicitly relevant to the point illustrated (e.g., Section 5.3.1). This does not mean that a programmer should do likewise. Avoiding code duplication and minimizing errors are important goals. Treat the programming examples as illustrations of data structure principles, but do not copy them directly into your own programs. One painful decision I had to make was whether to use templates in the code examples. In the first edition of this book, the decision was to leave templates out as it was felt that their syntax obscures the meaning of the code for those not famil- iar with C++. In the years following, the use of C++ in computer science curricula has greatly expanded. I now assume that readers of the text will be familiar with template syntax. Thus, templates are now used extensively in the code examples. My implementations are meant to provide concrete illustrations of data struc- ture principles, as an aid to the textual exposition. Code examples should not be read or used in isolation from the associated text because the bulk of each exam- ple’s documentation is contained in the text, not the code. The code complements the text, not the other way around. They are not meant to be a series of commercial- quality class implementations. If you are looking for a complete implementation of a standard data structure for use in your own code, you would do well to do an Internet search. For instance, the code examples provide less parameter checking than is sound programming practice, since including such checking would obscure rather than il- luminate the text. Some parameter checking and testing for other constraints (e.g., whether a value is being removed from an empty container) is included in the form of a call to Assert. The inputs to Assert are a Boolean expression and a charac- ter string. If this expression evaluates to false, then a message is printed and the program terminates immediately. Terminating a program when a function receives a bad parameter is generally considered undesirable in real programs, but is quite adequate for understanding how a data structure is meant to operate. In real pro- gramming applications, C++’s exception handling features should be used to deal with input data errors. However, assertions provide a simpler mechanism for indi- Preface xvii cating required conditions in a way that is both adequate for clarifying how a data structure is meant to operate, and is easily modified into true exception handling. See the Appendix for the implementation of Assert. I make a distinction in the text between “C++ implementations” and “pseu- docode.” Code labeled as a C++ implementation has actually been compiled and tested on one or more C++ compilers. Pseudocode examples often conform closely to C++ syntax, but typically contain one or more lines of higher-level description. Pseudocode is used where I perceived a greater pedagogical advantage to a simpler, but less precise, description. Exercises and Projects: Proper implementation and analysis of data structures cannot be learned simply by reading a book. You must practice by implementing real programs, constantly comparing different techniques to see what really works best in a given situation. One of the most important aspects of a course in data structures is that it is where students really learn to program using pointers and dynamic memory al- location, by implementing data structures such as linked lists and trees. It is often where students truly learn recursion. In our curriculum, this is the first course where students do significant design, because it often requires real data structures to mo- tivate significant design exercises. Finally, the fundamental differences between memory-based and disk-based data access cannot be appreciated without practical programming experience. For all of these reasons, a data structures course cannot succeed without a significant programming component. In our department, the data structures course is one of the most difficult programming course in the curriculum. Students should also work problems to develop their analytical abilities. I pro- vide over 450 exercises and suggestions for programming projects. I urge readers to take advantage of them. Contacting the Author and Supplementary Materials: A book such as this is sure to contain errors and have room for improvement. I welcome bug reports and constructive criticism. I can be reached by electronic mail via the Internet at Alternatively, comments can be mailed to Cliff Shaffer Department of Computer Science Virginia Tech Blacksburg, VA 24061 The electronic posting of this book, along with a set of lecture notes for use in class can be obtained at˜shaffer/book.html. The code examples used in the book are available at the same site. Online Web pages for Virginia Tech’s sophomore-level data structures class can be found at xviii Preface˜cs3114. This book was typeset by the author using LATEX. The bibliography was pre- pared using BIBTEX. The index was prepared using makeindex. The figures were mostly drawn with Xfig. Figures 3.1 and 9.10 were partially created using Math- ematica. Acknowledgments: It takes a lot of help from a lot of people to make a book. I wish to acknowledge a few of those who helped to make this book possible. I apologize for the inevitable omissions. Virginia Tech helped make this whole thing possible through sabbatical re- search leave during Fall 1994, enabling me to get the project off the ground. My de- partment heads during the time I have written the various editions of this book, Den- nis Kafura and Jack Carroll, provided unwavering moral support for this project. Mike Keenan, Lenny Heath, and Jeff Shaffer provided valuable input on early ver- sions of the chapters. I also wish to thank Lenny Heath for many years of stimulat- ing discussions about algorithms and analysis (and how to teach both to students). Steve Edwards deserves special thanks for spending so much time helping me on various redesigns of the C++ and Java code versions for the second and third edi- tions, and many hours of discussion on the principles of program design. Thanks to Layne Watson for his help with Mathematica, and to Bo Begole, Philip Isenhour, Jeff Nielsen, and Craig Struble for much technical assistance. Thanks to Bill Mc- Quain, Mark Abrams and Dennis Kafura for answering lots of silly questions about C++ and Java. I am truly indebted to the many reviewers of the various editions of this manu- script. For the first edition these reviewers included J. David Bezek (University of Evansville), Douglas Campbell (Brigham Young University), Karen Davis (Univer- sity of Cincinnati), Vijay Kumar Garg (University of Texas – Austin), Jim Miller (University of Kansas), Bruce Maxim (University of Michigan – Dearborn), Jeff Parker (Agile Networks/Harvard), Dana Richards (George Mason University), Jack Tan (University of Houston), and Lixin Tao (Concordia University). Without their help, this book would contain many more technical errors and many fewer insights. For the second edition, I wish to thank these reviewers: Gurdip Singh (Kansas State University), Peter Allen (Columbia University), Robin Hill (University of Wyoming), Norman Jacobson (University of California – Irvine), Ben Keller (East- ern Michigan University), and Ken Bosworth (Idaho State University). In addition, I wish to thank Neil Stewart and Frank J. Thesen for their comments and ideas for improvement. Third edition reviewers included Randall Lechlitner (University of Houstin, Clear Lake) and Brian C. Hipp (York Technical College). I thank them for their comments. Preface xix Prentice Hall was the original print publisher for the first and second editions. Without the hard work of many people there, none of this would be possible. Au- thors simply do not create printer-ready books on their own. Foremost thanks go to Kate Hargett, Petra Rector, Laura Steele, and Alan Apt, my editors over the years. My production editors, Irwin Zucker for the second edition, Kathleen Caren for the original C++ version, and Ed DeFelippis for the Java version, kept everything moving smoothly during that horrible rush at the end. Thanks to Bill Zobrist and Bruce Gregory (I think) for getting me into this in the first place. Others at Prentice Hall who helped me along the way include Truly Donovan, Linda Behrens, and Phyllis Bregman. Thanks to Tracy Dunkelberger for her help in returning the copy- right to me, thus enabling the electronic future of this work. I am sure I owe thanks to many others at Prentice Hall for their help in ways that I am not even aware of. I am thankful to Shelley Kronzek at Dover publications for her faith in taking on the print publication of this third edition. Much expanded, with both Java and C++ versions, and many inconsistencies corrected, I am confident that this is the best edition yet. But none of us really knows whether students will prefer a free online textbook or a low-cost, printed bound version. In the end, we believe that the two formats will be mutually supporting by offering more choices. Production editor James Miller and design manager Marie Zaczkiewicz have worked hard to ensure that the production is of the highest quality. I wish to express my appreciation to Hanan Samet for teaching me about data structures. I learned much of the philosophy presented here from him as well, though he is not responsible for any problems with the result. Thanks to my wife Terry, for her love and support, and to my daughters Irena and Kate for pleasant diversions from working too hard. Finally, and most importantly, to all of the data structures students over the years who have taught me what is important and what should be skipped in a data structures course, and the many new insights they have provided. This book is dedicated to them. Cliff Shaffer Blacksburg, Virginia PART I Preliminaries 1 1 Data Structures and Algorithms How many cities with more than 250,000 people lie within 500 miles of Dallas, Texas? How many people in my company make over $100,000 per year? Can we connect all of our telephone customers with less than 1,000 miles of cable? To answer questions like these, it is not enough to have the necessary information. We must organize that information in a way that allows us to find the answers in time to satisfy our needs. Representing information is fundamental to computer science. The primary purpose of most computer programs is not to perform calculations, but to store and retrieve information — usually as fast as possible. For this reason, the study of data structures and the algorithms that manipulate them is at the heart of computer science. And that is what this book is about — helping you to understand how to structure information to support efficient processing. This book has three primary goals. The first is to present the commonly used data structures. These form a programmer’s basic data structure “toolkit.” For many problems, some data structure in the toolkit provides a good solution. The second goal is to introduce the idea of tradeoffs and reinforce the concept that there are costs and benefits associated with every data structure. This is done by describing, for each data structure, the amount of space and time required for typical operations. The third goal is to teach how to measure the effectiveness of a data structure or algorithm. Only through such measurement can you determine which data structure in your toolkit is most appropriate for a new problem. The techniques presented also allow you to judge the merits of new data structures that you or others might invent. There are often many approaches to solving a problem. How do we choose between them? At the heart of computer program design are two (sometimes con- flicting) goals: 1. To design an algorithm that is easy to understand, code, and debug. 2. To design an algorithm that makes efficient use of the computer’s resources. 3 4 Chap. 1 Data Structures and Algorithms Ideally, the resulting program is true to both of these goals. We might say that such a program is “elegant.” While the algorithms and program code examples pre- sented here attempt to be elegant in this sense, it is not the purpose of this book to explicitly treat issues related to goal (1). These are primarily concerns of the disci- pline of Software Engineering. Rather, this book is mostly about issues relating to goal (2). How do we measure efficiency? Chapter 3 describes a method for evaluating the efficiency of an algorithm or computer program, called asymptotic analysis. Asymptotic analysis also allows you to measure the inherent difficulty of a problem. The remaining chapters use asymptotic analysis techniques to estimate the time cost for every algorithm presented. This allows you to see how each algorithm compares to other algorithms for solving the same problem in terms of its efficiency. This first chapter sets the stage for what is to follow, by presenting some higher- order issues related to the selection and use of data structures. We first examine the process by which a designer selects a data structure appropriate to the task at hand. We then consider the role of abstraction in program design. We briefly consider the concept of a design pattern and see some examples. The chapter ends with an exploration of the relationship between problems, algorithms, and programs. 1.1 A Philosophy of Data Structures 1.1.1 The Need for Data Structures You might think that with ever more powerful computers, program efficiency is becoming less important. After all, processor speed and memory size still con- tinue to improve. Won’t any efficiency problem we might have today be solved by tomorrow’s hardware? As we develop more powerful computers, our history so far has always been to use that additional computing power to tackle more complex problems, be it in the form of more sophisticated user interfaces, bigger problem sizes, or new problems previously deemed computationally infeasible. More complex problems demand more computation, making the need for efficient programs even greater. Worse yet, as tasks become more complex, they become less like our everyday experience. Today’s computer scientists must be trained to have a thorough understanding of the principles behind efficient program design, because their ordinary life experiences often do not apply when designing computer programs. In the most general sense, a data structure is any data representation and its associated operations. Even an integer or floating point number stored on the com- puter can be viewed as a simple data structure. More commonly, people use the term “data structure” to mean an organization or structuring for a collection of data items. A sorted list of integers stored in an array is an example of such a structuring. Sec. 1.1 A Philosophy of Data Structures 5 Given sufficient space to store a collection of data items, it is always possible to search for specified items within the collection, print or otherwise process the data items in any desired order, or modify the value of any particular data item. Thus, it is possible to perform all necessary operations on any data structure. However, using the proper data structure can make the difference between a program running in a few seconds and one requiring many days. A solution is said to be efficient if it solves the problem within the required resource constraints. Examples of resource constraints include the total space available to store the data — possibly divided into separate main memory and disk space constraints — and the time allowed to perform each subtask. A solution is sometimes said to be efficient if it requires fewer resources than known alternatives, regardless of whether it meets any particular requirements. The cost of a solution is the amount of resources that the solution consumes. Most often, cost is measured in terms of one key resource such as time, with the implied assumption that the solution meets the other resource constraints. It should go without saying that people write programs to solve problems. How- ever, it is crucial to keep this truism in mind when selecting a data structure to solve a particular problem. Only by first analyzing the problem to determine the perfor- mance goals that must be achieved can there be any hope of selecting the right data structure for the job. Poor program designers ignore this analysis step and apply a data structure that they are familiar with but which is inappropriate to the problem. The result is typically a slow program. Conversely, there is no sense in adopting a complex representation to “improve” a program that can meet its performance goals when implemented using a simpler design. When selecting a data structure to solve a problem, you should follow these steps. 1. Analyze your problem to determine the basic operations that must be sup- ported. Examples of basic operations include inserting a data item into the data structure, deleting a data item from the data structure, and finding a specified data item. 2. Quantify the resource constraints for each operation. 3. Select the data structure that best meets these requirements. This three-step approach to selecting a data structure operationalizes a data- centered view of the design process. The first concern is for the data and the op- erations to be performed on them, the next concern is the representation for those data, and the final concern is the implementation of that representation. Resource constraints on certain key operations, such as search, inserting data records, and deleting data records, normally drive the data structure selection pro- cess. Many issues relating to the relative importance of these operations are ad- dressed by the following three questions, which you should ask yourself whenever you must choose a data structure: 6 Chap. 1 Data Structures and Algorithms • Are all data items inserted into the data structure at the beginning, or are insertions interspersed with other operations? Static applications (where the data are loaded at the beginning and never change) typically require only simpler data structures to get an efficient implementation than do dynamic applications. • Can data items be deleted? If so, this will probably make the implementation more complicated. • Are all data items processed in some well-defined order, or is search for spe- cific data items allowed? “Random access” search generally requires more complex data structures. 1.1.2 Costs and Benefits Each data structure has associated costs and benefits. In practice, it is hardly ever true that one data structure is better than another for use in all situations. If one data structure or algorithm is superior to another in all respects, the inferior one will usually have long been forgotten. For nearly every data structure and algorithm presented in this book, you will see examples of where it is the best choice. Some of the examples might surprise you. A data structure requires a certain amount of space for each data item it stores, a certain amount of time to perform a single basic operation, and a certain amount of programming effort. Each problem has constraints on available space and time. Each solution to a problem makes use of the basic operations in some relative pro- portion, and the data structure selection process must account for this. Only after a careful analysis of your problem’s characteristics can you determine the best data structure for the task. Example 1.1 A bank must support many types of transactions with its customers, but we will examine a simple model where customers wish to open accounts, close accounts, and add money or withdraw money from accounts. We can consider this problem at two distinct levels: (1) the re- quirements for the physical infrastructure and workflow process that the bank uses in its interactions with its customers, and (2) the requirements for the database system that manages the accounts. The typical customer opens and closes accounts far less often than he or she accesses the account. Customers are willing to wait many minutes while accounts are created or deleted but are typically not willing to wait more than a brief time for individual account transactions such as a deposit or withdrawal. These observations can be considered as informal specifica- tions for the time constraints on the problem. It is common practice for banks to provide two tiers of service. Hu- man tellers or automated teller machines (ATMs) support customer access Sec. 1.1 A Philosophy of Data Structures 7 to account balances and updates such as deposits and withdrawals. Spe- cial service representatives are typically provided (during restricted hours) to handle opening and closing accounts. Teller and ATM transactions are expected to take little time. Opening or closing an account can take much longer (perhaps up to an hour from the customer’s perspective). From a database perspective, we see that ATM transactions do not mod- ify the database significantly. For simplicity, assume that if money is added or removed, this transaction simply changes the value stored in an account record. Adding a new account to the database is allowed to take several minutes. Deleting an account need have no time constraint, because from the customer’s point of view all that matters is that all the money be re- turned (equivalent to a withdrawal). From the bank’s point of view, the account record might be removed from the database system after business hours, or at the end of the monthly account cycle. When considering the choice of data structure to use in the database system that manages customer accounts, we see that a data structure that has little concern for the cost of deletion, but is highly efficient for search and moderately efficient for insertion, should meet the resource constraints imposed by this problem. Records are accessible by unique account number (sometimes called an exact-match query). One data structure that meets these requirements is the hash table described in Chapter 9.4. Hash tables allow for extremely fast exact-match search. A record can be modified quickly when the modification does not affect its space requirements. Hash tables also support efficient insertion of new records. While deletions can also be supported efficiently, too many deletions lead to some degradation in performance for the remaining operations. However, the hash table can be reorganized periodically to restore the system to peak efficiency. Such reorganization can occur offline so as not to affect ATM transactions. Example 1.2 A company is developing a database system containing in- formation about cities and towns in the United States. There are many thousands of cities and towns, and the database program should allow users to find information about a particular place by name (another example of an exact-match query). Users should also be able to find all places that match a particular value or range of values for attributes such as location or population size. This is known as a range query. A reasonable database system must answer queries quickly enough to satisfy the patience of a typical user. For an exact-match query, a few sec- onds is satisfactory. If the database is meant to support range queries that can return many cities that match the query specification, the entire opera- 8 Chap. 1 Data Structures and Algorithms tion may be allowed to take longer, perhaps on the order of a minute. To meet this requirement, it will be necessary to support operations that pro- cess range queries efficiently by processing all cities in the range as a batch, rather than as a series of operations on individual cities. The hash table suggested in the previous example is inappropriate for implementing our city database, because it cannot perform efficient range queries. The B+-tree of Section 10.5.1 supports large databases, insertion and deletion of data records, and range queries. However, a simple linear in- dex as described in Section 10.1 would be more appropriate if the database is created once, and then never changed, such as an atlas distributed on a CD or accessed from a website. 1.2 Abstract Data Types and Data Structures The previous section used the terms “data item” and “data structure” without prop- erly defining them. This section presents terminology and motivates the design process embodied in the three-step approach to selecting a data structure. This mo- tivation stems from the need to manage the tremendous complexity of computer programs. A type is a collection of values. For example, the Boolean type consists of the values true and false. The integers also form a type. An integer is a simple type because its values contain no subparts. A bank account record will typically contain several pieces of information such as name, address, account number, and account balance. Such a record is an example of an aggregate type or composite type.Adata item is a piece of information or a record whose value is drawn from a type. A data item is said to be a member of a type. A data type is a type together with a collection of operations to manipulate the type. For example, an integer variable is a member of the integer data type. Addition is an example of an operation on the integer data type. A distinction should be made between the logical concept of a data type and its physical implementation in a computer program. For example, there are two tra- ditional implementations for the list data type: the linked list and the array-based list. The list data type can therefore be implemented using a linked list or an ar- ray. Even the term “array” is ambiguous in that it can refer either to a data type or an implementation. “Array” is commonly used in computer programming to mean a contiguous block of memory locations, where each memory location stores one fixed-length data item. By this meaning, an array is a physical data structure. However, array can also mean a logical data type composed of a (typically ho- mogeneous) collection of data items, with each data item identified by an index number. It is possible to implement arrays in many different ways. For exam- Sec. 1.2 Abstract Data Types and Data Structures 9 ple, Section 12.2 describes the data structure used to implement a sparse matrix, a large two-dimensional array that stores only a relatively few non-zero values. This implementation is quite different from the physical representation of an array as contiguous memory locations. An abstract data type (ADT) is the realization of a data type as a software component. The interface of the ADT is defined in terms of a type and a set of operations on that type. The behavior of each operation is determined by its inputs and outputs. An ADT does not specify how the data type is implemented. These implementation details are hidden from the user of the ADT and protected from outside access, a concept referred to as encapsulation. A data structure is the implementation for an ADT. In an object-oriented lan- guage such as C++, an ADT and its implementation together make up a class. Each operation associated with the ADT is implemented by a member function or method. The variables that define the space required by a data item are referred to as data members.Anobject is an instance of a class, that is, something that is created and takes up storage during the execution of a computer program. The term “data structure” often refers to data stored in a computer’s main mem- ory. The related term file structure often refers to the organization of data on peripheral storage, such as a disk drive or CD. Example 1.3 The mathematical concept of an integer, along with oper- ations that manipulate integers, form a data type. The C++ int variable type is a physical representation of the abstract integer. The int variable type, along with the operations that act on an int variable, form an ADT. Unfortunately, the int implementation is not completely true to the ab- stract integer, as there are limitations on the range of values an int variable can store. If these limitations prove unacceptable, then some other repre- sentation for the ADT “integer” must be devised, and a new implementation must be used for the associated operations. Example 1.4 An ADT for a list of integers might specify the following operations: • Insert a new integer at a particular position in the list. • Return true if the list is empty. • Reinitialize the list. • Return the number of integers currently in the list. • Delete the integer at a particular position in the list. From this description, the input and output of each operation should be clear, but the implementation for lists has not been specified. 10 Chap. 1 Data Structures and Algorithms One application that makes use of some ADT might use particular member functions of that ADT more than a second application, or the two applications might have different time requirements for the various operations. These differences in the requirements of applications are the reason why a given ADT might be supported by more than one implementation. Example 1.5 Two popular implementations for large disk-based database applications are hashing (Section 9.4) and the B+-tree (Section 10.5). Both support efficient insertion and deletion of records, and both support exact- match queries. However, hashing is more efficient than the B+-tree for exact-match queries. On the other hand, the B+-tree can perform range queries efficiently, while hashing is hopelessly inefficient for range queries. Thus, if the database application limits searches to exact-match queries, hashing is preferred. On the other hand, if the application requires support for range queries, the B+-tree is preferred. Despite these performance is- sues, both implementations solve versions of the same problem: updating and searching a large collection of records. The concept of an ADT can help us to focus on key issues even in non-comp- uting applications. Example 1.6 When operating a car, the primary activities are steering, accelerating, and braking. On nearly all passenger cars, you steer by turn- ing the steering wheel, accelerate by pushing the gas pedal, and brake by pushing the brake pedal. This design for cars can be viewed as an ADT with operations “steer,” “accelerate,” and “brake.” Two cars might imple- ment these operations in radically different ways, say with different types of engine, or front- versus rear-wheel drive. Yet, most drivers can oper- ate many different cars because the ADT presents a uniform method of operation that does not require the driver to understand the specifics of any particular engine or drive design. These differences are deliberately hidden. The concept of an ADT is one instance of an important principle that must be understood by any successful computer scientist: managing complexity through abstraction. A central theme of computer science is complexity and techniques for handling it. Humans deal with complexity by assigning a label to an assembly of objects or concepts and then manipulating the label in place of the assembly. Cognitive psychologists call such a label a metaphor. A particular label might be related to other pieces of information or other labels. This collection can in turn be given a label, forming a hierarchy of concepts and labels. This hierarchy of labels allows us to focus on important issues while ignoring unnecessary details. Sec. 1.2 Abstract Data Types and Data Structures 11 Example 1.7 We apply the label “hard drive” to a collection of hardware that manipulates data on a particular type of storage device, and we ap- ply the label “CPU” to the hardware that controls execution of computer instructions. These and other labels are gathered together under the label “computer.” Because even the smallest home computers today have mil- lions of components, some form of abstraction is necessary to comprehend how a computer operates. Consider how you might go about the process of designing a complex computer program that implements and manipulates an ADT. The ADT is implemented in one part of the program by a particular data structure. While designing those parts of the program that use the ADT, you can think in terms of operations on the data type without concern for the data structure’s implementation. Without this ability to simplify your thinking about a complex program, you would have no hope of understanding or implementing it. Example 1.8 Consider the design for a relatively simple database system stored on disk. Typically, records on disk in such a program are accessed through a buffer pool (see Section 8.3) rather than directly. Variable length records might use a memory manager (see Section 12.3) to find an appro- priate location within the disk file to place the record. Multiple index struc- tures (see Chapter 10) will typically be used to access records in various ways. Thus, we have a chain of classes, each with its own responsibili- ties and access privileges. A database query from a user is implemented by searching an index structure. This index requests access to the record by means of a request to the buffer pool. If a record is being inserted or deleted, such a request goes through the memory manager, which in turn interacts with the buffer pool to gain access to the disk file. A program such as this is far too complex for nearly any human programmer to keep all of the details in his or her head at once. The only way to design and imple- ment such a program is through proper use of abstraction and metaphors. In object-oriented programming, such abstraction is handled using classes. Data types have both a logical and a physical form. The definition of the data type in terms of an ADT is its logical form. The implementation of the data type as a data structure is its physical form. Figure 1.1 illustrates this relationship between logical and physical forms for data types. When you implement an ADT, you are dealing with the physical form of the associated data type. When you use an ADT elsewhere in your program, you are concerned with the associated data type’s logical form. Some sections of this book focus on physical implementations for a 12 Chap. 1 Data Structures and Algorithms Data Type Data Structure: Storage Space Subroutines ADT: Type Operations Data Items: Data Items: Physical Form Logical Form Figure 1.1 The relationship between data items, abstract data types, and data structures. The ADT defines the logical form of the data type. The data structure implements the physical form of the data type. given data structure. Other sections use the logical ADT for the data structure in the context of a higher-level task. Example 1.9 A particular C++ environment might provide a library that includes a list class. The logical form of the list is defined by the public functions, their inputs, and their outputs that define the class. This might be all that you know about the list class implementation, and this should be all you need to know. Within the class, a variety of physical implementations for lists is possible. Several are described in Section 4.1. 1.3 Design Patterns At a higher level of abstraction than ADTs are abstractions for describing the design of programs — that is, the interactions of objects and classes. Experienced software designers learn and reuse patterns for combining software components. These have come to be referred to as design patterns. A design pattern embodies and generalizes important design concepts for a recurring problem. A primary goal of design patterns is to quickly transfer the knowledge gained by expert designers to newer programmers. Another goal is to allow for efficient communication between programmers. It is much easier to discuss a design issue when you share a technical vocabulary relevant to the topic. Specific design patterns emerge from the realization that a particular design problem appears repeatedly in many contexts. They are meant to solve real prob- lems. Design patterns are a bit like templates. They describe the structure for a design solution, with the details filled in for any given problem. Design patterns are a bit like data structures: Each one provides costs and benefits, which implies Sec. 1.3 Design Patterns 13 that tradeoffs are possible. Therefore, a given design pattern might have variations on its application to match the various tradeoffs inherent in a given situation. The rest of this section introduces a few simple design patterns that are used later in the book. 1.3.1 Flyweight The Flyweight design pattern is meant to solve the following problem. You have an application with many objects. Some of these objects are identical in the informa- tion that they contain, and the role that they play. But they must be reached from various places, and conceptually they really are distinct objects. Because there is so much duplication of the same information, we would like to take advantage of the opportunity to reduce memory cost by sharing that space. An example comes from representing the layout for a document. The letter “C” might reasonably be represented by an object that describes that character’s strokes and bounding box. However, we do not want to create a separate “C” object everywhere in the doc- ument that a “C” appears. The solution is to allocate a single copy of the shared representation for “C” objects. Then, every place in the document that needs a “C” in a given font, size, and typeface will reference this single copy. The various instances of references to a specific form of “C” are called flyweights. We could describe the layout of text on a page by using a tree structure. The root of the tree represents the entire page. The page has multiple child nodes, one for each column. The column nodes have child nodes for each row. And the rows have child nodes for each character. These representations for characters are the fly- weights. The flyweight includes the reference to the shared shape information, and might contain additional information specific to that instance. For example, each instance for “C” will contain a reference to the shared information about strokes and shapes, and it might also contain the exact location for that instance of the character on the page. Flyweights are used in the implementation for the PR quadtree data structure for storing collections of point objects, described in Section 13.3. In a PR quadtree, we again have a tree with leaf nodes. Many of these leaf nodes represent empty areas, and so the only information that they store is the fact that they are empty. These identical nodes can be implemented using a reference to a single instance of the flyweight for better memory efficiency. 1.3.2 Visitor Given a tree of objects to describe a page layout, we might wish to perform some activity on every node in the tree. Section 5.2 discusses tree traversal, which is the process of visiting every node in the tree in a defined order. A simple example for our text composition application might be to count the number of nodes in the tree 14 Chap. 1 Data Structures and Algorithms that represents the page. At another time, we might wish to print a listing of all the nodes for debugging purposes. We could write a separate traversal function for each such activity that we in- tend to perform on the tree. A better approach would be to write a generic traversal function, and pass in the activity to be performed at each node. This organization constitutes the visitor design pattern. The visitor design pattern is used in Sec- tions 5.2 (tree traversal) and 11.3 (graph traversal). 1.3.3 Composite There are two fundamental approaches to dealing with the relationship between a collection of actions and a hierarchy of object types. First consider the typical procedural approach. Say we have a base class for page layout entities, with a sub- class hierarchy to define specific subtypes (page, columns, rows, figures, charac- ters, etc.). And say there are actions to be performed on a collection of such objects (such as rendering the objects to the screen). The procedural design approach is for each action to be implemented as a method that takes as a parameter a pointer to the base class type. Each action such method will traverse through the collection of objects, visiting each object in turn. Each action method contains something like a switch statement that defines the details of the action for each subclass in the collection (e.g., page, column, row, character). We can cut the code down some by using the visitor design pattern so that we only need to write the traversal once, and then write a visitor subroutine for each action that might be applied to the collec- tion of objects. But each such visitor subroutine must still contain logic for dealing with each of the possible subclasses. In our page composition application, there are only a few activities that we would like to perform on the page representation. We might render the objects in full detail. Or we might want a “rough draft” rendering that prints only the bound- ing boxes of the objects. If we come up with a new activity to apply to the collection of objects, we do not need to change any of the code that implements the existing activities. But adding new activities won’t happen often for this application. In contrast, there could be many object types, and we might frequently add new ob- ject types to our implementation. Unfortunately, adding a new object type requires that we modify each activity, and the subroutines implementing the activities get rather long switch statements to distinguish the behavior of the many subclasses. An alternative design is to have each object subclass in the hierarchy embody the action for each of the various activities that might be performed. Each subclass will have code to perform each activity (such as full rendering or bounding box rendering). Then, if we wish to apply the activity to the collection, we simply call the first object in the collection and specify the action (as a method call on that object). In the case of our page layout and its hierarchical collection of objects, those objects that contain other objects (such as a row objects that contains letters) Sec. 1.3 Design Patterns 15 will call the appropriate method for each child. If we want to add a new activity with this organization, we have to change the code for every subclass. But this is relatively rare for our text compositing application. In contrast, adding a new object into the subclass hierarchy (which for this application is far more likely than adding a new rendering function) is easy. Adding a new subclass does not require changing any of the existing subclasses. It merely requires that we define the behavior of each activity that can be performed on the new subclass. This second design approach of burying the functional activity in the subclasses is called the Composite design pattern. A detailed example for using the Composite design pattern is presented in Section 5.3.1. 1.3.4 Strategy Our final example of a design pattern lets us encapsulate and make interchangeable a set of alternative actions that might be performed as part of some larger activity. Again continuing our text compositing example, each output device that we wish to render to will require its own function for doing the actual rendering. That is, the objects will be broken down into constituent pixels or strokes, but the actual mechanics of rendering a pixel or stroke will depend on the output device. We don’t want to build this rendering functionality into the object subclasses. Instead, we want to pass to the subroutine performing the rendering action a method or class that does the appropriate rendering details for that output device. That is, we wish to hand to the object the appropriate “strategy” for accomplishing the details of the rendering task. Thus, this approach is called the Strategy design pattern. The Strategy design pattern will be discussed further in Chapter 7. There, a sorting function is given a class (called a comparator) that understands how to extract and compare the key values for records to be sorted. In this way, the sorting function does not need to know any details of how its record type is implemented. One of the biggest challenges to understanding design patterns is that some- times one is only subtly different from another. For example, you might be con- fused about the difference between the composite pattern and the visitor pattern. The distinction is that the composite design pattern is about whether to give control of the traversal process to the nodes of the tree or to the tree itself. Both approaches can make use of the visitor design pattern to avoid rewriting the traversal function many times, by encapsulating the activity performed at each node. But isn’t the strategy design pattern doing the same thing? The difference be- tween the visitor pattern and the strategy pattern is more subtle. Here the difference is primarily one of intent and focus. In both the strategy design pattern and the visi- tor design pattern, an activity is being passed in as a parameter. The strategy design pattern is focused on encapsulating an activity that is part of a larger process, so that different ways of performing that activity can be substituted. The visitor de- sign pattern is focused on encapsulating an activity that will be performed on all 16 Chap. 1 Data Structures and Algorithms members of a collection so that completely different activities can be substituted within a generic method that accesses all of the collection members. 1.4 Problems, Algorithms, and Programs Programmers commonly deal with problems, algorithms, and computer programs. These are three distinct concepts. Problems: As your intuition would suggest, a problem is a task to be performed. It is best thought of in terms of inputs and matching outputs. A problem definition should not include any constraints on how the problem is to be solved. The solution method should be developed only after the problem is precisely defined and thor- oughly understood. However, a problem definition should include constraints on the resources that may be consumed by any acceptable solution. For any problem to be solved by a computer, there are always such constraints, whether stated or implied. For example, any computer program may use only the main memory and disk space available, and it must run in a “reasonable” amount of time. Problems can be viewed as functions in the mathematical sense. A function is a matching between inputs (the domain) and outputs (the range). An input to a function might be a single value or a collection of information. The values making up an input are called the parameters of the function. A specific selection of values for the parameters is called an instance of the problem. For example, the input parameter to a sorting function might be an array of integers. A particular array of integers, with a given size and specific values for each position in the array, would be an instance of the sorting problem. Different instances might generate the same output. However, any problem instance must always result in the same output every time the function is computed using that particular input. This concept of all problems behaving like mathematical functions might not match your intuition for the behavior of computer programs. You might know of programs to which you can give the same input value on two separate occasions, and two different outputs will result. For example, if you type “date” to a typical UNIX command line prompt, you will get the current date. Naturally the date will be different on different days, even though the same command is given. However, there is obviously more to the input for the date program than the command that you type to run the program. The date program computes a function. In other words, on any particular day there can only be a single answer returned by a properly running date program on a completely specified input. For all computer programs, the output is completely determined by the program’s full set of inputs. Even a “random number generator” is completely determined by its inputs (although some random number generating systems appear to get around this by accepting a random input from a physical process beyond the user’s control). The relationship between programs and functions is explored further in Section 17.3. Sec. 1.4 Problems, Algorithms, and Programs 17 Algorithms: An algorithm is a method or a process followed to solve a problem. If the problem is viewed as a function, then an algorithm is an implementation for the function that transforms an input to the corresponding output. A problem can be solved by many different algorithms. A given algorithm solves only one problem (i.e., computes a particular function). This book covers many problems, and for several of these problems I present more than one algorithm. For the important problem of sorting I present nearly a dozen algorithms! The advantage of knowing several solutions to a problem is that solution A might be more efficient than solution B for a specific variation of the problem, or for a specific class of inputs to the problem, while solution B might be more efficient than A for another variation or class of inputs. For example, one sorting algorithm might be the best for sorting a small collection of integers (which is important if you need to do this many times). Another might be the best for sorting a large collection of integers. A third might be the best for sorting a collection of variable-length strings. By definition, something can only be called an algorithm if it has all of the following properties. 1. It must be correct. In other words, it must compute the desired function, converting each input to the correct output. Note that every algorithm im- plements some function, because every algorithm maps every input to some output (even if that output is a program crash). At issue here is whether a given algorithm implements the intended function. 2. It is composed of a series of concrete steps. Concrete means that the action described by that step is completely understood — and doable — by the person or machine that must perform the algorithm. Each step must also be doable in a finite amount of time. Thus, the algorithm gives us a “recipe” for solving the problem by performing a series of steps, where each such step is within our capacity to perform. The ability to perform a step can depend on who or what is intended to execute the recipe. For example, the steps of a cookie recipe in a cookbook might be considered sufficiently concrete for instructing a human cook, but not for programming an automated cookie- making factory. 3. There can be no ambiguity as to which step will be performed next. Often it is the next step of the algorithm description. Selection (e.g., the if statement in C++) is normally a part of any language for describing algorithms. Selec- tion allows a choice for which step will be performed next, but the selection process is unambiguous at the time when the choice is made. 4. It must be composed of a finite number of steps. If the description for the alg- orithm were made up of an infinite number of steps, we could never hope to write it down, nor implement it as a computer program. Most languages for describing algorithms (including English and “pseudocode”) provide some 18 Chap. 1 Data Structures and Algorithms way to perform repeated actions, known as iteration. Examples of iteration in programming languages include the while and for loop constructs of C++. Iteration allows for short descriptions, with the number of steps actu- ally performed controlled by the input. 5. It must terminate. In other words, it may not go into an infinite loop. Programs: We often think of a computer program as an instance, or concrete representation, of an algorithm in some programming language. In this book, nearly all of the algorithms are presented in terms of programs, or parts of pro- grams. Naturally, there are many programs that are instances of the same alg- orithm, because any modern computer programming language can be used to im- plement the same collection of algorithms (although some programming languages can make life easier for the programmer). To simplify presentation, I often use the terms “algorithm” and “program” interchangeably, despite the fact that they are really separate concepts. By definition, an algorithm must provide sufficient detail that it can be converted into a program when needed. The requirement that an algorithm must terminate means that not all computer programs meet the technical definition of an algorithm. Your operating system is one such program. However, you can think of the various tasks for an operating sys- tem (each with associated inputs and outputs) as individual problems, each solved by specific algorithms implemented by a part of the operating system program, and each one of which terminates once its output is produced. To summarize: A problem is a function or a mapping of inputs to outputs. An algorithm is a recipe for solving a problem whose steps are concrete and un- ambiguous. Algorithms must be correct, of finite length, and must terminate for all inputs. A program is an instantiation of an algorithm in a programming language. 1.5 Further Reading An early authoritative work on data structures and algorithms was the series of books The Art of Computer Programming by Donald E. Knuth, with Volumes 1 and 3 being most relevant to the study of data structures [Knu97, Knu98]. A mod- ern encyclopedic approach to data structures and algorithms that should be easy to understand once you have mastered this book is Algorithms by Robert Sedge- wick [Sed11]. For an excellent and highly readable (but more advanced) teaching introduction to algorithms, their design, and their analysis, see Introduction to Al- gorithms: A Creative Approach by Udi Manber [Man89]. For an advanced, en- cyclopedic approach, see Introduction to Algorithms by Cormen, Leiserson, and Rivest [CLRS09]. Steven S. Skiena’s The Algorithm Design Manual [Ski10] pro- vides pointers to many implementations for data structures and algorithms that are available on the Web. Sec. 1.5 Further Reading 19 The claim that all modern programming languages can implement the same algorithms (stated more precisely, any function that is computable by one program- ming language is computable by any programming language with certain standard capabilities) is a key result from computability theory. For an easy introduction to this field see James L. Hein, Discrete Structures, Logic, and Computability [Hei09]. Much of computer science is devoted to problem solving. Indeed, this is what attracts many people to the field. How to Solve It by George P´olya [P´ol57] is con- sidered to be the classic work on how to improve your problem-solving abilities. If you want to be a better student (as well as a better problem solver in general), see Strategies for Creative Problem Solving by Folger and LeBlanc [FL95], Effective Problem Solving by Marvin Levine [Lev94], and Problem Solving & Comprehen- sion by Arthur Whimbey and Jack Lochhead [WL99], and Puzzle-Based Learning by Zbigniew and Matthew Michaelewicz [MM08]. See The Origin of Consciousness in the Breakdown of the Bicameral Mind by Julian Jaynes [Jay90] for a good discussion on how humans use the concept of metaphor to handle complexity. More directly related to computer science educa- tion and programming, see “Cogito, Ergo Sum! Cognitive Processes of Students Dealing with Data Structures” by Dan Aharoni [Aha00] for a discussion on mov- ing from programming-context thinking to higher-level (and more design-oriented) programming-free thinking. On a more pragmatic level, most people study data structures to write better programs. If you expect your program to work correctly and efficiently, it must first be understandable to yourself and your co-workers. Kernighan and Pike’s The Practice of Programming [KP99] discusses a number of practical issues related to programming, including good coding and documentation style. For an excellent (and entertaining!) introduction to the difficulties involved with writing large pro- grams, read the classic The Mythical Man-Month: Essays on Software Engineering by Frederick P. Brooks [Bro95]. If you want to be a successful C++ programmer, you need good reference manuals close at hand. The standard reference for C++ is The C++ Program- ming Language by Bjarne Stroustrup [Str00], with further information provided in The Annotated C++ Reference Manual by Ellis and Stroustrup [ES90]. No C++ programmer should be without Stroustrup’s book, as it provides the definitive de- scription of the language and also includes a great deal of information about the principles of object-oriented design. Unfortunately, it is a poor text for learning how to program in C++. A good, gentle introduction to the basics of the language is Patrick Henry Winston’s On to C++ [Win94]. A good introductory teaching text for a wider range of C++ is Deitel and Deitel’s C++ How to Program [DD08]. After gaining proficiency in the mechanics of program writing, the next step is to become proficient in program design. Good design is difficult to learn in any discipline, and good design for object-oriented software is one of the most difficult 20 Chap. 1 Data Structures and Algorithms of arts. The novice designer can jump-start the learning process by studying well- known and well-used design patterns. The classic reference on design patterns is Design Patterns: Elements of Reusable Object-Oriented Software by Gamma, Helm, Johnson, and Vlissides [GHJV95] (this is commonly referred to as the “gang of four” book). Unfortunately, this is an extremely difficult book to understand, in part because the concepts are inherently difficult. A number of Web sites are available that discuss design patterns, and which provide study guides for the De- sign Patterns book. Two other books that discuss object-oriented software design are Object-Oriented Software Design and Construction with C++ by Dennis Ka- fura [Kaf98], and Object-Oriented Design Heuristics by Arthur J. Riel [Rie96]. 1.6 Exercises The exercises for this chapter are different from those in the rest of the book. Most of these exercises are answered in the following chapters. However, you should not look up the answers in other parts of the book. These exercises are intended to make you think about some of the issues to be covered later on. Answer them to the best of your ability with your current knowledge. 1.1 Think of a program you have used that is unacceptably slow. Identify the spe- cific operations that make the program slow. Identify other basic operations that the program performs quickly enough. 1.2 Most programming languages have a built-in integer data type. Normally this representation has a fixed size, thus placing a limit on how large a value can be stored in an integer variable. Describe a representation for integers that has no size restriction (other than the limits of the computer’s available main memory), and thus no practical limit on how large an integer can be stored. Briefly show how your representation can be used to implement the operations of addition, multiplication, and exponentiation. 1.3 Define an ADT for character strings. Your ADT should consist of typical functions that can be performed on strings, with each function defined in terms of its input and output. Then define two different physical representa- tions for strings. 1.4 Define an ADT for a list of integers. First, decide what functionality your ADT should provide. Example 1.4 should give you some ideas. Then, spec- ify your ADT in C++ in the form of an abstract class declaration, showing the functions, their parameters, and their return types. 1.5 Briefly describe how integer variables are typically represented on a com- puter. (Look up one’s complement and two’s complement arithmetic in an introductory computer science textbook if you are not familiar with these.) Sec. 1.6 Exercises 21 Why does this representation for integers qualify as a data structure as de- fined in Section 1.2? 1.6 Define an ADT for a two-dimensional array of integers. Specify precisely the basic operations that can be performed on such arrays. Next, imagine an application that stores an array with 1000 rows and 1000 columns, where less than 10,000 of the array values are non-zero. Describe two different imple- mentations for such arrays that would be more space efficient than a standard two-dimensional array implementation requiring one million positions. 1.7 Imagine that you have been assigned to implement a sorting program. The goal is to make this program general purpose, in that you don’t want to define in advance what record or key types are used. Describe ways to generalize a simple sorting algorithm (such as insertion sort, or any other sort you are familiar with) to support this generalization. 1.8 Imagine that you have been assigned to implement a simple sequential search on an array. The problem is that you want the search to be as general as pos- sible. This means that you need to support arbitrary record and key types. Describe ways to generalize the search function to support this goal. Con- sider the possibility that the function will be used multiple times in the same program, on differing record types. Consider the possibility that the func- tion will need to be used on different keys (possibly with the same or differ- ent types) of the same record. For example, a student data record might be searched by zip code, by name, by salary, or by GPA. 1.9 Does every problem have an algorithm? 1.10 Does every algorithm have a C++ program? 1.11 Consider the design for a spelling checker program meant to run on a home computer. The spelling checker should be able to handle quickly a document of less than twenty pages. Assume that the spelling checker comes with a dictionary of about 20,000 words. What primitive operations must be imple- mented on the dictionary, and what is a reasonable time constraint for each operation? 1.12 Imagine that you have been hired to design a database service containing information about cities and towns in the United States, as described in Ex- ample 1.2. Suggest two possible implementations for the database. 1.13 Imagine that you are given an array of records that is sorted with respect to some key field contained in each record. Give two different algorithms for searching the array to find the record with a specified key value. Which one do you consider “better” and why? 1.14 How would you go about comparing two proposed algorithms for sorting an array of integers? In particular, (a) What would be appropriate measures of cost to use as a basis for com- paring the two sorting algorithms? 22 Chap. 1 Data Structures and Algorithms (b) What tests or analysis would you conduct to determine how the two algorithms perform under these cost measures? 1.15 A common problem for compilers and text editors is to determine if the parentheses (or other brackets) in a string are balanced and properly nested. For example, the string “((())())()” contains properly nested pairs of paren- theses, but the string “)()(” does not; and the string “())” does not contain properly matching parentheses. (a) Give an algorithm that returns true if a string contains properly nested and balanced parentheses, and false if otherwise. Hint: At no time while scanning a legal string from left to right will you have encoun- tered more right parentheses than left parentheses. (b) Give an algorithm that returns the position in the string of the first of- fending parenthesis if the string is not properly nested and balanced. That is, if an excess right parenthesis is found, return its position; if there are too many left parentheses, return the position of the first ex- cess left parenthesis. Return 1 if the string is properly balanced and nested. 1.16 A graph consists of a set of objects (called vertices) and a set of edges, where each edge connects two vertices. Any given pair of vertices can be connected by only one edge. Describe at least two different ways to represent the con- nections defined by the vertices and edges of a graph. 1.17 Imagine that you are a shipping clerk for a large company. You have just been handed about 1000 invoices, each of which is a single sheet of paper with a large number in the upper right corner. The invoices must be sorted by this number, in order from lowest to highest. Write down as many different approaches to sorting the invoices as you can think of. 1.18 How would you sort an array of about 1000 integers from lowest value to highest value? Write down at least five approaches to sorting the array. Do not write algorithms in C++ or pseudocode. Just write a sentence or two for each approach to describe how it would work. 1.19 Think of an algorithm to find the maximum value in an (unsorted) array. Now, think of an algorithm to find the second largest value in the array. Which is harder to implement? Which takes more time to run (as measured by the number of comparisons performed)? Now, think of an algorithm to find the third largest value. Finally, think of an algorithm to find the middle value. Which is the most difficult of these problems to solve? 1.20 An unsorted list allows for constant-time insert by adding a new element at the end of the list. Unfortunately, searching for the element with key value X requires a sequential search through the unsorted list until X is found, which on average requires looking at half the list element. On the other hand, a Sec. 1.6 Exercises 23 sorted array-based list of n elements can be searched in log n time with a binary search. Unfortunately, inserting a new element requires a lot of time because many elements might be shifted in the array if we want to keep it sorted. How might data be organized to support both insertion and search in log n time? 2 Mathematical Preliminaries This chapter presents mathematical notation, background, and techniques used throughout the book. This material is provided primarily for review and reference. You might wish to return to the relevant sections when you encounter unfamiliar notation or mathematical techniques in later chapters. Section 2.7 on estimation might be unfamiliar to many readers. Estimation is not a mathematical technique, but rather a general engineering skill. It is enor- mously useful to computer scientists doing design work, because any proposed solution whose estimated resource requirements fall well outside the problem’s re- source constraints can be discarded immediately, allowing time for greater analysis of more promising solutions. 2.1 Sets and Relations The concept of a set in the mathematical sense has wide application in computer science. The notations and techniques of set theory are commonly used when de- scribing and implementing algorithms because the abstractions associated with sets often help to clarify and simplify algorithm design. A set is a collection of distinguishable members or elements. The members are typically drawn from some larger population known as the base type. Each member of a set is either a primitive element of the base type or is a set itself. There is no concept of duplication in a set. Each value from the base type is either in the set or not in the set. For example, a set named P might consist of the three integers 7, 11, and 42. In this case, P’s members are 7, 11, and 42, and the base type is integer. Figure 2.1 shows the symbols commonly used to express sets and their rela- tionships. Here are some examples of this notation in use. First define two sets, P and Q. P = {2, 3, 5}, Q = {5, 10}. 25 26 Chap. 2 Mathematical Preliminaries {1, 4} A set composed of the members 1 and 4 {x | x is a positive integer} A set definition using a set former Example: the set of all positive integers x 2 P x is a member of set P x /2 P x is not a member of set P ; The null or empty set |P| Cardinality: size of set P or number of members for set P P ✓ Q, Q ◆ P Set P is included in set Q, set P is a subset of set Q, set Q is a superset of set P P [ Q Set Union: all elements appearing in P OR Q P \ Q Set Intersection: all elements appearing in P AND Q P Q Set difference: all elements of set P NOT in set Q Figure 2.1 Set notation. |P| =3(because P has three members) and |Q| =2(because Q has two members). The union of P and Q, written P [ Q, is the set of elements in either P or Q, which is {2, 3, 5, 10}. The intersection of P and Q, written P \ Q, is the set of elements that appear in both P and Q, which is {5}. The set difference of P and Q, written P Q, is the set of elements that occur in P but not in Q, which is {2, 3}. Note that P [ Q = Q [ P and that P \ Q = Q \ P, but in general P Q 6= Q P. In this example, Q P = {10}. Note that the set {4, 3, 5} is indistinguishable from set P, because sets have no concept of order. Likewise, set {4, 3, 4, 5} is also indistinguishable from P, because sets have no concept of duplicate elements. The powerset of a set S is the set of all possible subsets for S. Consider the set S = {a, b, c}. The powerset of S is {;, {a}, {b}, {c}, {a, b}, {a, c}, {b, c}, {a, b, c}}. A collection of elements with no order (like a set), but with duplicate-valued el- ements is called a bag.1 To distinguish bags from sets, I use square brackets [] around a bag’s elements. For example, bag [3, 4, 5, 4] is distinct from bag [3, 4, 5], while set {3, 4, 5, 4} is indistinguishable from set {3, 4, 5}. However, bag [3, 4, 5, 4] is indistinguishable from bag [3, 4, 4, 5]. 1The object referred to here as a bag is sometimes called a multilist. But, I reserve the term multilist for a list that may contain sublists (see Section 12.1). Sec. 2.1 Sets and Relations 27 A sequence is a collection of elements with an order, and which may contain duplicate-valued elements. A sequence is also sometimes called a tuple or a vec- tor. In a sequence, there is a 0th element, a 1st element, 2nd element, and so on. I indicate a sequence by using angle brackets hi to enclose its elements. For example, h3, 4, 5, 4i is a sequence. Note that sequence h3, 5, 4, 4i is distinct from sequence h3, 4, 5, 4i, and both are distinct from sequence h3, 4, 5i. A relation R over set S is a set of ordered pairs from S. As an example of a relation, if S is {a, b, c}, then {ha, ci, hb, ci, hc, bi} is a relation, and {ha, ai, ha, ci, hb, bi, hb, ci, hc, ci} is a different relation. If tuple hx, yi is in relation R, we may use the infix notation xRy. We often use relations such as the less than operator (<) on the natural numbers, which includes ordered pairs such as h1, 3i and h2, 23i, but not h3, 2i or h2, 2i. Rather than writing the relationship in terms of ordered pairs, we typically use an infix notation for such relations, writing 1 < 3. Define the properties of relations as follows, with R a binary relation over set S. • R is reflexive if aRa for all a 2 S. • R is symmetric if whenever aRb, then bRa, for all a, b 2 S. • R is antisymmetric if whenever aRb and bRa, then a = b, for all a, b 2 S. • R is transitive if whenever aRb and bRc, then aRc, for all a, b, c 2 S. As examples, for the natural numbers, < is antisymmetric (because there is no case where aRb and bRa) and transitive;  is reflexive, antisymmetric, and transitive, and = is reflexive, symmetric (and antisymmetric!), and transitive. For people, the relation “is a sibling of” is symmetric and transitive. If we define a person to be a sibling of himself, then it is reflexive; if we define a person not to be a sibling of himself, then it is not reflexive. R is an equivalence relation on set S if it is reflexive, symmetric, and transitive. An equivalence relation can be used to partition a set into equivalence classes. If two elements a and b are equivalent to each other, we write a ⌘ b.Apartition of a set S is a collection of subsets that are disjoint from each other and whose union is S. An equivalence relation on set S partitions the set into subsets whose elements are equivalent. See Section 6.2 for a discussion on how to represent equivalence classes on a set. One application for disjoint sets appears in Section 11.5.2. Example 2.1 For the integers, = is an equivalence relation that partitions each element into a distinct subset. In other words, for any integer a, three things are true. 1. a = a, 28 Chap. 2 Mathematical Preliminaries 2. if a = b then b = a, and 3. if a = b and b = c, then a = c. Of course, for distinct integers a, b, and c there are never cases where a = b, b = a, or b = c. So the claims that = is symmetric and transitive are vacuously true (there are never examples in the relation where these events occur). But because the requirements for symmetry and transitivity are not violated, the relation is symmetric and transitive. Example 2.2 If we clarify the definition of sibling to mean that a person is a sibling of him- or herself, then the sibling relation is an equivalence relation that partitions the set of people. Example 2.3 We can use the modulus function (defined in the next sec- tion) to define an equivalence relation. For the set of integers, use the mod- ulus function to define a binary relation such that two numbers x and y are in the relation if and only if x mod m = y mod m. Thus, for m =4, h1, 5i is in the relation because 1mod4=5mod4. We see that modulus used in this way defines an equivalence relation on the integers, and this re- lation can be used to partition the integers into m equivalence classes. This relation is an equivalence relation because 1. x mod m = x mod m for all x; 2. if x mod m = y mod m, then y mod m = x mod m; and 3. if x mod m = y mod m and y mod m = z mod m, then x mod m = z mod m. A binary relation is called a partial order if it is antisymmetric and transitive.2 The set on which the partial order is defined is called a partially ordered set or a poset. Elements x and y of a set are comparable under a given relation if either xRy or yRx. If every pair of distinct elements in a partial order are comparable, then the order is called a total order or linear order. Example 2.4 For the integers, relations < and  define partial orders. Operation < is a total order because, for every pair of integers x and y such that x 6= y, either x void permute(E A[], int n) { for (int i=n; i>0; i--) swap(A, i-1, Random(i)); } Boolean variables: A Boolean variable is a variable (of type bool in C++) that takes on one of the two values true and false. These two values are often associated with the values 1 and 0, respectively, although there is no reason why this needs to be the case. It is poor programming practice to rely on the corre- spondence between 0 and false, because these are logically distinct objects of different types. Logic Notation: We will occasionally make use of the notation of symbolic or Boolean logic. A ) B means “A implies B” or “If A then B.” A , B means “A if and only if B” or “A is equivalent to B.” A _ B means “A or B” (useful both in the context of symbolic logic or when performing a Boolean operation). A ^ B means “A and B.” ⇠A and A both mean “not A” or the negation of A where A is a Boolean variable. Floor and ceiling: The floor of x (written bxc) takes real value x and returns the greatest integer  x. For example, b3.4c =3, as does b3.0c, while b3.4c = 4 and b3.0c = 3. The ceiling of x (written dxe) takes real value x and returns the least integer x. For example, d3.4e =4, as does d4.0e, while d3.4e = d3.0e = 3. Modulus operator: The modulus (or mod) function returns the remainder of an integer division. Sometimes written n mod m in mathematical expressions, the syntax for the C++ modulus operator is n%m. From the definition of remainder, n mod m is the integer r such that n = qm + r for q an integer, and |r| < |m|. Therefore, the result of n mod m must be between 0 and m 1 when n and m are positive integers. For example, 5mod3=2; 25 mod 3 = 1, 5mod7=5, and 5mod5=0. There is more than one way to assign values to q and r, depending on how in- teger division is interpreted. The most common mathematical definition computes the mod function as n mod m = n mbn/mc. In this case, 3mod5= 2. However, Java and C++ compilers typically use the underlying processor’s ma- chine instruction for computing integer arithmetic. On many computers this is done by truncating the resulting fraction, meaning n mod m = n m(trunc(n/m)). Under this definition, 3mod5=3. Sec. 2.3 Logarithms 31 Unfortunately, for many applications this is not what the user wants or expects. For example, many hash systems will perform some computation on a record’s key value and then take the result modulo the hash table size. The expectation here would be that the result is a legal index into the hash table, not a negative number. Implementers of hash functions must either insure that the result of the computation is always positive, or else add the hash table size to the result of the modulo function when that result is negative. 2.3 Logarithms A logarithm of base b for value y is the power to which b is raised to get y. Nor- mally, this is written as logb y = x. Thus, if logb y = x then bx = y, and blogby = y. Logarithms are used frequently by programmers. Here are two typical uses. Example 2.6 Many programs require an encoding for a collection of ob- jects. What is the minimum number of bits needed to represent n distinct code values? The answer is dlog2 ne bits. For example, if you have 1000 codes to store, you will require at least dlog2 1000e = 10 bits to have 1000 different codes (10 bits provide 1024 distinct code values). Example 2.7 Consider the binary search algorithm for finding a given value within an array sorted by value from lowest to highest. Binary search first looks at the middle element and determines if the value being searched for is in the upper half or the lower half of the array. The algorithm then continues splitting the appropriate subarray in half until the desired value is found. (Binary search is described in more detail in Section 3.5.) How many times can an array of size n be split in half until only one element remains in the final subarray? The answer is dlog2 ne times. In this book, nearly all logarithms used have a base of two. This is because data structures and algorithms most often divide things in half, or store codes with binary bits. Whenever you see the notation log n in this book, either log2 n is meant or else the term is being used asymptotically and so the actual base does not matter. Logarithms using any base other than two will show the base explicitly. Logarithms have the following properties, for any positive values of m, n, and r, and any positive integers a and b. 1. log(nm) = log n + log m. 2. log(n/m) = log n log m. 3. log(nr)=r log n. 4. loga n = logb n/ logb a. 32 Chap. 2 Mathematical Preliminaries The first two properties state that the logarithm of two numbers multiplied (or divided) can be found by adding (or subtracting) the logarithms of the two num- bers.4 Property (3) is simply an extension of property (1). Property (4) tells us that, for variable n and any two integer constants a and b, loga n and logb n differ by the constant factor logb a, regardless of the value of n. Most runtime analyses in this book are of a type that ignores constant factors in costs. Property (4) says that such analyses need not be concerned with the base of the logarithm, because this can change the total cost only by a constant factor. Note that 2log n = n. When discussing logarithms, exponents often lead to confusion. Property (3) tells us that log n2 = 2 log n. How do we indicate the square of the logarithm (as opposed to the logarithm of n2)? This could be written as (log n)2, but it is traditional to use log2 n. On the other hand, we might want to take the logarithm of the logarithm of n. This is written log log n. A special notation is used in the rare case when we need to know how many times we must take the log of a number before we reach a value  1. This quantity is written log⇤ n. For example, log⇤ 1024 = 4 because log 1024 = 10, log 10 ⇡3.33, log 3.33 ⇡ 1.74, and log 1.74 < 1, which is a total of 4 log operations. 2.4 Summations and Recurrences Most programs contain loop constructs. When analyzing running time costs for programs with loops, we need to add up the costs for each time the loop is executed. This is an example of a summation. Summations are simply the sum of costs for some function applied to a range of parameter values. Summations are typically written with the following “Sigma” notation: n Xi=1 f(i). This notation indicates that we are summing the value of f(i) over some range of (integer) values. The parameter to the expression and its initial value are indicated below the P symbol. Here, the notation i =1indicates that the parameter is i and that it begins with the value 1. At the top of the P symbol is the expression n. This indicates the maximum value for the parameter i. Thus, this notation means to sum the values of f(i) as i ranges across the integers from 1 through n. This can also be 4These properties are the idea behind the slide rule. Adding two numbers can be viewed as joining two lengths together and measuring their combined length. Multiplication is not so easily done. However, if the numbers are first converted to the lengths of their logarithms, then those lengths can be added and the inverse logarithm of the resulting length gives the answer for the multiplication (this is simply logarithm property (1)). A slide rule measures the length of the logarithm for the numbers, lets you slide bars representing these lengths to add up the total length, and finally converts this total length to the correct numeric answer by taking the inverse of the logarithm for the result. Sec. 2.4 Summations and Recurrences 33 written f(1) + f(2) + ···+ f(n 1) + f(n). Within a sentence, Sigma notation is typeset as P n i=1 f(i). Given a summation, you often wish to replace it with an algebraic equation with the same value as the summation. This is known as a closed-form solution, and the process of replacing the summation with its closed-form solution is known as solving the summation. For example, the summation P n i=1 1 is simply the ex- pression “1” summed n times (remember that i ranges from 1 to n). Because the sum of n 1s is n, the closed-form solution is n. The following is a list of useful summations, along with their closed-form solutions. n Xi=1 i = n(n + 1) 2 . (2.1) n Xi=1 i2 = 2n3 +3n2 + n 6 = n(2n + 1)(n + 1) 6 . (2.2) log n Xi=1 n = n log n. (2.3) 1 Xi=0 ai = 1 1 a for 0 1; 1! = 0! = 1. Another standard example of a recurrence is the Fibonacci sequence: Fib(n)=Fib(n 1) + Fib(n 2) for n>2; Fib(1) = Fib(2) = 1. From this definition, the first seven numbers of the Fibonacci sequence are 1, 1, 2, 3, 5, 8, and 13. Notice that this definition contains two parts: the general definition for Fib(n) and the base cases for Fib(1) and Fib(2). Likewise, the definition for factorial contains a recursive part and base cases. Recurrence relations are often used to model the cost of recursive functions. For example, the number of multiplications required by function fact of Section 2.5 for an input of size n will be zero when n =0or n =1(the base cases), and it will be one plus the cost of calling fact on a value of n 1. This can be defined using the following recurrence: T(n)=T(n 1) + 1 for n>1; T(0) = T(1) = 0. As with summations, we typically wish to replace the recurrence relation with a closed-form solution. One approach is to expand the recurrence by replacing any occurrences of T on the right-hand side with its definition. Example 2.8 If we expand the recurrence T(n)=T(n 1) + 1, we get T(n)=T(n 1) + 1 =(T(n 2) + 1) + 1. Sec. 2.4 Summations and Recurrences 35 We can expand the recurrence as many steps as we like, but the goal is to detect some pattern that will permit us to rewrite the recurrence in terms of a summation. In this example, we might notice that (T(n 2) + 1) + 1 = T(n 2) + 2 and if we expand the recurrence again, we get T(n)=T(n 2) + 2 = T(n 3) + 1 + 2 = T(n 3) + 3 which generalizes to the pattern T(n)=T(n i)+i. We might conclude that T(n)=T(n (n 1)) + (n 1) = T(1) + n 1 = n 1. Because we have merely guessed at a pattern and not actually proved that this is the correct closed form solution, we should use an induction proof to complete the process (see Example 2.13). Example 2.9 A slightly more complicated recurrence is T(n)=T(n 1) + n; T(1) = 1. Expanding this recurrence a few steps, we get T(n)=T(n 1) + n = T(n 2) + (n 1) + n = T(n 3) + (n 2) + (n 1) + n. We should then observe that this recurrence appears to have a pattern that leads to T(n)=T(n (n 1)) + (n (n 2)) + ···+(n 1) + n =1+2+···+(n 1) + n. This is equivalent to the summation P n i=1 i, for which we already know the closed-form solution. Techniques to find closed-form solutions for recurrence relations are discussed in Section 14.2. Prior to Chapter 14, recurrence relations are used infrequently in this book, and the corresponding closed-form solution and an explanation for how it was derived will be supplied at the time of use. 36 Chap. 2 Mathematical Preliminaries 2.5 Recursion An algorithm is recursive if it calls itself to do part of its work. For this approach to be successful, the “call to itself” must be on a smaller problem then the one originally attempted. In general, a recursive algorithm must have two parts: the base case, which handles a simple input that can be solved without resorting to a recursive call, and the recursive part which contains one or more recursive calls to the algorithm where the parameters are in some sense “closer” to the base case than those of the original call. Here is a recursive C++ function to compute the factorial of n. A trace of fact’s execution for a small value of n is presented in Section 4.2.4. long fact(int n) { // Compute n! recursively // To fit n! into a long variable, we require n <= 12 Assert((n >= 0) && (n <= 12), "Input out of range"); if (n <= 1) return 1; // Base case: return base solution return n * fact(n-1); // Recursive call for n > 1 } The first two lines of the function constitute the base cases. If n  1, then one of the base cases computes a solution for the problem. If n>1, then fact calls a function that knows how to find the factorial of n 1. Of course, the function that knows how to compute the factorial of n 1 happens to be fact itself. But we should not think too hard about this while writing the algorithm. The design for recursive algorithms can always be approached in this way. First write the base cases. Then think about solving the problem by combining the results of one or more smaller — but similar — subproblems. If the algorithm you write is correct, then certainly you can rely on it (recursively) to solve the smaller subproblems. The secret to success is: Do not worry about how the recursive call solves the subproblem. Simply accept that it will solve it correctly, and use this result to in turn correctly solve the original problem. What could be simpler? Recursion has no counterpart in everyday, physical-world problem solving. The concept can be difficult to grasp because it requires you to think about problems in a new way. To use recursion effectively, it is necessary to train yourself to stop analyzing the recursive process beyond the recursive call. The subproblems will take care of themselves. You just worry about the base cases and how to recombine the subproblems. The recursive version of the factorial function might seem unnecessarily com- plicated to you because the same effect can be achieved by using a while loop. Here is another example of recursion, based on a famous puzzle called “Towers of Hanoi.” The natural algorithm to solve this problem has multiple recursive calls. It cannot be rewritten easily using while loops. The Towers of Hanoi puzzle begins with three poles and n rings, where all rings start on the leftmost pole (labeled Pole 1). The rings each have a different size, and Sec. 2.5 Recursion 37 (a) (b) Figure 2.2 Towers of Hanoi example. (a) The initial conditions for a problem with six rings. (b) A necessary intermediate step on the road to a solution. are stacked in order of decreasing size with the largest ring at the bottom, as shown in Figure 2.2(a). The problem is to move the rings from the leftmost pole to the rightmost pole (labeled Pole 3) in a series of steps. At each step the top ring on some pole is moved to another pole. There is one limitation on where rings may be moved: A ring can never be moved on top of a smaller ring. How can you solve this problem? It is easy if you don’t think too hard about the details. Instead, consider that all rings are to be moved from Pole 1 to Pole 3. It is not possible to do this without first moving the bottom (largest) ring to Pole 3. To do that, Pole 3 must be empty, and only the bottom ring can be on Pole 1. The remaining n 1 rings must be stacked up in order on Pole 2, as shown in Figure 2.2(b). How can you do this? Assume that a function X is available to solve the problem of moving the top n 1 rings from Pole 1 to Pole 2. Then move the bottom ring from Pole 1 to Pole 3. Finally, again use function X to move the remaining n 1 rings from Pole 2 to Pole 3. In both cases, “function X” is simply the Towers of Hanoi function called on a smaller version of the problem. The secret to success is relying on the Towers of Hanoi algorithm to do the work for you. You need not be concerned about the gory details of how the Towers of Hanoi subproblem will be solved. That will take care of itself provided that two things are done. First, there must be a base case (what to do if there is only one ring) so that the recursive process will not go on forever. Second, the recursive call to Towers of Hanoi can only be used to solve a smaller problem, and then only one of the proper form (one that meets the original definition for the Towers of Hanoi problem, assuming appropriate renaming of the poles). Here is an implementation for the recursive Towers of Hanoi algorithm. Func- tion move(start, goal) takes the top ring from Pole start and moves it to Pole goal. If move were to print the values of its parameters, then the result of calling TOH would be a list of ring-moving instructions that solves the problem. 38 Chap. 2 Mathematical Preliminaries void TOH(int n, Pole start, Pole goal, Pole temp) { if (n == 0) return; // Base case TOH(n-1, start, temp, goal); // Recursive call: n-1 rings move(start, goal); // Move bottom disk to goal TOH(n-1, temp, goal, start); // Recursive call: n-1 rings } Those who are unfamiliar with recursion might find it hard to accept that it is used primarily as a tool for simplifying the design and description of algorithms. A recursive algorithm usually does not yield the most efficient computer program for solving the problem because recursion involves function calls, which are typi- cally more expensive than other alternatives such as a while loop. However, the recursive approach usually provides an algorithm that is reasonably efficient in the sense discussed in Chapter 3. (But not always! See Exercise 2.11.) If necessary, the clear, recursive solution can later be modified to yield a faster implementation, as described in Section 4.2.4. Many data structures are naturally recursive, in that they can be defined as be- ing made up of self-similar parts. Tree structures are an example of this. Thus, the algorithms to manipulate such data structures are often presented recursively. Many searching and sorting algorithms are based on a strategy of divide and con- quer. That is, a solution is found by breaking the problem into smaller (similar) subproblems, solving the subproblems, then combining the subproblem solutions to form the solution to the original problem. This process is often implemented using recursion. Thus, recursion plays an important role throughout this book, and many more examples of recursive functions will be given. 2.6 Mathematical Proof Techniques Solving any problem has two distinct parts: the investigation and the argument. Students are too used to seeing only the argument in their textbooks and lectures. But to be successful in school (and in life after school), one needs to be good at both, and to understand the differences between these two phases of the process. To solve the problem, you must investigate successfully. That means engaging the problem, and working through until you find a solution. Then, to give the answer to your client (whether that “client” be your instructor when writing answers on a homework assignment or exam, or a written report to your boss), you need to be able to make the argument in a way that gets the solution across clearly and succinctly. The argument phase involves good technical writing skills — the ability to make a clear, logical argument. Being conversant with standard proof techniques can help you in this process. Knowing how to write a good proof helps in many ways. First, it clarifies your thought process, which in turn clarifies your explanations. Second, if you use one of the standard proof structures such as proof by contradiction or an induction proof, Sec. 2.6 Mathematical Proof Techniques 39 then both you and your reader are working from a shared understanding of that structure. That makes for less complexity to your reader to understand your proof, because the reader need not decode the structure of your argument from scratch. This section briefly introduces three commonly used proof techniques: (i) de- duction, or direct proof; (ii) proof by contradiction, and (iii) proof by mathematical induction. 2.6.1 Direct Proof In general, a direct proof is just a “logical explanation.” A direct proof is some- times referred to as an argument by deduction. This is simply an argument in terms of logic. Often written in English with words such as “if ... then,” it could also be written with logic notation such as “P ) Q.” Even if we don’t wish to use symbolic logic notation, we can still take advantage of fundamental theorems of logic to structure our arguments. For example, if we want to prove that P and Q are equivalent, we can first prove P ) Q and then prove Q ) P. In some domains, proofs are essentially a series of state changes from a start state to an end state. Formal predicate logic can be viewed in this way, with the vari- ous “rules of logic” being used to make the changes from one formula or combining a couple of formulas to make a new formula on the route to the destination. Sym- bolic manipulations to solve integration problems in introductory calculus classes are similar in spirit, as are high school geometry proofs. 2.6.2 Proof by Contradiction The simplest way to disprove a theorem or statement is to find a counterexample to the theorem. Unfortunately, no number of examples supporting a theorem is sufficient to prove that the theorem is correct. However, there is an approach that is vaguely similar to disproving by counterexample, called Proof by Contradiction. To prove a theorem by contradiction, we first assume that the theorem is false.We then find a logical contradiction stemming from this assumption. If the logic used to find the contradiction is correct, then the only way to resolve the contradiction is to recognize that the assumption that the theorem is false must be incorrect. That is, we conclude that the theorem must be true. Example 2.10 Here is a simple proof by contradiction. Theorem 2.1 There is no largest integer. Proof: Proof by contradiction. Step 1. Contrary assumption: Assume that there is a largest integer. Call it B (for “biggest”). Step 2. Show this assumption leads to a contradiction: Consider C = B +1. C is an integer because it is the sum of two integers. Also, 40 Chap. 2 Mathematical Preliminaries C>B, which means that B is not the largest integer after all. Thus, we have reached a contradiction. The only flaw in our reasoning is the initial assumption that the theorem is false. Thus, we conclude that the theorem is correct. 2 A related proof technique is proving the contrapositive. We can prove that P ) Q by proving (not Q) ) (not P). 2.6.3 Proof by Mathematical Induction Mathematical induction can be used to prove a wide variety of theorems. Induction also provides a useful way to think about algorithm design, because it encourages you to think about solving a problem by building up from simple subproblems. Induction can help to prove that a recursive function produces the correct result.. Understanding recursion is a big step toward understanding induction, and vice versa, since they work by essentially the same process. Within the context of algorithm analysis, one of the most important uses for mathematical induction is as a method to test a hypothesis. As explained in Sec- tion 2.4, when seeking a closed-form solution for a summation or recurrence we might first guess or otherwise acquire evidence that a particular formula is the cor- rect solution. If the formula is indeed correct, it is often an easy matter to prove that fact with an induction proof. Let Thrm be a theorem to prove, and express Thrm in terms of a positive integer parameter n. Mathematical induction states that Thrm is true for any value of parameter n (for n c, where c is some constant) if the following two conditions are true: 1. Base Case: Thrm holds for n = c, and 2. Induction Step: If Thrm holds for n 1, then Thrm holds for n. Proving the base case is usually easy, typically requiring that some small value such as 1 be substituted for n in the theorem and applying simple algebra or logic as necessary to verify the theorem. Proving the induction step is sometimes easy, and sometimes difficult. An alternative formulation of the induction step is known as strong induction. The induction step for strong induction is: 2a. Induction Step: If Thrm holds for all k, c  k 0) && (n < 47), "Input out of range"); long past, prev, curr; // Store temporary values past = prev = curr = 1; // initialize for (int i=3; i<=n; i++) { // Compute next value past = prev; // past holds fibi(i-2) prev = curr; // prev holds fibi(i-1) curr = past + prev; // curr now holds fibi(i) } return curr; } Function Fibi executes the for loop n 2 times. (a) Which version is easier to understand? Why? (b) Explain why Fibr is so much slower than Fibi. 2.12 Write a recursive function to solve a generalization of the Towers of Hanoi problem where each ring may begin on any pole so long as no ring sits on top of a smaller ring. 2.13 Revise the recursive implementation for Towers of Hanoi from Section 2.5 to return the list of moves needed to solve the problem. 2.14 Consider the following function: void foo (double val) { if (val != 0.0) foo(val/2.0); } This function makes progress towards the base case on every recursive call. In theory (that is, if double variables acted like true real numbers), would this function ever terminate for input val a nonzero number? In practice (an actual computer implementation), will it terminate? 2.15 Write a function to print all of the permutations for the elements of an array containing n distinct integer values. 2.16 Write a recursive algorithm to print all of the subsets for the set of the first n positive integers. 2.17 The Largest Common Factor (LCF) for two positive integers n and m is the largest integer that divides both n and m evenly. LCF(n, m) is at least one, and at most m, assuming that n m. Over two thousand years ago, Euclid provided an efficient algorithm based on the observation that, when n mod m 6=0, LCF(n, m) = LCF(m, n mod m). Use this fact to write two algorithms to find the LCF for two positive integers. The first version should compute the value iteratively. The second version should compute the value using recursion. Sec. 2.9 Exercises 51 2.18 Prove by contradiction that the number of primes is infinite. 2.19 (a) Use induction to show that n2 n is always even. (b) Give a direct proof in one or two sentences that n2 n is always even. (c) Show that n3 n is always divisible by three. (d) Is n5 n aways divisible by 5? Explain your answer. 2.20 Prove that p2 is irrational. 2.21 Explain why n Xi=1 i = n Xi=1 (n i + 1) = n1 Xi=0 (n i). 2.22 Prove Equation 2.2 using mathematical induction. 2.23 Prove Equation 2.6 using mathematical induction. 2.24 Prove Equation 2.7 using mathematical induction. 2.25 Find a closed-form solution and prove (using induction) that your solution is correct for the summation n Xi=1 3i. 2.26 Prove that the sum of the first n even numbers is n2 + n (a) by assuming that the sum of the first n odd numbers is n2. (b) by mathematical induction. 2.27 Give a closed-form formula for the summation P n i=a i where a is an integer between 1 and n. 2.28 Prove that Fib(n) < ( 5 3 )n. 2.29 Prove, for n 1, that n Xi=1 i3 = n2(n + 1)2 4 . 2.30 The following theorem is called the Pigeonhole Principle. Theorem 2.10 When n+1pigeons roost in n holes, there must be some hole containing at least two pigeons. (a) Prove the Pigeonhole Principle using proof by contradiction. (b) Prove the Pigeonhole Principle using mathematical induction. 2.31 For this problem, you will consider arrangements of infinite lines in the plane such that three or more lines never intersect at a single point and no two lines are parallel. (a) Give a recurrence relation that expresses the number of regions formed by n lines, and explain why your recurrence is correct. (b) Give the summation that results from expanding your recurrence. 52 Chap. 2 Mathematical Preliminaries (c) Give a closed-form solution for the summation. 2.32 Prove (using induction) that the recurrence T(n)=T(n 1) +n; T(1) = 1 has as its closed-form solution T(n)=n(n + 1)/2. 2.33 Expand the following recurrence to help you find a closed-form solution, and then use induction to prove your answer is correct. T(n)=2T(n 1) + 1 for n>0; T(0) = 0. 2.34 Expand the following recurrence to help you find a closed-form solution, and then use induction to prove your answer is correct. T(n)=T(n 1) + 3n + 1 for n>0; T(0) = 1. 2.35 Assume that an n-bit integer (represented by standard binary notation) takes any value in the range 0 to 2n 1 with equal probability. (a) For each bit position, what is the probability of its value being 1 and what is the probability of its value being 0? (b) What is the average number of “1” bits for an n-bit random number? (c) What is the expected value for the position of the leftmost “1” bit? In other words, how many positions on average must we examine when moving from left to right before encountering a “1” bit? Show the appropriate summation. 2.36 What is the total volume of your body in liters (or, if you prefer, gallons)? 2.37 An art historian has a database of 20,000 full-screen color images. (a) About how much space will this require? How many CDs would be required to store the database? (A CD holds about 600MB of data). Be sure to explain all assumptions you made to derive your answer. (b) Now, assume that you have access to a good image compression tech- nique that can store the images in only 1/10 of the space required for an uncompressed image. Will the entire database fit onto a single CD if the images are compressed? 2.38 How many cubic miles of water flow out of the mouth of the Mississippi River each day? DO NOT look up the answer or any supplemental facts. Be sure to describe all assumptions made in arriving at your answer. 2.39 When buying a home mortgage, you often have the option of paying some money in advance (called “discount points”) to get a lower interest rate. As- sume that you have the choice between two 15-year fixed-rate mortgages: one at 8% with no up-front charge, and the other at 73 4 % with an up-front charge of 1% of the mortgage value. How long would it take to recover the 1% charge when you take the mortgage at the lower rate? As a second, more Sec. 2.9 Exercises 53 precise estimate, how long would it take to recover the charge plus the in- terest you would have received if you had invested the equivalent of the 1% charge in the bank at 5% interest while paying the higher rate? DO NOT use a calculator to help you answer this question. 2.40 When you build a new house, you sometimes get a “construction loan” which is a temporary line of credit out of which you pay construction costs as they occur. At the end of the construction period, you then replace the construc- tion loan with a regular mortgage on the house. During the construction loan, you only pay each month for the interest charged against the actual amount borrowed so far. Assume that your house construction project starts at the beginning of April, and is complete at the end of six months. Assume that the total construction cost will be $300,000 with the costs occurring at the be- ginning of each month in $50,000 increments. The construction loan charges 6% interest. Estimate the total interest payments that must be paid over the life of the construction loan. 2.41 Here are some questions that test your working knowledge of how fast com- puters operate. Is disk drive access time normally measured in milliseconds (thousandths of a second) or microseconds (millionths of a second)? Does your RAM memory access a word in more or less than one microsecond? How many instructions can your CPU execute in one year if the machine is left running at full speed all the time? DO NOT use paper or a calculator to derive your answers. 2.42 Does your home contain enough books to total one million pages? How many total pages are stored in your school library building? Explain how you got your answer. 2.43 How many words are in this book? Explain how you got your answer. 2.44 How many hours are one million seconds? How many days? Answer these questions doing all arithmetic in your head. Explain how you got your an- swer. 2.45 How many cities and towns are there in the United States? Explain how you got your answer. 2.46 How many steps would it take to walk from Boston to San Francisco? Ex- plain how you got your answer. 2.47 A man begins a car trip to visit his in-laws. The total distance is 60 miles, and he starts off at a speed of 60 miles per hour. After driving exactly 1 mile, he loses some of his enthusiasm for the journey, and (instantaneously) slows down to 59 miles per hour. After traveling another mile, he again slows to 58 miles per hour. This continues, progressively slowing by 1 mile per hour for each mile traveled until the trip is complete. (a) How long does it take the man to reach his in-laws? 54 Chap. 2 Mathematical Preliminaries (b) How long would the trip take in the continuous case where the speed smoothly diminishes with the distance yet to travel? 3 Algorithm Analysis How long will it take to process the company payroll once we complete our planned merger? Should I buy a new payroll program from vendor X or vendor Y? If a particular program is slow, is it badly implemented or is it solving a hard problem? Questions like these ask us to consider the difficulty of a problem, or the relative efficiency of two or more approaches to solving a problem. This chapter introduces the motivation, basic notation, and fundamental tech- niques of algorithm analysis. We focus on a methodology known as asymptotic algorithm analysis, or simply asymptotic analysis. Asymptotic analysis attempts to estimate the resource consumption of an algorithm. It allows us to compare the relative costs of two or more algorithms for solving the same problem. Asymptotic analysis also gives algorithm designers a tool for estimating whether a proposed solution is likely to meet the resource constraints for a problem before they imple- ment an actual program. After reading this chapter, you should understand • the concept of a growth rate, the rate at which the cost of an algorithm grows as the size of its input grows; • the concept of upper and lower bounds for a growth rate, and how to estimate these bounds for a simple program, algorithm, or problem; and • the difference between the cost of an algorithm (or program) and the cost of a problem. The chapter concludes with a brief discussion of the practical difficulties encoun- tered when empirically measuring the cost of a program, and some principles for code tuning to improve program efficiency. 3.1 Introduction How do you compare two algorithms for solving some problem in terms of effi- ciency? We could implement both algorithms as computer programs and then run 55 56 Chap. 3 Algorithm Analysis them on a suitable range of inputs, measuring how much of the resources in ques- tion each program uses. This approach is often unsatisfactory for four reasons. First, there is the effort involved in programming and testing two algorithms when at best you want to keep only one. Second, when empirically comparing two al- gorithms there is always the chance that one of the programs was “better written” than the other, and therefor the relative qualities of the underlying algorithms are not truly represented by their implementations. This can easily occur when the programmer has a bias regarding the algorithms. Third, the choice of empirical test cases might unfairly favor one algorithm. Fourth, you could find that even the better of the two algorithms does not fall within your resource budget. In that case you must begin the entire process again with yet another program implementing a new algorithm. But, how would you know if any algorithm can meet the resource budget? Perhaps the problem is simply too difficult for any implementation to be within budget. These problems can often be avoided by using asymptotic analysis. Asymp- totic analysis measures the efficiency of an algorithm, or its implementation as a program, as the input size becomes large. It is actually an estimating technique and does not tell us anything about the relative merits of two programs where one is always “slightly faster” than the other. However, asymptotic analysis has proved useful to computer scientists who must determine if a particular algorithm is worth considering for implementation. The critical resource for a program is most often its running time. However, you cannot pay attention to running time alone. You must also be concerned with other factors such as the space required to run the program (both main memory and disk space). Typically you will analyze the time required for an algorithm (or the instantiation of an algorithm in the form of a program), and the space required for a data structure. Many factors affect the running time of a program. Some relate to the environ- ment in which the program is compiled and run. Such factors include the speed of the computer’s CPU, bus, and peripheral hardware. Competition with other users for the computer’s (or the network’s) resources can make a program slow to a crawl. The programming language and the quality of code generated by a particular com- piler can have a significant effect. The “coding efficiency” of the programmer who converts the algorithm to a program can have a tremendous impact as well. If you need to get a program working within time and space constraints on a particular computer, all of these factors can be relevant. Yet, none of these factors address the differences between two algorithms or data structures. To be fair, pro- grams derived from two algorithms for solving the same problem should both be compiled with the same compiler and run on the same computer under the same conditions. As much as possible, the same amount of care should be taken in the programming effort devoted to each program to make the implementations “equally Sec. 3.1 Introduction 57 efficient.” In this sense, all of the factors mentioned above should cancel out of the comparison because they apply to both algorithms equally. If you truly wish to understand the running time of an algorithm, there are other factors that are more appropriate to consider than machine speed, programming language, compiler, and so forth. Ideally we would measure the running time of the algorithm under standard benchmark conditions. However, we have no way to calculate the running time reliably other than to run an implementation of the algorithm on some computer. The only alternative is to use some other measure as a surrogate for running time. Of primary consideration when estimating an algorithm’s performance is the number of basic operations required by the algorithm to process an input of a certain size. The terms “basic operations” and “size” are both rather vague and depend on the algorithm being analyzed. Size is often the number of inputs pro- cessed. For example, when comparing sorting algorithms, the size of the problem is typically measured by the number of records to be sorted. A basic operation must have the property that its time to complete does not depend on the particular values of its operands. Adding or comparing two integer variables are examples of basic operations in most programming languages. Summing the contents of an array containing n integers is not, because the cost depends on the value of n (i.e., the size of the input). Example 3.1 Consider a simple algorithm to solve the problem of finding the largest value in an array of n integers. The algorithm looks at each integer in turn, saving the position of the largest value seen so far. This algorithm is called the largest-value sequential search and is illustrated by the following function: // Return position of largest value in "A" of size "n" int largest(int A[], int n) { int currlarge = 0; // Holds largest element position for (int i=1; i5, the algorithm with running time T(n)=2n2 is already much slower. This is despite the fact that 10n has a greater constant factor than 2n2. Comparing the two curves marked 20n and 2n2 shows that changing the constant factor for one of the equations only shifts the point at which the two curves cross. For n>10, the algorithm with cost T(n)=2n2 is slower than the algorithm with cost T(n) = 20n. This graph also shows that the equation T(n)=5n log n grows somewhat more quickly than both T(n) = 10n and T(n) = 20n, but not nearly so quickly as the equation T(n)=2n2. For constants a, b > 1, na grows faster than either logb n or log nb. Finally, algorithms with cost T(n)=2n or T(n)=n! are prohibitively expensive for even modest values of n. Note that for constants a, b 1, an grows faster than nb. We can get some further insight into relative growth rates for various algorithms from Figure 3.2. Most of the growth rates that appear in typical algorithms are shown, along with some representative input sizes. Once again, we see that the growth rate has a tremendous effect on the resources consumed by an algorithm. Sec. 3.2 Best, Worst, and Average Cases 61 3.2 Best, Worst, and Average Cases Consider the problem of finding the factorial of n. For this problem, there is only one input of a given “size” (that is, there is only a single instance for each size of n). Now consider our largest-value sequential search algorithm of Example 3.1, which always examines every array value. This algorithm works on many inputs of a given size n. That is, there are many possible arrays of any given size. However, no matter what array of size n that the algorithm looks at, its cost will always be the same in that it always looks at every element in the array one time. For some algorithms, different inputs of a given size require different amounts of time. For example, consider the problem of searching an array containing n integers to find the one with a particular value K (assume that K appears exactly once in the array). The sequential search algorithm begins at the first position in the array and looks at each value in turn until K is found. Once K is found, the algorithm stops. This is different from the largest-value sequential search algorithm of Example 3.1, which always examines every array value. There is a wide range of possible running times for the sequential search alg- orithm. The first integer in the array could have value K, and so only one integer is examined. In this case the running time is short. This is the best case for this algorithm, because it is not possible for sequential search to look at less than one value. Alternatively, if the last position in the array contains K, then the running time is relatively long, because the algorithm must examine n values. This is the worst case for this algorithm, because sequential search never looks at more than n values. If we implement sequential search as a program and run it many times on many different arrays of size n, or search for many different values of K within the same array, we expect the algorithm on average to go halfway through the array before finding the value we seek. On average, the algorithm examines about n/2 values. We call this the average case for this algorithm. When analyzing an algorithm, should we study the best, worst, or average case? Normally we are not interested in the best case, because this might happen only rarely and generally is too optimistic for a fair characterization of the algorithm’s running time. In other words, analysis based on the best case is not likely to be representative of the behavior of the algorithm. However, there are rare instances where a best-case analysis is useful — in particular, when the best case has high probability of occurring. In Chapter 7 you will see some examples where taking advantage of the best-case running time for one sorting algorithm makes a second more efficient. How about the worst case? The advantage to analyzing the worst case is that you know for certain that the algorithm must perform at least that well. This is es- pecially important for real-time applications, such as for the computers that monitor an air traffic control system. Here, it would not be acceptable to use an algorithm 62 Chap. 3 Algorithm Analysis that can handle n airplanes quickly enough most of the time, but which fails to perform quickly enough when all n airplanes are coming from the same direction. For other applications — particularly when we wish to aggregate the cost of running the program many times on many different inputs — worst-case analy- sis might not be a representative measure of the algorithm’s performance. Often we prefer to know the average-case running time. This means that we would like to know the typical behavior of the algorithm on inputs of size n. Unfortunately, average-case analysis is not always possible. Average-case analysis first requires that we understand how the actual inputs to the program (and their costs) are dis- tributed with respect to the set of all possible inputs to the program. For example, it was stated previously that the sequential search algorithm on average examines half of the array values. This is only true if the element with value K is equally likely to appear in any position in the array. If this assumption is not correct, then the algorithm does not necessarily examine half of the array values in the average case. See Section 9.2 for further discussion regarding the effects of data distribution on the sequential search algorithm. The characteristics of a data distribution have a significant effect on many search algorithms, such as those based on hashing (Section 9.4) and search trees (e.g., see Section 5.4). Incorrect assumptions about data distribution can have dis- astrous consequences on a program’s space or time performance. Unusual data distributions can also be used to advantage, as shown in Section 9.2. In summary, for real-time applications we are likely to prefer a worst-case anal- ysis of an algorithm. Otherwise, we often desire an average-case analysis if we know enough about the distribution of our input to compute the average case. If not, then we must resort to worst-case analysis. 3.3 A Faster Computer, or a Faster Algorithm? Imagine that you have a problem to solve, and you know of an algorithm whose running time is proportional to n2. Unfortunately, the resulting program takes ten times too long to run. If you replace your current computer with a new one that is ten times faster, will the n2 algorithm become acceptable? If the problem size remains the same, then perhaps the faster computer will allow you to get your work done quickly enough even with an algorithm having a high growth rate. But a funny thing happens to most people who get a faster computer. They don’t run the same problem faster. They run a bigger problem! Say that on your old computer you were content to sort 10,000 records because that could be done by the computer during your lunch break. On your new computer you might hope to sort 100,000 records in the same time. You won’t be back from lunch any sooner, so you are better off solving a larger problem. And because the new machine is ten times faster, you would like to sort ten times as many records. Sec. 3.3 A Faster Computer, or a Faster Algorithm? 63 f(n) n n0 Change n0/n 10n 1000 10, 000 n0 = 10n 10 20n 500 5000 n0 = 10n 10 5n log n 250 1842 p10n < n0 < 10n 7.37 2n2 70 223 n0 = p10n 3.16 2n 13 16 n0 = n + 3 Figure 3.3 The increase in problem size that can be run in a fixed period of time on a computer that is ten times faster. The first column lists the right-hand sides for each of five growth rate equations from Figure 3.1. For the purpose of this example, arbitrarily assume that the old machine can run 10,000 basic operations in one hour. The second column shows the maximum value for n that can be run in 10,000 basic operations on the old machine. The third column shows the value for n0, the new maximum size for the problem that can be run in the same time on the new machine that is ten times faster. Variable n0 is the greatest size for the problem that can run in 100,000 basic operations. The fourth column shows how the size of n changed to become n0 on the new machine. The fifth column shows the increase in the problem size as the ratio of n0 to n. If your algorithm’s growth rate is linear (i.e., if the equation that describes the running time on input size n is T(n)=cn for some constant c), then 100,000 records on the new machine will be sorted in the same time as 10,000 records on the old machine. If the algorithm’s growth rate is greater than cn, such as c1n2, then you will not be able to do a problem ten times the size in the same amount of time on a machine that is ten times faster. How much larger a problem can be solved in a given amount of time by a faster computer? Assume that the new machine is ten times faster than the old. Say that the old machine could solve a problem of size n in an hour. What is the largest problem that the new machine can solve in one hour? Figure 3.3 shows how large a problem can be solved on the two machines for five of the running-time functions from Figure 3.1. This table illustrates many important points. The first two equations are both linear; only the value of the constant factor has changed. In both cases, the machine that is ten times faster gives an increase in problem size by a factor of ten. In other words, while the value of the constant does affect the absolute size of the problem that can be solved in a fixed amount of time, it does not affect the improvement in problem size (as a proportion to the original size) gained by a faster computer. This relationship holds true regardless of the algorithm’s growth rate: Constant factors never affect the relative improvement gained by a faster computer. An algorithm with time equation T(n)=2n2 does not receive nearly as great an improvement from the faster machine as an algorithm with linear growth rate. Instead of an improvement by a factor of ten, the improvement is only the square 64 Chap. 3 Algorithm Analysis root of that: p10 ⇡ 3.16. Thus, the algorithm with higher growth rate not only solves a smaller problem in a given time in the first place, it also receives less of a speedup from a faster computer. As computers get ever faster, the disparity in problem sizes becomes ever greater. The algorithm with growth rate T(n)=5n log n improves by a greater amount than the one with quadratic growth rate, but not by as great an amount as the algo- rithms with linear growth rates. Note that something special happens in the case of the algorithm whose running time grows exponentially. In Figure 3.1, the curve for the algorithm whose time is proportional to 2n goes up very quickly. In Figure 3.3, the increase in problem size on the machine ten times as fast is shown to be about n +3(to be precise, it is n + log2 10). The increase in problem size for an algorithm with exponential growth rate is by a constant addition, not by a multiplicative factor. Because the old value of n was 13, the new problem size is 16. If next year you buy another computer ten times faster yet, then the new computer (100 times faster than the original computer) will only run a problem of size 19. If you had a second program whose growth rate is 2n and for which the original computer could run a problem of size 1000 in an hour, than a machine ten times faster can run a problem only of size 1003 in an hour! Thus, an exponential growth rate is radically different than the other growth rates shown in Figure 3.3. The significance of this difference is explored in Chapter 17. Instead of buying a faster computer, consider what happens if you replace an algorithm whose running time is proportional to n2 with a new algorithm whose running time is proportional to n log n. In the graph of Figure 3.1, a fixed amount of time would appear as a horizontal line. If the line for the amount of time available to solve your problem is above the point at which the curves for the two growth rates in question meet, then the algorithm whose running time grows less quickly is faster. An algorithm with running time T(n)=n2 requires 1024 ⇥ 1024 = 1, 048, 576 time steps for an input of size n = 1024. An algorithm with running time T(n)=n log n requires 1024 ⇥ 10 = 10, 240 time steps for an input of size n = 1024, which is an improvement of much more than a factor of ten when compared to the algorithm with running time T(n)=n2. Because n2 > 10n log n whenever n>58, if the typical problem size is larger than 58 for this example, then you would be much better off changing algorithms instead of buying a computer ten times faster. Furthermore, when you do buy a faster computer, an algorithm with a slower growth rate provides a greater benefit in terms of larger problem size that can run in a certain time on the new computer. Sec. 3.4 Asymptotic Analysis 65 3.4 Asymptotic Analysis Despite the larger constant for the curve labeled 10n in Figure 3.1, 2n2 crosses it at the relatively small value of n =5. What if we double the value of the constant in front of the linear equation? As shown in the graph, 20n is surpassed by 2n2 once n = 10. The additional factor of two for the linear growth rate does not much matter. It only doubles the x-coordinate for the intersection point. In general, changes to a constant factor in either equation only shift where the two curves cross, not whether the two curves cross. When you buy a faster computer or a faster compiler, the new problem size that can be run in a given amount of time for a given growth rate is larger by the same factor, regardless of the constant on the running-time equation. The time curves for two algorithms with different growth rates still cross, regardless of their running-time equation constants. For these reasons, we usually ignore the con- stants when we want an estimate of the growth rate for the running time or other resource requirements of an algorithm. This simplifies the analysis and keeps us thinking about the most important aspect: the growth rate. This is called asymp- totic algorithm analysis. To be precise, asymptotic analysis refers to the study of an algorithm as the input size “gets big” or reaches a limit (in the calculus sense). However, it has proved to be so useful to ignore all constant factors that asymptotic analysis is used for most algorithm comparisons. It is not always reasonable to ignore the constants. When comparing algorithms meant to run on small values of n, the constant can have a large effect. For exam- ple, if the problem is to sort a collection of exactly five records, then an algorithm designed for sorting thousands of records is probably not appropriate, even if its asymptotic analysis indicates good performance. There are rare cases where the constants for two algorithms under comparison can differ by a factor of 1000 or more, making the one with lower growth rate impractical for most purposes due to its large constant. Asymptotic analysis is a form of “back of the envelope” esti- mation for algorithm resource consumption. It provides a simplified model of the running time or other resource needs of an algorithm. This simplification usually helps you understand the behavior of your algorithms. Just be aware of the limi- tations to asymptotic analysis in the rare situation where the constant is important. 3.4.1 Upper Bounds Several terms are used to describe the running-time equation for an algorithm. These terms — and their associated symbols — indicate precisely what aspect of the algorithm’s behavior is being described. One is the upper bound for the growth of the algorithm’s running time. It indicates the upper or highest growth rate that the algorithm can have. 66 Chap. 3 Algorithm Analysis Because the phrase “has an upper bound to its growth rate of f(n)” is long and often used when discussing algorithms, we adopt a special notation, called big-Oh notation. If the upper bound for an algorithm’s growth rate (for, say, the worst case) is f(n), then we would write that this algorithm is “in the set O(f(n))in the worst case” (or just “in O(f(n))in the worst case”). For example, if n2 grows as fast as T(n) (the running time of our algorithm) for the worst-case input, we would say the algorithm is “in O(n2) in the worst case.” The following is a precise definition for an upper bound. T(n) represents the true running time of the algorithm. f(n) is some expression for the upper bound. For T(n) a non-negatively valued function, T(n) is in set O(f(n)) if there exist two positive constants c and n0 such that T(n)  cf(n) for all n>n0. Constant n0 is the smallest value of n for which the claim of an upper bound holds true. Usually n0 is small, such as 1, but does not need to be. You must also be able to pick some constant c, but it is irrelevant what the value for c actually is. In other words, the definition says that for all inputs of the type in question (such as the worst case for all inputs of size n) that are large enough (i.e., n>n0), the algorithm always executes in less than cf(n) steps for some constant c. Example 3.4 Consider the sequential search algorithm for finding a spec- ified value in an array of integers. If visiting and examining one value in the array requires cs steps where cs is a positive number, and if the value we search for has equal probability of appearing in any position in the ar- ray, then in the average case T(n)=csn/2. For all values of n>1, csn/2  csn. Therefore, by the definition, T(n) is in O(n) for n0 =1and c = cs. Example 3.5 For a particular algorithm, T(n)=c1n2 + c2n in the av- erage case where c1 and c2 are positive numbers. Then, c1n2 + c2n c1n2 + c2n2  (c1 + c2)n2 for all n>1. So, T(n)  cn2 for c = c1 + c2, and n0 =1. Therefore, T(n) is in O(n2) by the second definition. Example 3.6 Assigning the value from the first position of an array to a variable takes constant time regardless of the size of the array. Thus, T(n)=c (for the best, worst, and average cases). We could say in this case that T(n) is in O(c). However, it is traditional to say that an algorithm whose running time has a constant upper bound is in O(1). Sec. 3.4 Asymptotic Analysis 67 If someone asked you out of the blue “Who is the best?” your natural reaction should be to reply “Best at what?” In the same way, if you are asked “What is the growth rate of this algorithm,” you would need to ask “When? Best case? Average case? Or worst case?” Some algorithms have the same behavior no matter which input instance they receive. An example is finding the maximum in an array of integers. But for many algorithms, it makes a big difference, such as when searching an unsorted array for a particular value. So any statement about the upper bound of an algorithm must be in the context of some class of inputs of size n. We measure this upper bound nearly always on the best-case, average-case, or worst-case inputs. Thus, we cannot say, “this algorithm has an upper bound to its growth rate of n2.” We must say something like, “this algorithm has an upper bound to its growth rate of n2 in the average case.” Knowing that something is in O(f(n)) says only how bad things can be. Per- haps things are not nearly so bad. Because sequential search is in O(n) in the worst case, it is also true to say that sequential search is in O(n2). But sequential search is practical for large n, in a way that is not true for some other algorithms in O(n2). We always seek to define the running time of an algorithm with the tightest (low- est) possible upper bound. Thus, we prefer to say that sequential search is in O(n). This also explains why the phrase “is in O(f(n))” or the notation “2 O(f(n))” is used instead of “is O(f(n))” or “=O(f(n)).” There is no strict equality to the use of big-Oh notation. O(n) is in O(n2), but O(n2) is not in O(n). 3.4.2 Lower Bounds Big-Oh notation describes an upper bound. In other words, big-Oh notation states a claim about the greatest amount of some resource (usually time) that is required by an algorithm for some class of inputs of size n (typically the worst such input, the average of all possible inputs, or the best such input). Similar notation is used to describe the least amount of a resource that an alg- orithm needs for some class of input. Like big-Oh notation, this is a measure of the algorithm’s growth rate. Like big-Oh notation, it works for any resource, but we most often measure the least amount of time required. And again, like big-Oh no- tation, we are measuring the resource required for some particular class of inputs: the worst-, average-, or best-case input of size n. The lower bound for an algorithm (or a problem, as explained later) is denoted by the symbol ⌦, pronounced “big-Omega” or just “Omega.” The following defi- nition for ⌦ is symmetric with the definition of big-Oh. For T(n) a non-negatively valued function, T(n) is in set ⌦(g(n)) if there exist two positive constants c and n0 such that T(n) cg(n) for all n>n0.1 1An alternate (non-equivalent) definition for ⌦ is 68 Chap. 3 Algorithm Analysis Example 3.7 Assume T(n)=c1n2 + c2n for c1 and c2 > 0. Then, c1n2 + c2n c1n2 for all n>1. So, T(n) cn2 for c = c1 and n0 =1. Therefore, T(n) is in ⌦(n2) by the definition. It is also true that the equation of Example 3.7 is in ⌦(n). However, as with big-Oh notation, we wish to get the “tightest” (for ⌦ notation, the largest) bound possible. Thus, we prefer to say that this running time is in ⌦(n2). Recall the sequential search algorithm to find a value K within an array of integers. In the average and worst cases this algorithm is in ⌦(n), because in both the average and worst cases we must examine at least cn values (where c is 1/2 in the average case and 1 in the worst case). 3.4.3 ⇥ Notation The definitions for big-Oh and ⌦ give us ways to describe the upper bound for an algorithm (if we can find an equation for the maximum cost of a particular class of inputs of size n) and the lower bound for an algorithm (if we can find an equation for the minimum cost for a particular class of inputs of size n). When the upper and lower bounds are the same within a constant factor, we indicate this by using ⇥ (big-Theta) notation. An algorithm is said to be ⇥(h(n)) if it is in O(h(n)) and T(n) is in the set ⌦(g(n)) if there exists a positive constant c such that T(n) cg(n) for an infinite number of values for n. This definition says that for an “interesting” number of cases, the algorithm takes at least cg(n) time. Note that this definition is not symmetric with the definition of big-Oh. For g(n) to be a lower bound, this definition does not require that T(n) cg(n) for all values of n greater than some constant. It only requires that this happen often enough, in particular that it happen for an infinite number of values for n. Motivation for this alternate definition can be found in the following example. Assume a particular algorithm has the following behavior: T(n)= ⇢ n for all odd n 1 n2/100 for all even n 0 From this definition, n2/100 1 100 n2 for all even n 0. So, T(n) cn2 for an infinite number of values of n (i.e., for all even n) for c =1/100. Therefore, T(n) is in ⌦(n2) by the definition. For this equation for T(n), it is true that all inputs of size n take at least cn time. But an infinite number of inputs of size n take cn2 time, so we would like to say that the algorithm is in ⌦(n2). Unfortunately, using our first definition will yield a lower bound of ⌦(n) because it is not possible to pick constants c and n0 such that T(n) cn2 for all n>n0. The alternative definition does result in a lower bound of ⌦(n2) for this algorithm, which seems to fit common sense more closely. Fortu- nately, few real algorithms or computer programs display the pathological behavior of this example. Our first definition for ⌦ generally yields the expected result. As you can see from this discussion, asymptotic bounds notation is not a law of nature. It is merely a powerful modeling tool used to describe the behavior of algorithms. Sec. 3.4 Asymptotic Analysis 69 it is in ⌦(h(n)). Note that we drop the word “in” for ⇥ notation, because there is a strict equality for two equations with the same ⇥. In other words, if f(n) is ⇥(g(n)), then g(n) is ⇥(f(n)). Because the sequential search algorithm is both in O(n) and in ⌦(n) in the average case, we say it is ⇥(n) in the average case. Given an algebraic equation describing the time requirement for an algorithm, the upper and lower bounds always meet. That is because in some sense we have a perfect analysis for the algorithm, embodied by the running-time equation. For many algorithms (or their instantiations as programs), it is easy to come up with the equation that defines their runtime behavior. Most algorithms presented in this book are well understood and we can almost always give a ⇥ analysis for them. However, Chapter 17 discusses a whole class of algorithms for which we have no ⇥ analysis, just some unsatisfying big-Oh and ⌦ analyses. Exercise 3.14 presents a short, simple program fragment for which nobody currently knows the true upper or lower bounds. While some textbooks and programmers will casually say that an algorithm is “order of” or “big-Oh” of some cost function, it is generally better to use ⇥ notation rather than big-Oh notation whenever we have sufficient knowledge about an alg- orithm to be sure that the upper and lower bounds indeed match. Throughout this book, ⇥ notation will be used in preference to big-Oh notation whenever our state of knowledge makes that possible. Limitations on our ability to analyze certain algorithms may require use of big-Oh or ⌦ notations. In rare occasions when the discussion is explicitly about the upper or lower bound of a problem or algorithm, the corresponding notation will be used in preference to ⇥ notation. 3.4.4 Simplifying Rules Once you determine the running-time equation for an algorithm, it really is a simple matter to derive the big-Oh, ⌦, and ⇥ expressions from the equation. You do not need to resort to the formal definitions of asymptotic analysis. Instead, you can use the following rules to determine the simplest form. 1. If f(n) is in O(g(n)) and g(n) is in O(h(n)), then f(n) is in O(h(n)). 2. If f(n) is in O(kg(n)) for any constant k>0, then f(n) is in O(g(n)). 3. If f1(n) is in O(g1(n)) and f2(n) is in O(g2(n)), then f1(n)+f2(n) is in O(max(g1(n),g2(n))). 4. If f1(n) is in O(g1(n)) and f2(n) is in O(g2(n)), then f1(n)f2(n) is in O(g1(n)g2(n)). The first rule says that if some function g(n) is an upper bound for your cost function, then any upper bound for g(n) is also an upper bound for your cost func- tion. A similar property holds true for ⌦ notation: If g(n) is a lower bound for your 70 Chap. 3 Algorithm Analysis cost function, then any lower bound for g(n) is also a lower bound for your cost function. Likewise for ⇥ notation. The significance of rule (2) is that you can ignore any multiplicative constants in your equations when using big-Oh notation. This rule also holds true for ⌦ and ⇥ notations. Rule (3) says that given two parts of a program run in sequence (whether two statements or two sections of code), you need consider only the more expensive part. This rule applies to ⌦ and ⇥ notations as well: For both, you need consider only the more expensive part. Rule (4) is used to analyze simple loops in programs. If some action is repeated some number of times, and each repetition has the same cost, then the total cost is the cost of the action multiplied by the number of times that the action takes place. This rule applies to ⌦ and ⇥ notations as well. Taking the first three rules collectively, you can ignore all constants and all lower-order terms to determine the asymptotic growth rate for any cost function. The advantages and dangers of ignoring constants were discussed near the begin- ning of this section. Ignoring lower-order terms is reasonable when performing an asymptotic analysis. The higher-order terms soon swamp the lower-order terms in their contribution to the total cost as n becomes larger. Thus, if T(n)=3n4 +5n2, then T(n) is in O(n4). The n2 term contributes relatively little to the total cost for large n. Throughout the rest of this book, these simplifying rules are used when dis- cussing the cost for a program or algorithm. 3.4.5 Classifying Functions Given functions f(n) and g(n) whose growth rates are expressed as algebraic equa- tions, we might like to determine if one grows faster than the other. The best way to do this is to take the limit of the two functions as n grows towards infinity, limn!1 f(n) g(n) . If the limit goes to 1, then f(n) is in ⌦(g(n)) because f(n) grows faster. If the limit goes to zero, then f(n) is in O(g(n)) because g(n) grows faster. If the limit goes to some constant other than zero, then f(n)=⇥(g(n)) because both grow at the same rate. Example 3.8 If f(n)=2n log n and g(n)=n2,isf(n) in O(g(n)), ⌦(g(n)), or ⇥(g(n))? Because n2 2n log n = n 2 log n, Sec. 3.5 Calculating the Running Time for a Program 71 we easily see that limn!1 n2 2n log n = 1 because n grows faster than 2 log n. Thus, n2 is in ⌦(2n log n). 3.5 Calculating the Running Time for a Program This section presents the analysis for several simple code fragments. Example 3.9 We begin with an analysis of a simple assignment to an integer variable. a = b; Because the assignment statement takes constant time, it is ⇥(1). Example 3.10 Consider a simple for loop. sum = 0; for (i=1; i<=n; i++) sum += n; The first line is ⇥(1). The for loop is repeated n times. The third line takes constant time so, by simplifying rule (4) of Section 3.4.4, the total cost for executing the two lines making up the for loop is ⇥(n). By rule (3), the cost of the entire code fragment is also ⇥(n). Example 3.11 We now analyze a code fragment with several for loops, some of which are nested. sum = 0; for (i=1; i<=n; i++) // First for loop for (j=1; j<=i; j++) // is a double loop sum++; for (k=0; k1; T(1) = 0. We know from Examples 2.8 and 2.13 that the closed-form solution for this recur- rence relation is ⇥(n). 74 Chap. 3 Algorithm Analysis Key Position 0 2 3 4 5 6 7 8 26 29 36 10 11 12 13 14 15 11 13 21 41 45 51 54 1 56 65 72 77 9 8340 Figure 3.4 An illustration of binary search on a sorted array of 16 positions. Consider a search for the position with value K = 45. Binary search first checks the value at position 7. Because 41 K, the desired value (if it exists) must be between positions 7 and 11. Position 9 is checked next. Again, its value is too great. The final search is at position 8, which contains the desired value. Thus, function binary returns position 8. Alternatively, if K were 44, then the same series of record accesses would be made. After checking position 8, binary would return a value of n, indicating that the search is unsuccessful. The final example of algorithm analysis for this section will compare two algo- rithms for performing search in an array. Earlier, we determined that the running time for sequential search on an array where the search value K is equally likely to appear in any location is ⇥(n) in both the average and worst cases. We would like to compare this running time to that required to perform a binary search on an array whose values are stored in order from lowest to highest. Binary search begins by examining the value in the middle position of the ar- ray; call this position mid and the corresponding value kmid. If kmid = K, then processing can stop immediately. This is unlikely to be the case, however. Fortu- nately, knowing the middle value provides useful information that can help guide the search process. In particular, if kmid >K, then you know that the value K cannot appear in the array at any position greater than mid. Thus, you can elim- inate future search in the upper half of the array. Conversely, if kmid 1; T(1) = 1. Sec. 3.5 Calculating the Running Time for a Program 75 // Return the position of an element in sorted array "A" of // size "n" with value "K". If "K" is not in "A", return // the value "n". int binary(int A[], int n, int K) { int l = -1; int r = n; // l and r are beyond array bounds while (l+1 != r) { // Stop when l and r meet int i = (l+r)/2; // Check middle of remaining subarray if (K < A[i]) r = i; // In left half if (K == A[i]) return i; // Found it if (K > A[i]) l = i; // In right half } return n; // Search value not in A } Figure 3.5 Implementation for binary search. If we expand the recurrence, we find that we can do so only log n times before we reach the base case, and each expansion adds one to the cost. Thus, the closed- form solution for the recurrence is T(n) = log n. Function binary is designed to find the (single) occurrence of K and return its position. A special value is returned if K does not appear in the array. This algorithm can be modified to implement variations such as returning the position of the first occurrence of K in the array if multiple occurrences are allowed, and returning the position of the greatest value less than K when K is not in the array. Comparing sequential search to binary search, we see that as n grows, the ⇥(n) running time for sequential search in the average and worst cases quickly becomes much greater than the ⇥(log n) running time for binary search. Taken in isolation, binary search appears to be much more efficient than sequential search. This is despite the fact that the constant factor for binary search is greater than that for sequential search, because the calculation for the next search position in binary search is more expensive than just incrementing the current position, as sequential search does. Note however that the running time for sequential search will be roughly the same regardless of whether or not the array values are stored in order. In contrast, binary search requires that the array values be ordered from lowest to highest. De- pending on the context in which binary search is to be used, this requirement for a sorted array could be detrimental to the running time of a complete program, be- cause maintaining the values in sorted order requires to greater cost when inserting new elements into the array. This is an example of a tradeoff between the advan- tage of binary search during search and the disadvantage related to maintaining a sorted array. Only in the context of the complete problem to be solved can we know whether the advantage outweighs the disadvantage. 76 Chap. 3 Algorithm Analysis 3.6 Analyzing Problems You most often use the techniques of “algorithm” analysis to analyze an algorithm, or the instantiation of an algorithm as a program. You can also use these same techniques to analyze the cost of a problem. It should make sense to you to say that the upper bound for a problem cannot be worse than the upper bound for the best algorithm that we know for that problem. But what does it mean to give a lower bound for a problem? Consider a graph of cost over all inputs of a given size n for some algorithm for a given problem. Define A to be the collection of all algorithms that solve the problem (theoretically, there are an infinite number of such algorithms). Now, consider the collection of all the graphs for all of the (infinitely many) algorithms in A. The worst case lower bound is the least of all the highest points on all the graphs. It is much easier to show that an algorithm (or program) is in ⌦(f(n)) than it is to show that a problem is in ⌦(f(n)). For a problem to be in ⌦(f(n)) means that every algorithm that solves the problem is in ⌦(f(n)), even algorithms that we have not thought of! So far all of our examples of algorithm analysis give “obvious” results, with big-Oh always matching ⌦. To understand how big-Oh, ⌦, and ⇥ notations are properly used to describe our understanding of a problem or an algorithm, it is best to consider an example where you do not already know a lot about the problem. Let us look ahead to analyzing the problem of sorting to see how this process works. What is the least possible cost for any sorting algorithm in the worst case? The algorithm must at least look at every element in the input, just to determine that the input is truly sorted. Thus, any sorting algorithm must take at least cn time. For many problems, this observation that each of the n inputs must be looked at leads to an easy ⌦(n) lower bound. In your previous study of computer science, you have probably seen an example of a sorting algorithm whose running time is in O(n2) in the worst case. The simple Bubble Sort and Insertion Sort algorithms typically given as examples in a first year programming course have worst case running times in O(n2). Thus, the problem of sorting can be said to have an upper bound in O(n2). How do we close the gap between ⌦(n) and O(n2)? Can there be a better sorting algorithm? If you can think of no algorithm whose worst-case growth rate is better than O(n2), and if you have discovered no analysis technique to show that the least cost for the problem of sorting in the worst case is greater than ⌦(n), then you cannot know for sure whether or not there is a better algorithm. Chapter 7 presents sorting algorithms whose running time is in O(n log n) for the worst case. This greatly narrows the gap. With this new knowledge, we now have a lower bound in ⌦(n) and an upper bound in O(n log n). Should we search Sec. 3.7 Common Misunderstandings 77 for a faster algorithm? Many have tried, without success. Fortunately (or perhaps unfortunately?), Chapter 7 also includes a proof that any sorting algorithm must have running time in ⌦(n log n) in the worst case.2 This proof is one of the most important results in the field of algorithm analysis, and it means that no sorting algorithm can possibly run faster than cn log n for the worst-case input of size n. Thus, we can conclude that the problem of sorting is ⇥(n log n) in the worst case, because the upper and lower bounds have met. Knowing the lower bound for a problem does not give you a good algorithm. But it does help you to know when to stop looking. If the lower bound for the problem matches the upper bound for the algorithm (within a constant factor), then we know that we can find an algorithm that is better only by a constant factor. 3.7 Common Misunderstandings Asymptotic analysis is one of the most intellectually difficult topics that undergrad- uate computer science majors are confronted with. Most people find growth rates and asymptotic analysis confusing and so develop misconceptions about either the concepts or the terminology. It helps to know what the standard points of confusion are, in hopes of avoiding them. One problem with differentiating the concepts of upper and lower bounds is that, for most algorithms that you will encounter, it is easy to recognize the true growth rate for that algorithm. Given complete knowledge about a cost function, the upper and lower bound for that cost function are always the same. Thus, the distinction between an upper and a lower bound is only worthwhile when you have incomplete knowledge about the thing being measured. If this distinction is still not clear, reread Section 3.6. We use ⇥-notation to indicate that there is no meaningful difference between what we know about the growth rates of the upper and lower bound (which is usually the case for simple algorithms). It is a common mistake to confuse the concepts of upper bound or lower bound on the one hand, and worst case or best case on the other. The best, worst, or average cases each give us a concrete input instance (or concrete set of instances) that we can apply to an algorithm description to get a cost measure. The upper and lower bounds describe our understanding of the growth rate for that cost measure. So to define the growth rate for an algorithm or problem, we need to determine what we are measuring (the best, worst, or average case) and also our description for what we know about the growth rate of that cost measure (big-Oh, ⌦, or ⇥). The upper bound for an algorithm is not the same as the worst case for that algorithm for a given input of size n. What is being bounded is not the actual cost (which you can determine for a given value of n), but rather the growth rate for the 2While it is fortunate to know the truth, it is unfortunate that sorting is ⇥(n log n) rather than ⇥(n)! 78 Chap. 3 Algorithm Analysis cost. There cannot be a growth rate for a single point, such as a particular value of n. The growth rate applies to the change in cost as a change in input size occurs. Likewise, the lower bound is not the same as the best case for a given size n. Another common misconception is thinking that the best case for an algorithm occurs when the input size is as small as possible, or that the worst case occurs when the input size is as large as possible. What is correct is that best- and worse- case instances exist for each possible size of input. That is, for all inputs of a given size, say i, one (or more) of the inputs of size i is the best and one (or more) of the inputs of size i is the worst. Often (but not always!), we can characterize the best input case for an arbitrary size, and we can characterize the worst input case for an arbitrary size. Ideally, we can determine the growth rate for the characterized best, worst, and average cases as the input size grows. Example 3.14 What is the growth rate of the best case for sequential search? For any array of size n, the best case occurs when the value we are looking for appears in the first position of the array. This is true regard- less of the size of the array. Thus, the best case (for arbitrary size n) occurs when the desired value is in the first of n positions, and its cost is 1. It is not correct to say that the best case occurs when n =1. Example 3.15 Imagine drawing a graph to show the cost of finding the maximum value among n values, as n grows. That is, the x axis would be n, and the y value would be the cost. Of course, this is a diagonal line going up to the right, as n increases (you might want to sketch this graph for yourself before reading further). Now, imagine the graph showing the cost for each instance of the prob- lem of finding the maximum value among (say) 20 elements in an array. The first position along the x axis of the graph might correspond to having the maximum element in the first position of the array. The second position along the x axis of the graph might correspond to having the maximum el- ement in the second position of the array, and so on. Of course, the cost is always 20. Therefore, the graph would be a horizontal line with value 20. You should sketch this graph for yourself. Now, let us switch to the problem of doing a sequential search for a given value in an array. Think about the graph showing all the problem instances of size 20. The first problem instance might be when the value we search for is in the first position of the array. This has cost 1. The second problem instance might be when the value we search for is in the second position of the array. This has cost 2. And so on. If we arrange the problem instances of size 20 from least expensive on the left to most expensive on Sec. 3.8 Multiple Parameters 79 the right, we see that the graph forms a diagonal line from lower left (with value 0) to upper right (with value 20). Sketch this graph for yourself. Finally, let us consider the cost for performing sequential search as the size of the array n gets bigger. What will this graph look like? Unfortu- nately, there’s not one simple answer, as there was for finding the maximum value. The shape of this graph depends on whether we are considering the best case cost (that would be a horizontal line with value 1), the worst case cost (that would be a diagonal line with value i at position i along the x axis), or the average cost (that would be a a diagonal line with value i/2 at position i along the x axis). This is why we must always say that function f(n) is in O(g(n)) in the best, average, or worst case! If we leave off which class of inputs we are discussing, we cannot know which cost measure we are referring to for most algorithms. 3.8 Multiple Parameters Sometimes the proper analysis for an algorithm requires multiple parameters to de- scribe the cost. To illustrate the concept, consider an algorithm to compute the rank ordering for counts of all pixel values in a picture. Pictures are often represented by a two-dimensional array, and a pixel is one cell in the array. The value of a pixel is either the code value for the color, or a value for the intensity of the picture at that pixel. Assume that each pixel can take any integer value in the range 0 to C 1. The problem is to find the number of pixels of each color value and then sort the color values with respect to the number of times each value appears in the picture. Assume that the picture is a rectangle with P pixels. A pseudocode algorithm to solve the problem follows. for (i=0; i12, the value is too large to store as an int variable anyway.) Compared to the time required to compute factorials, it may be well worth the small amount of additional space needed to store the lookup table. Lookup tables can also store approximations for an expensive function such as sine or cosine. If you compute this function only for exact degrees or are willing to approximate the answer with the value for the nearest degree, then a lookup table storing the computation for exact degrees can be used instead of repeatedly computing the sine function. Note that initially building the lookup table requires a certain amount of time. Your application must use the lookup table often enough to make this initialization worthwhile. Another example of the space/time tradeoff is typical of what a programmer might encounter when trying to optimize space. Here is a simple code fragment for sorting an array of integers. We assume that this is a special case where there are n integers whose values are a permutation of the integers from 0 to n 1. This is an example of a Binsort, which is discussed in Section 7.7. Binsort assigns each value to an array position corresponding to its value. for (i=0; i 1) if (ODD(n)) n=3* n + 1; else n = n / 2; Do you think that the upper bound is likely to be the same as the answer you gave for the lower bound? 3.15 Does every algorithm have a ⇥ running-time equation? In other words, are the upper and lower bounds for the running time (on any specified class of inputs) always the same? 3.16 Does every problem for which there exists some algorithm have a ⇥ running- time equation? In other words, for every problem, and for any specified class of inputs, is there some algorithm whose upper bound is equal to the problem’s lower bound? 3.17 Given an array storing integers ordered by value, modify the binary search routine to return the position of the first integer with value K in the situation where K can appear multiple times in the array. Be sure that your algorithm is ⇥(log n), that is, do not resort to sequential search once an occurrence of K is found. 3.18 Given an array storing integers ordered by value, modify the binary search routine to return the position of the integer with the greatest value less than K when K itself does not appear in the array. Return ERROR if the least value in the array is greater than K. 3.19 Modify the binary search routine to support search in an array of infinite size. In particular, you are given as input a sorted array and a key value K to search for. Call n the position of the smallest value in the array that 90 Chap. 3 Algorithm Analysis is equal to or larger than X. Provide an algorithm that can determine n in O(log n) comparisons in the worst case. Explain why your algorithm meets the required time bound. 3.20 It is possible to change the way that we pick the dividing point in a binary search, and still get a working search routine. However, where we pick the dividing point could affect the performance of the algorithm. (a) If we change the dividing point computation in function binary from i =(l + r)/2 to i =(l +((r l)/3)), what will the worst-case run- ning time be in asymptotic terms? If the difference is only a constant time factor, how much slower or faster will the modified program be compared to the original version of binary? (b) If we change the dividing point computation in function binary from i =(l + r)/2 to i = r 2, what will the worst-case running time be in asymptotic terms? If the difference is only a constant time factor, how much slower or faster will the modified program be compared to the original version of binary? 3.21 Design an algorithm to assemble a jigsaw puzzle. Assume that each piece has four sides, and that each piece’s final orientation is known (top, bottom, etc.). Assume that you have available a function bool compare(Piece a, Piece b, Side ad) that can tell, in constant time, whether piece a connects to piece b on a’s side ad and b’s opposite side bd. The input to your algorithm should consist of an n ⇥ m array of random pieces, along with dimensions n and m. The algorithm should put the pieces in their correct positions in the array. Your algorithm should be as efficient as possible in the asymptotic sense. Write a summation for the running time of your algorithm on n pieces, and then derive a closed-form solution for the summation. 3.22 Can the average case cost for an algorithm be worse than the worst case cost? Can it be better than the best case cost? Explain why or why not. 3.23 Prove that if an algorithm is ⇥(f(n)) in the average case, then it is ⌦(f(n)) in the worst case. 3.24 Prove that if an algorithm is ⇥(f(n)) in the average case, then it is O(f(n)) in the best case. 3.14 Projects 3.1 Imagine that you are trying to store 32 Boolean values, and must access them frequently. Compare the time required to access Boolean values stored alternatively as a single bit field, a character, a short integer, or a long integer. There are two things to be careful of when writing your program. First, be Sec. 3.14 Projects 91 sure that your program does enough variable accesses to make meaningful measurements. A single access takes much less time than a single unit of measurement (typically milliseconds) for all four methods. Second, be sure that your program spends as much time as possible doing variable accesses rather than other things such as calling timing functions or incrementing for loop counters. 3.2 Implement sequential search and binary search algorithms on your computer. Run timings for each algorithm on arrays of size n = 10i for i ranging from 1 to as large a value as your computer’s memory and compiler will allow. For both algorithms, store the values 0 through n 1 in order in the array, and use a variety of random search values in the range 0 to n 1 on each size n. Graph the resulting times. When is sequential search faster than binary search for a sorted array? 3.3 Implement a program that runs and gives timings for the two Fibonacci se- quence functions provided in Exercise 2.11. Graph the resulting running times for as many values of n as your computer can handle. PART II Fundamental Data Structures 93 4 Lists, Stacks, and Queues If your program needs to store a few things — numbers, payroll records, or job de- scriptions for example — the simplest and most effective approach might be to put them in a list. Only when you have to organize and search through a large number of things do more sophisticated data structures usually become necessary. (We will study how to organize and search through medium amounts of data in Chapters 5, 7, and 9, and discuss how to deal with large amounts of data in Chapters 8–10.) Many applications don’t require any form of search, and they do not require that any or- dering be placed on the objects being stored. Some applications require processing in a strict chronological order, processing objects in the order that they arrived, or perhaps processing objects in the reverse of the order that they arrived. For all these situations, a simple list structure is appropriate. This chapter describes representations for lists in general, as well as two impor- tant list-like structures called the stack and the queue. Along with presenting these fundamental data structures, the other goals of the chapter are to: (1) Give examples of separating a logical representation in the form of an ADT from a physical im- plementation for a data structure. (2) Illustrate the use of asymptotic analysis in the context of some simple operations that you might already be familiar with. In this way you can begin to see how asymptotic analysis works, without the complica- tions that arise when analyzing more sophisticated algorithms and data structures. (3) Introduce the concept and use of dictionaries. We begin by defining an ADT for lists in Section 4.1. Two implementations for the list ADT — the array-based list and the linked list — are covered in detail and their relative merits discussed. Sections 4.2 and 4.3 cover stacks and queues, re- spectively. Sample implementations for each of these data structures are presented. Section 4.4 presents the Dictionary ADT for storing and retrieving data, which sets a context for implementing search structures such as the Binary Search Tree of Section 5.4. 95 96 Chap. 4 Lists, Stacks, and Queues 4.1 Lists We all have an intuitive understanding of what we mean by a “list.” Our first step is to define precisely what is meant so that this intuitive understanding can eventually be converted into a concrete data structure and its operations. The most important concept related to lists is that of position. In other words, we perceive that there is a first element in the list, a second element, and so on. We should view a list as embodying the mathematical concepts of a sequence, as defined in Section 2.1. We define a list to be a finite, ordered sequence of data items known as ele- ments. “Ordered” in this definition means that each element has a position in the list. (We will not use “ordered” in this context to mean that the list elements are sorted by value.) Each list element has a data type. In the simple list implemen- tations discussed in this chapter, all elements of the list have the same data type, although there is no conceptual objection to lists whose elements have differing data types if the application requires it (see Section 12.1). The operations defined as part of the list ADT do not depend on the elemental data type. For example, the list ADT can be used for lists of integers, lists of characters, lists of payroll records, even lists of lists. A list is said to be empty when it contains no elements. The number of ele- ments currently stored is called the length of the list. The beginning of the list is called the head, the end of the list is called the tail. There might or might not be some relationship between the value of an element and its position in the list. For example, sorted lists have their elements positioned in ascending order of value, while unsorted lists have no particular relationship between element values and positions. This section will consider only unsorted lists. Chapters 7 and 9 treat the problems of how to create and search sorted lists efficiently. When presenting the contents of a list, we use the same notation as was in- troduced for sequences in Section 2.1. To be consistent with C++ array indexing, the first position on the list is denoted as 0. Thus, if there are n elements in the list, they are given positions 0 through n 1 as ha0,a1,...,an1i. The subscript indicates an element’s position within the list. Using this notation, the empty list would appear as hi. Before selecting a list implementation, a program designer should first consider what basic operations the implementation must support. Our common intuition about lists tells us that a list should be able to grow and shrink in size as we insert and remove elements. We should be able to insert and remove elements from any- where in the list. We should be able to gain access to any element’s value, either to read it or to change it. We must be able to create and clear (or reinitialize) lists. It is also convenient to access the next or previous element from the “current” one. The next step is to define the ADT for a list object in terms of a set of operations on that object. We will use the C++ notation of anabstract class to formally define Sec. 4.1 Lists 97 the list ADT. An abstract class is one whose member functions are all declared to be “pure virtual” as indicated by the “=0” notation at the end of the member function declarations. Class List defines the member functions that any list implementa- tion inheriting from it must support, along with their parameters and return types. We increase the flexibility of the list ADT by writing it as a C++ template. True to the notion of an ADT, anabstract class does not specify how operations are implemented. Two complete implementations are presented later in this sec- tion, both of which use the same list ADT to define their operations, but they are considerably different in approaches and in their space/time tradeoffs. Figure 4.1 presents our list ADT. Class List is a template of one parameter, named E for “element”. E serves as a placeholder for whatever element type the user would like to store in a list. The comments given in Figure 4.1 describe pre- cisely what each member function is intended to do. However, some explanation of the basic design is in order. Given that we wish to support the concept of a se- quence, with access to any position in the list, the need for many of the member functions such as insert and moveToPos is clear. The key design decision em- bodied in this ADT is support for the concept of a current position. For example, member moveToStart sets the current position to be the first element on the list, while methods next and prev move the current position to the next and previ- ous elements, respectively. The intention is that any implementation for this ADT support the concept of a current position. The current position is where any action such as insertion or deletion will take place. Since insertions take place at the current position, and since we want to be able to insert to the front or the back of the list as well as anywhere in between, there are actually n +1possible “current positions” when there are n elements in the list. It is helpful to modify our list display notation to show the position of the current element. I will use a vertical bar, such as h20, 23 | 12, 15i to indicate the list of four elements, with the current position being to the right of the bar at element 12. Given this configuration, calling insert with value 10 will change the list to be h20, 23 | 10, 12, 15i. If you examine Figure 4.1, you should find that the list member functions pro- vided allow you to build a list with elements in any desired order, and to access any desired position in the list. You might notice that the clear method is not necessary, in that it could be implemented by means of the other member functions in the same asymptotic time. It is included merely for convenience. Method getValue returns a pointer to the current element. It is considered a violation of getValue’s preconditions to ask for the value of a non-existent ele- ment (i.e., there must be something to the right of the vertical bar). In our concrete list implementations, assertions are used to enforce such preconditions. In a com- mercial implementation, such violations would be best implemented by the C++ exception mechanism. 98 Chap. 4 Lists, Stacks, and Queues template class List { // List ADT private: void operator =(const List&) {} // Protect assignment List(const List&) {} // Protect copy constructor public: List() {} // Default constructor virtual ˜List() {} // Base destructor // Clear contents from the list, to make it empty. virtual void clear() = 0; // Insert an element at the current location. // item: The element to be inserted virtual void insert(const E& item) = 0; // Append an element at the end of the list. // item: The element to be appended. virtual void append(const E& item) = 0; // Remove and return the current element. // Return: the element that was removed. virtual E remove() = 0; // Set the current position to the start of the list virtual void moveToStart() = 0; // Set the current position to the end of the list virtual void moveToEnd() = 0; // Move the current position one step left. No change // if already at beginning. virtual void prev() = 0; // Move the current position one step right. No change // if already at end. virtual void next() = 0; // Return: The number of elements in the list. virtual int length() const = 0; // Return: The position of the current element. virtual int currPos() const = 0; // Set current position. // pos: The position to make current. virtual void moveToPos(int pos) = 0; // Return: The current element. virtual const E& getValue() const = 0; }; Figure 4.1 The ADT for a list. Sec. 4.1 Lists 99 A list can be iterated through as shown in the following code fragment. for (L.moveToStart(); L.currPos()& L, int K) { int it; for (L.moveToStart(); L.currPos() // Array-based list implementation class AList : public List { private: int maxSize; // Maximum size of list int listSize; // Number of list items now int curr; // Position of current element E* listArray; // Array holding list elements public: AList(int size=defaultSize) { // Constructor maxSize = size; listSize = curr = 0; listArray = new E[maxSize]; } ˜AList() { delete [] listArray; } // Destructor void clear() { // Reinitialize the list delete [] listArray; // Remove the array listSize = curr = 0; // Reset the size listArray = new E[maxSize]; // Recreate array } // Insert "it" at current position void insert(const E& it) { Assert(listSize < maxSize, "List capacity exceeded"); for(int i=listSize; i>curr; i--) // Shift elements up listArray[i] = listArray[i-1]; // to make room listArray[curr] = it; listSize++; // Increment list size } void append(const E& it) { // Append "it" Assert(listSize < maxSize, "List capacity exceeded"); listArray[listSize++] = it; } // Remove and return the current element. E remove() { Assert((curr>=0) && (curr < listSize), "No element"); E it = listArray[curr]; // Copy the element for(int i=curr; i=0)&&(pos<=listSize), "Pos out of range"); curr = pos; } const E& getValue() const { // Return current element Assert((curr>=0)&&(curr class Link { public: E element; // Value for this node Link *next; // Pointer to next node in list // Constructors Link(const E& elemval, Link* nextval =NULL) { element = elemval; next = nextval; } Link(Link* nextval =NULL) { next = nextval; } }; Figure 4.4 A simple singly linked list node implementation. more than constant time are the constructor, the destructor, and clear. These three member functions each make use of the system free-storeoperators new and delete. As discussed further in Section 4.1.2, system free-store operations can be expensive. In particular, the cost to delete listArray depends in part on the type of elements it stores, and whether the delete operator must call a destructor on each one. 4.1.2 Linked Lists The second traditional approach to implementing lists makes use of pointers and is usually called a linked list. The linked list uses dynamic memory allocation, that is, it allocates memory for new list elements as needed. A linked list is made up of a series of objects, called the nodes of the list. Because a list node is a distinct object (as opposed to simply a cell in an array), it is good practice to make a separate list node class. An additional benefit to creating a list node class is that it can be reused by the linked implementations for the stack and queue data structures presented later in this chapter. Figure 4.4 shows the implementation for list nodes, called the Link class. Objects in the Link class contain an element field to store the element value, and a next field to store a pointer to the next node on the list. The list built from such nodes is called a singly linked list, or a one-way list, because each list node has a single pointer to the next node on the list. The Link class is quite simple. There are two forms for its constructor, one with an initial element value and one without. Because the Link class is also used by the stack and queue implementations presented later, its data members are made public. While technically this is breaking encapsulation, in practice the Link class should be implemented as a private class of the linked list (or stack or queue) implementation, and thus not visible to the rest of the program. Figure 4.5(a) shows a graphical depiction for a linked list storing four integers. The value stored in a pointer variable is indicated by an arrow “pointing” to some- thing. C++ uses the special symbol NULL for a pointer value that points nowhere, such as for the last list node’s next field. A NULL pointer is indicated graphically 104 Chap. 4 Lists, Stacks, and Queues head 20 23 15 (a) head tail 1512102320 (b) curr curr tail 12 Figure 4.5 Illustration of a faulty linked-list implementation where curr points directly to the current node. (a) Linked list prior to inserting element with value 10. (b) Desired effect of inserting element with value 10. by a diagonal slash through a pointer variable’s box. The vertical line between the nodes labeled 23 and 12 in Figure 4.5(a) indicates the current position (immediately to the right of this line). The list’s first node is accessed from a pointer named head. To speed access to the end of the list, and to allow the append method to be performed in constant time, a pointer named tail is also kept to the last link of the list. The position of the current element is indicated by another pointer, named curr. Finally, because there is no simple way to compute the length of the list simply from these three pointers, the list length must be stored explicitly, and updated by every operation that modifies the list size. The value cnt stores the length of the list. Class LList also includes private helper methods init and removeall. They are used by LList’s constructor, destructor, and clear methods. Note that LList’s constructor maintains the optional parameter for minimum list size introduced for Class AList. This is done simply to keep the calls to the constructor the same for both variants. Because the linked list class does not need to declare a fixed-size array when the list is created, this parameter is unnecessary for linked lists. It is ignored by the implementation. A key design decision for the linked list implementation is how to represent the current position. The most reasonable choices appear to be a pointer to the current element. But there is a big advantage to making curr point to the element preceding the current element. Figure 4.5(a) shows the list’s curr pointer pointing to the current element. The vertical line between the nodes containing 23 and 12 indicates the logical position of the current element. Consider what happens if we wish to insert a new node with value 10 into the list. The result should be as shown in Figure 4.5(b). However, there is a problem. To “splice” the list node containing the new element into the list, the list node storing 23 must have its next pointer changed to point to the new Sec. 4.1 Lists 105 tailcurrhead 20 23 12 15 (a) head tail 20 23 10 12 (b) 15 curr Figure 4.6 Insertion using a header node, with curr pointing one node head of the current element. (a) Linked list before insertion. The current node contains 12. (b) Linked list after inserting the node containing 10. node. Unfortunately, there is no convenient access to the node preceding the one pointed to by curr. There is an easy solution to this problem. If we set curr to point directly to the preceding element, there is no difficulty in adding a new element after curr. Figure 4.6 shows how the list looks when pointer variable curr is set to point to the node preceding the physical current node. See Exercise 4.5 for further discussion of why making curr point directly to the current element fails. We encounter a number of potential special cases when the list is empty, or when the current position is at an end of the list. In particular, when the list is empty we have no element for head, tail, and curr to point to. Implementing special cases for insert and remove increases code complexity, making it harder to understand, and thus increases the chance of introducing a programming bug. These special cases can be eliminated by implementing linked lists with an additional header node as the first node of the list. This header node is a link node like any other, but its value is ignored and it is not considered to be an actual element of the list. The header node saves coding effort because we no longer need to consider special cases for empty lists or when the current position is at one end of the list. The cost of this simplification is the space for the header node. However, there are space savings due to smaller code size, because statements to handle the special cases are omitted. In practice, this reduction in code size typically saves more space than that required for the header node, depending on the number of lists created. Figure 4.7 shows the state of an initialized or empty list when using a header node. Figure 4.8 shows the definition for the linked list class, named LList. Class LList inherits from the abstract list class and thus must implement all of Class List’s member functions. 106 Chap. 4 Lists, Stacks, and Queues tail head curr Figure 4.7 Initial state of a linked list when using a header node. Implementations for most member functions of the list class are straightfor- ward. However, insert and remove should be studied carefully. Inserting a new element is a three-step process. First, the new list node is created and the new element is stored into it. Second, the next field of the new list node is assigned to point to the current node (the one after the node that curr points to). Third, the next field of node pointed to by curr is assigned to point to the newly inserted node. The following line in the insert method of Figure 4.8 does all three of these steps. curr->next = new Link(it, curr->next); Operator new creates the new link node and calls the Link class constructor, which takes two parameters. The first is the element. The second is the value to be placed in the list node’s next field, in this case“curr->next.” Figure 4.9 illustrates this three-step process. Once the new node is added, tail is pushed forward if the new element was added to the end of the list. Insertion requires ⇥(1) time. Removing a node from the linked list requires only that the appropriate pointer be redirected around the node to be deleted. The following lines from the remove method of Figure 4.8 do precisely this. Link* ltemp = curr->next; // Remember link node curr->next = curr->next->next; // Remove from list We must be careful not to “lose” the memory for the deleted link node. So, tem- porary pointer ltemp is first assigned to point to the node being removed. A call to delete is later used to return the old node to free storage. Figure 4.10 illus- trates the remove method.Assuming that the free-store delete operator requires constant time, removing an element requires ⇥(1) time. Method next simply moves curr one position toward the tail of the list, which takes ⇥(1) time. Method prev moves curr one position toward the head of the list, but its implementation is more difficult. In a singly linked list, there is no pointer to the previous node. Thus, the only alternative is to march down the list from the beginning until we reach the current node (being sure always to remember the node before it, because that is what we really want). This takes ⇥(n) time in the average and worst cases. Implementation of method moveToPos is similar in that finding the ith position requires marching down i positions from the head of the list, taking ⇥(i) time. Implementations for the remaining operations each require ⇥(1) time. Sec. 4.1 Lists 107 // Linked list implementation template class LList: public List { private: Link* head; // Pointer to list header Link* tail; // Pointer to last element Link* curr; // Access to current element int cnt; // Size of list void init() { // Intialization helper method curr = tail = head = new Link; cnt = 0; } void removeall() { // Return link nodes to free store while(head != NULL) { curr = head; head = head->next; delete curr; } } public: LList(int size=defaultSize) { init(); } // Constructor ˜LList() { removeall(); } // Destructor void print() const; // Print list contents void clear() { removeall(); init(); } // Clear list // Insert "it" at current position void insert(const E& it) { curr->next = new Link(it, curr->next); if (tail == curr) tail = curr->next; // New tail cnt++; } void append(const E& it) { // Append "it" to list tail = tail->next = new Link(it, NULL); cnt++; } // Remove and return current element E remove() { Assert(curr->next != NULL, "No element"); E it = curr->next->element; // Remember value Link* ltemp = curr->next; // Remember link node if (tail == curr->next) tail = curr; // Reset tail curr->next = curr->next->next; // Remove from list delete ltemp; // Reclaim space cnt--; // Decrement the count return it; } Figure 4.8 A linked list implementation. 108 Chap. 4 Lists, Stacks, and Queues void moveToStart() // Place curr at list start { curr = head; } void moveToEnd() // Place curr at list end { curr = tail; } // Move curr one step left; no change if already at front void prev() { if (curr == head) return; // No previous element Link* temp = head; // March down list until we find the previous element while (temp->next!=curr) temp=temp->next; curr = temp; } // Move curr one step right; no change if already at end void next() { if (curr != tail) curr = curr->next; } int length() const { return cnt; } // Return length // Return the position of the current element int currPos() const { Link* temp = head; int i; for (i=0; curr != temp; i++) temp = temp->next; return i; } // Move down list to "pos" position void moveToPos(int pos) { Assert ((pos>=0)&&(pos<=cnt), "Position out of range"); curr = head; for(int i=0; inext; } const E& getValue() const { // Return current element Assert(curr->next != NULL, "No value"); return curr->next->element; } }; Figure 4.8 (continued) Sec. 4.1 Lists 109 ... ... (a) ... ... (b) curr curr 23 12 Insert 10: 10 23 12 10 12 3 Figure 4.9 The linked list insertion process. (a) The linked list before insertion. (b) The linked list after insertion. 1 marks the element field of the new link node. 2 marks the next field of the new link node, which is set to point to what used to be the current node (the node with value 12). 3 marks the next field of the node preceding the current position. It used to point to the node containing 12; now it points to the new node containing 10. ... ... ... ... (a) (b) it curr 23 12 1210 10 23 curr 2 1 Figure 4.10 The linked list removal process. (a) The linked list before removing the node with value 10. (b) The linked list after removal. 1 marks the list node being removed. it is set to point to the element. 2 marks the next field of the preceding list node, which is set to point to the node following the one being deleted. 110 Chap. 4 Lists, Stacks, and Queues Freelists TheC++ free-store management operators new and delete are relatively expen- sive to use. Section 12.3 discusses how general-purpose memory managers are implemented. The expense comes from the fact that free-store routines must be ca- pable of handling requests to and from free store with no particular pattern, as well as memory requests of vastly different sizes. Thismakes them inefficient compared to what might be implemented for more controlled patterns of memory access. List nodes are created and deleted in a linked list implementation in a way that allows the Link class programmer to provide simple but efficient memory management routines. Instead of making repeated calls to new and delete, the Link class can handle its own freelist. A freelist holds those list nodes that are not currently being used. When a node is deleted from a linked list, it is placed at the head of the freelist. When a new element is to be added to a linked list, the freelist is checked to see if a list node is available. If so, the node is taken from the freelist. If the freelist is empty, the standard new operator must then be called. Freelists are particularly useful for linked lists that periodically grow and then shrink. The freelist will never grow larger than the largest size yet reached by the linked list. Requests for new nodes (after the list has shrunk) can be handled by the freelist. Another good opportunity to use a freelist occurs when a program uses multiple lists. So long as they do not all grow and shrink together, the free list can let link nodes move between the lists. One approach to implementing freelists would be to create two new operators to use instead of the standard free-store routines new and delete. This requires that the user’s code, such as the linked list class implementation of Figure 4.8, be modified to call these freelist operators. A second approach is to use C++ operator overloading to replace the meaning of new and delete when operating on Link class objects. In this way, programs that use the LList class need not be modified at all to take advantage of a freelist. Whether the Link class is implemented with freelists, or relies on the regular free-store mechanism, is entirely hidden from the list class user. Figure 4.11 shows the reimplementation for the Link classwith freelist methods overloading the standard free-store operators. Note how simple they are, because they need only remove and add an element to the front of the freelist, respectively. The freelist versions of new and delete both run in ⇥(1) time, except in the case where the freelist is exhausted and the new operation must be called. On my computer, a call to the overloaded new and delete operators requires about one tenth of the time required by the system free-store operators. There is an additional efficiency gain to be had from a freelist implementation. The implementation of Figure 4.11 makes a separate call to the system new oper- ator for each link node requested whenever the freelist is empty. These link nodes tend to be small — only a few bytes more than the size of the element field. If at some point in time the program requires thousands of active link nodes, these will Sec. 4.1 Lists 111 // Singly linked list node with freelist support template class Link { private: static Link* freelist; // Reference to freelist head public: E element; // Value for this node Link* next; // Point to next node in list // Constructors Link(const E& elemval, Link* nextval =NULL) { element = elemval; next = nextval; } Link(Link* nextval =NULL) { next = nextval; } void* operator new(size t) { // Overloaded new operator if (freelist == NULL) return ::new Link; // Create space Link* temp = freelist; // Can take from freelist freelist = freelist->next; return temp; // Return the link } // Overloaded delete operator void operator delete(void* ptr) { ((Link*)ptr)->next = freelist; // Put on freelist freelist = (Link*)ptr; } }; // The freelist head pointer is actually created here template Link* Link::freelist = NULL; Figure 4.11 Implementation for the Link class with a freelist. Note that the redefinition for new refers to ::new on the third line. This indicates that the standard C++ new operator is used, rather than the redefined new operator. If the colons had not been used, then the Link class new operator would be called, setting up an infinite recursion. The static declaration for member freelist means that all Link class objects share the same freelist pointer variable instead of each object storing its own copy. have been created by many calls to the system version of new. An alternative is to allocate many link nodes in a single call to the system version of new, anticipating that if the freelist is exhausted now, more nodes will be needed soon. It is faster to make one call to new to get space for 100 link nodes, and then load all 100 onto the freelist at once, rather than to make 100 separate calls to new. The following statement will assign ptr to point to an array of 100 link nodes. ptr = ::new Link[100]; The implementation for the new operator in the link class could then place each of these 100 nodes onto the freelist. 112 Chap. 4 Lists, Stacks, and Queues The freelist variable declaration uses the keyword static. This creates a single variable shared among all instances of the Link nodes. We want only a single freelist for all Link nodes of a given type. A program might create multiple lists. If they are all of the same type (that is, their element types are the same), then they can and should share the same freelist. This will happen with the implemen- tation of Figure 4.11. If lists are created that have different element types, because this code is implemented with a template, the need for different list implementa- tions will be discovered by the compiler at compile time. Separate versions of the list class will be generated for each element type. Thus, each element type will also get its own separate copy of the Link class. And each distinct Link class implementation will get a separate freelist. 4.1.3 Comparison of List Implementations Now that you have seen two substantially different implementations for lists, it is natural to ask which is better. In particular, if you must implement a list for some task, which implementation should you choose? Array-based lists have the disadvantage that their size must be predetermined before the array can be allocated. Array-based lists cannot grow beyond their pre- determined size. Whenever the list contains only a few elements, a substantial amount of space might be tied up in a largely empty array. Linked lists have the advantage that they only need space for the objects actually on the list. There is no limit to the number of elements on a linked list, as long as there is free-store memory available. The amount of space required by a linked list is ⇥(n), while the space required by the array-based list implementation is ⌦(n), but can be greater. Array-based lists have the advantage that there is no wasted space for an in- dividual element. Linked lists require that an extra pointer be added to every list node. If the element size is small, then the overhead for links can be a significant fraction of the total storage. When the array for the array-based list is completely filled, there is no storage overhead. The array-based list will then be more space efficient, by a constant factor, than the linked implementation. A simple formula can be used to determine whether the array-based list or linked list implementation will be more space efficient in a particular situation. Call n the number of elements currently in the list, P the size of a pointer in stor- age units (typically four bytes), E the size of a data element in storage units (this could be anything, from one bit for a Boolean variable on up to thousands of bytes or more for complex records), and D the maximum number of list elements that can be stored in the array. The amount of space required for the array-based list is DE, regardless of the number of elements actually stored in the list at any given time. The amount of space required for the linked list is n(P + E). The smaller of these expressions for a given value n determines the more space-efficient imple- mentation for n elements. In general, the linked implementation requires less space Sec. 4.1 Lists 113 than the array-based implementation when relatively few elements are in the list. Conversely, the array-based implementation becomes more space efficient when the array is close to full. Using the equation, we can solve for n to determine the break-even point beyond which the array-based implementation is more space efficient in any particular situation. This occurs when n>DE/(P + E). If P = E, then the break-even point is at D/2. This would happen if the element field is either a four-byte int value or a pointer, and the next field is a typical four- byte pointer. That is, the array-based implementation would be more efficient (if the link field and the element field are the same size) whenever the array is more than half full. As a rule of thumb, linked lists are more space efficient when implementing lists whose number of elements varies widely or is unknown. Array-based lists are generally more space efficient when the user knows in advance approximately how large the list will become. Array-based lists are faster for random access by position. Positions can easily be adjusted forwards or backwards by the next and prev methods. These opera- tions always take ⇥(1) time. In contrast, singly linked lists have no explicit access to the previous element, and access by position requires that we march down the list from the front (or the current position) to the specified position. Both of these operations require ⇥(n) time in the average and worst cases, if we assume that each position on the list is equally likely to be accessed on any call to prev or moveToPos. Given a pointer to a suitable location in the list, the insert and remove methods for linked lists require only ⇥(1) time. Array-based lists must shift the re- mainder of the list up or down within the array. This requires ⇥(n) time in the aver- age and worst cases. For many applications, the time to insert and delete elements dominates all other operations. For this reason, linked lists are often preferred to array-based lists. When implementing the array-based list, an implementor could allow the size of the array to grow and shrink depending on the number of elements that are actually stored. This data structure is known as a dynamic array. Both the Java and C++/STL Vector classes implement a dynamic array. Dynamic arrays allow the programmer to get around the limitation on the standard array that its size cannot be changed once the array has been created. This also means that space need not be allocated to the dynamic array until it is to be used. The disadvantage of this approach is that it takes time to deal with space adjustments on the array. Each time the array grows in size, its contents must be copied. A good implementation of the dynamic array will grow and shrink the array in such a way as to keep the overall cost for a series of insert/delete operations relatively inexpensive, even though an 114 Chap. 4 Lists, Stacks, and Queues occasional insert/delete operation might be expensive. A simple rule of thumb is to double the size of the array when it becomes full, and to cut the array size in half when it becomes one quarter full. To analyze the overall cost of dynamic array operations over time, we need to use a technique known as amortized analysis, which is discussed in Section 14.3. 4.1.4 Element Implementations List users must decide whether they wish to store a copy of any given element on each list that contains it. For small elements such as an integer, this makes sense. If the elements are payroll records, it might be desirable for the list node to store a pointer to the record rather than store a copy of the record itself. This change would allow multiple list nodes (or other data structures) to point to the same record, rather than make repeated copies of the record. Not only might this save space, but it also means that a modification to an element’s value is automatically reflected at all locations where it is referenced. The disadvantage of storing a pointer to each element is that the pointer requires space of its own. If elements are never duplicated, then this additional space adds unnecessary overhead. The C++ implementations for lists presented in this section give the user of the list the choice of whether to store copies of elements or pointers to elements. The user can declare E to be, for example, a pointer to a payroll record. In this case, multiple lists can point to the same copy of the record. On the other hand, if the user declares E to be the record itself, then a new copy of the record will be made when it is inserted into the list. Whether it is more advantageous to use pointers to shared elements or separate copies depends on the intended application. In general, the larger the elements and the more they are duplicated, the more likely that pointers to shared elements is the better approach. A second issue faced by implementors of a list class (or any other data structure that stores a collection of user-defined data elements) is whether the elements stored are all required to be of the same type. This is known as homogeneity in a data structure. In some applications, the user would like to define the class of the data element that is stored on a given list, and then never permit objects of a different class to be stored on that same list. In other applications, the user would like to permit the objects stored on a single list to be of differing types. For the list implementations presented in this section, the compiler requires that all objects stored on the list be of the same type. In fact, because the lists are implemented using templates, a new class is created by the compiler for each data type. For implementors who wish to minimize the number of classes created by the compiler, the lists can all store a void* pointer, with the user performing the necessary casting to and from the actual object type for each element. However, this Sec. 4.1 Lists 115 approach requires that the user do his or her own type checking, either to enforce homogeneity or to differentiate between the various object types. Besides C++ templates, there are other techniques that implementors of a list class can use to ensure that the element type for a given list remains fixed, while still permitting different lists to store different element types. One approach is to store an object of the appropriate type in the header node of the list (perhaps an object of the appropriate type is supplied as a parameter to the list constructor), and then check that all insert operations on that list use the same element type. The third issue that users of the list implementations must face is primarily of concern when programming in languages that do not support automatic garbage collection. That is how to deal with the memory of the objects stored on the list when the list is deleted or the clear method is called. The list destructor and the clear method are problematic in that there is a potential that they will bemisused, thus causing a memory leak. The type of the element stored determines whether there is a potential for trouble here. If the elements are of a simple type such as an int, then there is no need to delete the elements explicitly. If the elements are of a user-defined class, then their own destructor will be called. However, what if the list elements are pointers to objects? Then deleting listArray in the array-based implementation, or deleting a link node in the linked list implementation, might remove the only reference to an object, leaving its memory space inaccessible. Unfortunately, there is no way for the list implementation to know whether a given object is pointed to in another part of the program or not. Thus, the user of the list must be responsible for deleting these objects when that is appropriate. 4.1.5 Doubly Linked Lists The singly linked list presented in Section 4.1.2 allows for direct access from a list node only to the next node in the list. A doubly linked list allows convenient access from a list node to the next node and also to the preceding node on the list. The doubly linked list node accomplishes this in the obvious way by storing two pointers: one to the node following it (as in the singly linked list), and a second pointer to the node preceding it. The most common reason to use a doubly linked list is because it is easier to implement than a singly linked list. While the code for the doubly linked implementation is a little longer than for the singly linked version, it tends to be a bit more “obvious” in its intention, and so easier to implement and debug. Figure 4.12 illustrates the doubly linked list concept. Whether a list implementation is doubly or singly linked should be hidden from the List class user. Like our singly linked list implementation, the doubly linked list implementa- tion makes use of a header node. We also add a tailer node to the end of the list. The tailer is similar to the header, in that it is a node that contains no value, and it always exists. When the doubly linked list is initialized, the header and tailer nodes 116 Chap. 4 Lists, Stacks, and Queues head 20 23 curr 12 15 tail Figure 4.12 A doubly linked list. are created. Data member head points to the header node, and tail points to the tailer node. The purpose of these nodes is to simplify the insert, append, and remove methods by eliminating all need for special-case code when the list is empty, or when we insert at the head or tail of the list. For singly linked lists we set curr to point to the node preceding the node that contained the actual current element, due to lack of access to the previous node during insertion and deletion. Since we do have access to the previous node in a doubly linked list, this is no longer necessary. We could set curr to point directly to the node containing the current element. However, I have chosen to keep the same convention for the curr pointer as we set up for singly linked lists, purely for the sake of consistency. Figure 4.13 shows the complete implementation for a Link class to be used with doubly linked lists. This code is a little longer than that for the singly linked list node implementation since the doubly linked list nodes have an extra data member. Figure 4.14 shows the implementation for the insert, append, remove, and prev doubly linked list methods. The class declaration and the remaining member functions for the doubly linked list class are nearly identical to the singly linked list version. The insert method is especially simple for our doubly linked list implemen- tation, because most of the work is done by the node’s constructor. Figure 4.15 shows the list before and after insertion of a node with value 10. The three parameters to the new operator allow the list node class constructor to set the element, prev, and next fields, respectively, for the new link node. The new operator returns a pointer to the newly created node. The nodes to either side have their pointers updated to point to the newly created node. The existence of the header and tailer nodes mean that there are no special cases to worry about when inserting into an empty list. The append method is also simple. Again, the Link class constructor sets the element, prev, and next fields of the node when the new operator is executed. Method remove (illustrated by Figure 4.16) is straightforward, though the code is somewhat longer. First, the variable it is assigned the value being re- moved. Note that we must separate the element, which is returned to the caller, from the link object. The following lines then adjust the list. Sec. 4.1 Lists 117 // Doubly linked list link node with freelist support template class Link { private: static Link* freelist; // Reference to freelist head public: E element; // Value for this node Link* next; // Pointer to next node in list Link* prev; // Pointer to previous node // Constructors Link(const E& it, Link* prevp, Link* nextp) { element = it; prev = prevp; next = nextp; } Link(Link* prevp =NULL, Link* nextp =NULL) { prev = prevp; next = nextp; } void* operator new(size t) { // Overloaded new operator if (freelist == NULL) return ::new Link; // Create space Link* temp = freelist; // Can take from freelist freelist = freelist->next; return temp; // Return the link } // Overloaded delete operator void operator delete(void* ptr) { ((Link*)ptr)->next = freelist; // Put on freelist freelist = (Link*)ptr; } }; // The freelist head pointer is actually created here template Link* Link::freelist = NULL; Figure 4.13 Doubly linked list node implementation with a freelist. 118 Chap. 4 Lists, Stacks, and Queues // Insert "it" at current position void insert(const E& it) { curr->next = curr->next->prev = new Link(it, curr, curr->next); cnt++; } // Append "it" to the end of the list. void append(const E& it) { tail->prev = tail->prev->next = new Link(it, tail->prev, tail); cnt++; } // Remove and return current element E remove() { if (curr->next == tail) // Nothing to remove return NULL; E it = curr->next->element; // Remember value Link* ltemp = curr->next; // Remember link node curr->next->next->prev = curr; curr->next = curr->next->next; // Remove from list delete ltemp; // Reclaim space cnt--; // Decrement cnt return it; } // Move fence one step left; no change if left is empty void prev() { if (curr != head) // Can’t back up from list head curr = curr->prev; } Figure 4.14 Implementations for doubly linked list insert, append, remove, and prev methods. Link* ltemp = curr->next; // Remember link node curr->next->next->prev = curr; curr->next = curr->next->next; // Remove from list delete ltemp; // Reclaim space The first line sets a temporary pointer to the node being removed. The second line makes the next node’s prev pointer point to the left of the node being removed. Finally, the next field of the node preceding the one being deleted is adjusted. The final steps of method remove are to update the listlength, return the deleted node to free store, and return the value of the deleted element. The only disadvantage of the doubly linked list as compared to the singly linked list is the additional space used. The doubly linked list requires two pointers per node, and so in the implementation presented it requires twice as much overhead as the singly linked list. Sec. 4.1 Lists 119 ...1223 5 ... 20 ... 20 4 curr ...23 1210 32 (b) curr 10Insert 10: 1 (a) Figure 4.15 Insertion for doubly linked lists. The labels 1 , 2 , and 3 cor- respond to assignments done by the linked list node constructor. 4 marks the assignment to curr->next. 5 marks the assignment to the prev pointer of the node following the newly inserted node. ... 20 curr ...23 12 ... ...20 12 curr (b) 23it (a) Figure 4.16 Doubly linked list removal. Element it stores the element of the node being removed. Then the nodes to either side have their pointers adjusted. 120 Chap. 4 Lists, Stacks, and Queues Example 4.1 There is a space-saving technique that can be employed to eliminate the additional space requirement, though it will complicate the implementation and be somewhat slower. Thus, this is an example of a space/time tradeoff. It is based on observing that, if we store the sum of two values, then we can get either value back by subtracting the other. That is, if we store a + b in variable c, then b = c a and a = c b. Of course, to recover one of the values out of the stored summation, the other value must be supplied. A pointer to the first node in the list, along with the value of one of its two link fields, will allow access to all of the remaining nodes of the list in order. This is because the pointer to the node must be the same as the value of the following node’s prev pointer, as well as the previous node’s next pointer. It is possible to move down the list breaking apart the summed link fields as though you were opening a zipper. Details for implementing this variation are left as an exercise. The principle behind this technique is worth remembering, as it has many applications. The following code fragment will swap the contents of two variables without using a temporary variable (at the cost of three arithmetic operations). a = a + b; b = a - b; // Now b contains original value of a a = a - b; // Now a contains original value of b A similar effect can be had by using the exclusive-or operator. This fact is widely used in computer graphics. A region of the computer screen can be highlighted by XORing the outline of a box around it. XORing the box outline a second time restores the original contents of the screen. 4.2 Stacks The stack is a list-like structure in which elements may be inserted or removed from only one end. While this restriction makes stacks less flexible than lists, it also makes stacks both efficient (for those operations they can do) and easy to im- plement. Many applications require only the limited form of insert and remove operations that stacks provide. In such cases, it is more efficient to use the sim- pler stack data structure rather than the generic list. For example, the freelist of Section 4.1.2 is really a stack. Despite their restrictions, stacks have many uses. Thus, a special vocabulary for stacks has developed. Accountants used stacks long before the invention of the computer. They called the stack a “LIFO” list, which stands for “Last-In, First- Sec. 4.2 Stacks 121 // Stack abtract class template class Stack { private: void operator =(const Stack&) {} // Protect assignment Stack(const Stack&) {} // Protect copy constructor public: Stack() {} // Default constructor virtual ˜Stack() {} // Base destructor // Reinitialize the stack. The user is responsible for // reclaiming the storage used by the stack elements. virtual void clear() = 0; // Push an element onto the top of the stack. // it: The element being pushed onto the stack. virtual void push(const E& it) = 0; // Remove the element at the top of the stack. // Return: The element at the top of the stack. virtual E pop() = 0; // Return: A copy of the top element. virtual const E& topValue() const = 0; // Return: The number of elements in the stack. virtual int length() const = 0; }; Figure 4.17 The stack ADT. Out.” Note that one implication of the LIFO policy is that stacks remove elements in reverse order of their arrival. The accessible element of the stack is called the top element. Elements are not said to be inserted, they are pushed onto the stack. When removed, an element is said to be popped from the stack. Figure 4.17 shows a sample stack ADT. As with lists, there are many variations on stack implementation. The two ap- proaches presented here are array-based and linked stacks, which are analogous to array-based and linked lists, respectively. 4.2.1 Array-Based Stacks Figure 4.18 shows a complete implementation for the array-based stack class. As with the array-based list implementation, listArray must be declared of fixed size when the stack is created. In the stack constructor, size serves to indicate this size. Method top acts somewhat like a current position value (because the “current” position is always at the top of the stack), as well as indicating the number of elements currently in the stack. 122 Chap. 4 Lists, Stacks, and Queues // Array-based stack implementation template class AStack: public Stack { private: int maxSize; // Maximum size of stack int top; // Index for top element E *listArray; // Array holding stack elements public: AStack(int size =defaultSize) // Constructor { maxSize = size; top = 0; listArray = new E[size]; } ˜AStack() { delete [] listArray; } // Destructor void clear() { top = 0; } // Reinitialize void push(const E& it) { // Put "it" on stack Assert(top != maxSize, "Stack is full"); listArray[top++] = it; } E pop() { // Pop top element Assert(top != 0, "Stack is empty"); return listArray[--top]; } const E& topValue() const { // Return top element Assert(top != 0, "Stack is empty"); return listArray[top-1]; } int length() const { return top; } // Return length }; Figure 4.18 Array-based stack class implementation. The array-based stack implementation is essentially a simplified version of the array-based list. The only important design decision to be made is which end of the array should represent the top of the stack. One choice is to make the top be at position 0 in the array. In terms of list functions, all insert and remove operations would then be on the element in position 0. This implementation is inefficient, because now every push or pop operation will require that all elements currently in the stack be shifted one position in the array, for a cost of ⇥(n) if there are n elements. The other choice is have the top element be at position n 1 when there are n elements in the stack. In other words, as elements are pushed onto the stack, they are appended to the tail of the list. Method pop removes the tail element. In this case, the cost for each push or pop operation is only ⇥(1). For the implementation of Figure 4.18, top is defined to be the array index of the first free position in the stack. Thus, an empty stack has top set to 0, the first available free position in the array. (Alternatively, top could have been defined to Sec. 4.2 Stacks 123 be the index for the top element in the stack, rather than the first free position. If this had been done, the empty list would initialize top as 1.) Methods push and pop simply place an element into, or remove an element from, the array position indicated by top. Because top is assumed to be at the first free position, push first inserts its value into the top position and then increments top, while pop first decrements top and then removes the top element. 124 Chap. 4 Lists, Stacks, and Queues // Linked stack implementation template class LStack: public Stack { private: Link* top; // Pointer to first element int size; // Number of elements public: LStack(int sz =defaultSize) // Constructor { top = NULL; size = 0; } ˜LStack() { clear(); } // Destructor void clear() { // Reinitialize while (top != NULL) { // Delete link nodes Link* temp = top; top = top->next; delete temp; } size = 0; } void push(const E& it) { // Put "it" on stack top = new Link(it, top); size++; } E pop() { // Remove "it" from stack Assert(top != NULL, "Stack is empty"); E it = top->element; Link* ltemp = top->next; delete top; top = ltemp; size--; return it; } const E& topValue() const { // Return top value Assert(top != 0, "Stack is empty"); return top->element; } int length() const { return size; } // Return length }; Figure 4.19 Linked stack class implementation. 4.2.2 Linked Stacks The linked stack implementation is quite simple. The freelist of Section 4.1.2 is an example of a linked stack. Elements are inserted and removed only from the head of the list. A header node is not used because no special-case code is required for lists of zero or one elements. Figure 4.19 shows the complete linked stack Sec. 4.2 Stacks 125 top1 top2 Figure 4.20 Two stacks implemented within in a single array, both growing toward the middle. implementation. The only data member is top, a pointer to the first (top) link node of the stack. Method push first modifies the next field of the newly created link node to point to the top of the stack and then sets top to point to the new link node. Method pop is also quite simple. Variable temp stores the top nodes’ value, while ltemp links to the top node as it is removed from the stack. The stack is updated by setting top to point to the next link in the stack. The old top node is then returned to free store (or the freelist), and the element value is returned. 4.2.3 Comparison of Array-Based and Linked Stacks All operations for the array-based and linked stack implementations take constant time, so from a time efficiency perspective, neither has a significant advantage. Another basis for comparison is the total space required. The analysis is similar to that done for list implementations. The array-based stack must declare a fixed-size array initially, and some of that space is wasted whenever the stack is not full. The linked stack can shrink and grow but requires the overhead of a link field for every element. When multiple stacks are to be implemented, it is possible to take advantage of the one-way growth of the array-based stack. This can be done by using a single array to store two stacks. One stack grows inward from each end as illustrated by Figure 4.20, hopefully leading to less wasted space. However, this only works well when the space requirements of the two stacks are inversely correlated. In other words, ideally when one stack grows, the other will shrink. This is particularly effective when elements are taken from one stack and given to the other. If instead both stacks grow at the same time, then the free space in the middle of the array will be exhausted quickly. 4.2.4 Implementing Recursion Perhaps the most common computer application that uses stacks is not even visible to its users. This is the implementation of subroutine calls in most programming language runtime environments. A subroutine call is normally implemented by placing necessary information about the subroutine (including the return address, parameters, and local variables) onto a stack. This information is called an ac- tivation record. Further subroutine calls add to the stack. Each return from a subroutine pops the top activation record off the stack. Figure 4.21 illustrates the 126 Chap. 4 Lists, Stacks, and Queues βββ ββ β β β β β1 β β β β ββ11 11 2 2 2 3 44 3 2 3 4 Call fact(1)Call fact(2)Call fact(3)Call fact(4) Return 1 4 3 Return 2 4 Return 6 Return 24 Currptr Currptr Currptr Currptr Currptr Currptr Currptr Currptr Currptr Currptr CurrptrCurrptr CurrptrCurrptr nn nn n n nn n Currptr Currptr Figure 4.21 Implementing recursion with a stack. values indicate the address of the program instruction to return to after completing the current function call. On each recursive function call to fact (as implemented in Section 2.5), both the return address and the current value of n must be saved. Each return from fact pops the top activation record off the stack. implementation of the recursive factorial function of Section 2.5 from the runtime environment’s point of view. Consider what happens when we call fact with the value 4. We use to indicate the address of the program instruction where the call to fact is made. Thus, the stack must first store the address , and the value 4 is passed to fact. Next, a recursive call to fact is made, this time with value 3. We will name the program address from which the call is made 1. The address 1, along with the current value for n (which is 4), is saved on the stack. Function fact is invoked with input parameter 3. In similar manner, another recursive call is made with input parameter 2, re- quiring that the address from which the call is made (say 2) and the current value for n (which is 3) are stored on the stack. A final recursive call with input parame- ter 1 is made, requiring that the stack store the calling address (say 3) and current value (which is 2). Sec. 4.2 Stacks 127 At this point, we have reached the base case for fact, and so the recursion begins to unwind. Each return from fact involves popping the stored value for n from the stack, along with the return address from the function call. The return value for fact is multiplied by the restored value for n, and the result is returned. Because an activation record must be created and placed onto the stack for each subroutine call, making subroutine calls is a relatively expensive operation. While recursion is often used to make implementation easy and clear, sometimes you might want to eliminate the overhead imposed by the recursive function calls. In some cases, such as the factorial function of Section 2.5, recursion can easily be replaced by iteration. Example 4.2 As a simple example of replacing recursion with a stack, consider the following non-recursive version of the factorial function. long fact(int n, Stack& S) { // Compute n! // To fit n! in a long variable, require n <= 12 Assert((n >= 0) && (n <= 12), "Input out of range"); while (n > 1) S.push(n--); // Load up the stack long result = 1; // Holds final result while(S.length() > 0) result = result * S.pop(); // Compute return result; } Here, we simply push successively smaller values of n onto the stack un- til the base case is reached, then repeatedly pop off the stored values and multiply them into the result. An iterative form of the factorial function is both simpler and faster than the version shown in Example 4.2. But it is not always possible to replace recursion with iteration. Recursion, or some imitation of it, is necessary when implementing algorithms that require multiple branching such as in the Towers of Hanoi alg- orithm, or when traversing a binary tree. The Mergesort and Quicksort algorithms of Chapter 7 are also examples in which recursion is required. Fortunately, it is al- ways possible to imitate recursion with a stack. Let us now turn to a non-recursive version of the Towers of Hanoi function, which cannot be done iteratively. Example 4.3 The TOH function shown in Figure 2.2 makes two recursive calls: one to move n 1 rings off the bottom ring, and another to move these n 1 rings back to the goal pole. We can eliminate the recursion by using a stack to store a representation of the three operations that TOH must perform: two recursive calls and a move operation. To do so, we must first come up with a representation of the various operations, implemented as a class whose objects will be stored on the stack. 128 Chap. 4 Lists, Stacks, and Queues // Operation choices: DOMOVE will move a disk // DOTOH corresponds to a recursive call enum TOHop { DOMOVE, DOTOH }; class TOHobj { // An operation object public: TOHop op; // This operation type int num; // How many disks Pole start, goal, tmp; // Define pole order // DOTOH operation constructor TOHobj(int n, Pole s, Pole g, Pole t) { op = DOTOH; num = n; start = s; goal = g; tmp = t; } // DOMOVE operation constructor TOHobj(Pole s, Pole g) { op = DOMOVE; start = s; goal = g; } }; void TOH(int n, Pole start, Pole goal, Pole tmp, Stack& S) { S.push(new TOHobj(n, start, goal, tmp)); // Initial TOHobj* t; while (S.length() > 0) { // Grab next task t = S.pop(); if (t->op == DOMOVE) // Do a move move(t->start, t->goal); else if (t->num > 0) { // Store (in reverse) 3 recursive statements int num = t->num; Pole tmp = t->tmp; Pole goal = t->goal; Pole start = t->start; S.push(new TOHobj(num-1, tmp, goal, start)); S.push(new TOHobj(start, goal)); S.push(new TOHobj(num-1, start, tmp, goal)); } delete t; // Must delete the TOHobj we made } } Figure 4.22 Stack-based implementation for Towers of Hanoi. Figure 4.22 shows such a class. We first define an enumerated type called TOHop, with two values MOVE and TOH, to indicate calls to the move function and recursive calls to TOH, respectively. Class TOHobj stores five values: an operation field (indicating either a move or a new TOH operation), the number of rings, and the three poles. Note that the move operation actually needs only to store information about two poles. Thus, there are two constructors: one to store the state when imitating a recursive call, and one to store the state for a move operation. An array-based stack is used because we know that the stack will need to store exactly 2n+1elements. The new version of TOH begins by placing Sec. 4.3 Queues 129 on the stack a description of the initial problem for n rings. The rest of the function is simply a while loop that pops the stack and executes the appropriate operation. In the case of a TOH operation (for n>0), we store on the stack representations for the three operations executed by the recursive version. However, these operations must be placed on the stack in reverse order, so that they will be popped off in the correct order. Recursive algorithms lend themselves to efficient implementation with a stack when the amount of information needed to describe a sub-problem is small. For example, Section 7.5 discusses a stack-based implementation for Quicksort. 4.3 Queues Like the stack, the queue is a list-like structure that provides restricted access to its elements. Queue elements may only be inserted at the back (called an enqueue operation) and removed from the front (called a dequeue operation). Queues oper- ate like standing in line at a movie theater ticket counter.1 If nobody cheats, then newcomers go to the back of the line. The person at the front of the line is the next to be served. Thus, queues release their elements in order of arrival. Accountants have used queues since long before the existence of computers. They call a queue a “FIFO” list, which stands for “First-In, First-Out.” Figure 4.23 shows a sample queue ADT. This section presents two implementations for queues: the array-based queue and the linked queue. 4.3.1 Array-Based Queues The array-based queue is somewhat tricky to implement effectively. A simple con- version of the array-based list implementation is not efficient. Assume that there are n elements in the queue. By analogy to the array-based list implementation, we could require that all elements of the queue be stored in the first n positions of the array. If we choose the rear element of the queue to be in position 0, then dequeue operations require only ⇥(1) time because the front ele- ment of the queue (the one being removed) is the last element in the array. However, enqueue operations will require ⇥(n) time, because the n elements currently in the queue must each be shifted one position in the array. If instead we chose the rear element of the queue to be in position n 1, then an enqueue operation is equivalent to an append operation on a list. This requires only ⇥(1) time. But now, a dequeue operation requires ⇥(n) time, because all of the elements must be shifted down by one position to retain the property that the remaining n 1 queue elements reside in the first n 1 positions of the array. 1In Britain, a line of people is called a “queue,” and getting into line to wait for service is called “queuing up.” 130 Chap. 4 Lists, Stacks, and Queues // Abstract queue class template class Queue { private: void operator =(const Queue&) {} // Protect assignment Queue(const Queue&) {} // Protect copy constructor public: Queue() {} // Default virtual ˜Queue() {} // Base destructor // Reinitialize the queue. The user is responsible for // reclaiming the storage used by the queue elements. virtual void clear() = 0; // Place an element at the rear of the queue. // it: The element being enqueued. virtual void enqueue(const E&) = 0; // Remove and return element at the front of the queue. // Return: The element at the front of the queue. virtual E dequeue() = 0; // Return: A copy of the front element. virtual const E& frontValue() const = 0; // Return: The number of elements in the queue. virtual int length() const = 0; }; Figure 4.23 The C++ ADT for a queue. A far more efficient implementation can be obtained by relaxing the require- ment that all elements of the queue must be in the first n positions of the array. We will still require that the queue be stored be in contiguous array positions, but the contents of the queue will be permitted to drift within the array, as illustrated by Figure 4.24. Now, both the enqueue and the dequeue operations can be performed in ⇥(1) time because no other elements in the queue need be moved. This implementation raises a new problem. Assume that the front element of the queue is initially at position 0, and that elements are added to successively higher-numbered positions in the array. When elements are removed from the queue, the front index increases. Over time, the entire queue will drift toward the higher-numbered positions in the array. Once an element is inserted into the highest-numbered position in the array, the queue has run out of space. This hap- pens despite the fact that there might be free positions at the low end of the array where elements have previously been removed from the queue. The “drifting queue” problem can be solved by pretending that the array is circular and so allow the queue to continue directly from the highest-numbered position in the array to the lowest-numbered position. This is easily implemented Sec. 4.3 Queues 131 front rear 20 5 12 17 (a) rear (b) 12 17 3 30 4 front Figure 4.24 After repeated use, elements in the array-based queue will drift to the back of the array. (a) The queue after the initial four numbers 20, 5, 12, and 17 have been inserted. (b) The queue after elements 20 and 5 are deleted, following which 3, 30, and 4 are inserted. rear front rear (a) (b) 20 5 12 17 12 17 3 30 4 front Figure 4.25 The circular queue with array positions increasing in the clockwise direction. (a) The queue after the initial four numbers 20, 5, 12, and 17 have been inserted. (b) The queue after elements 20 and 5 are deleted, following which 3, 30, and 4 are inserted. through use of the modulus operator (denoted by % in C++). In this way, positions in the array are numbered from 0 through size1, and position size1 is de- fined to immediately precede position 0 (which is equivalent to position size % size). Figure 4.25 illustrates this solution. There remains one more serious, though subtle, problem to the array-based queue implementation. How can we recognize when the queue is empty or full? Assume that front stores the array index for the front element in the queue, and rear stores the array index for the rear element. If both front and rear have the same position, then with this scheme there must be one element in the queue. Thus, an empty queue would be recognized by having rear be one less than front (tak- ing into account the fact that the queue is circular, so position size1 is actually considered to be one less than position 0). But what if the queue is completely full? In other words, what is the situation when a queue with n array positions available 132 Chap. 4 Lists, Stacks, and Queues contains n elements? In this case, if the front element is in position 0, then the rear element is in position size1. But this means that the value for rear is one less than the value for front when the circular nature of the queue is taken into account. In other words, the full queue is indistinguishable from the empty queue! You might think that the problem is in the assumption about front and rear being defined to store the array indices of the front and rear elements, respectively, and that some modification in this definition will allow a solution. Unfortunately, the problem cannot be remedied by a simple change to the definition for front and rear, because of the number of conditions or states that the queue can be in. Ignoring the actual position of the first element, and ignoring the actual values of the elements stored in the queue, how many different states are there? There can be no elements in the queue, one element, two, and so on. At most there can be n elements in the queue if there are n array positions. This means that there are n +1different states for the queue (0 through n elements are possible). If the value of front is fixed, then n+1different values for rear are needed to distinguish among the n+1states. However, there are only n possible values for rear unless we invent a special case for, say, empty queues. This is an example of the Pigeonhole Principle defined in Exercise 2.30. The Pigeonhole Principle states that, given n pigeonholes and n +1pigeons, when all of the pigeons go into the holes we can be sure that at least one hole contains more than one pigeon. In similar manner, we can be sure that two of the n +1states are indistinguishable by the n relative values of front and rear. We must seek some other way to distinguish full from empty queues. One obvious solution is to keep an explicit count of the number of elements in the queue, or at least a Boolean variable that indicates whether the queue is empty or not. Another solution is to make the array be of size n +1, and only allow n elements to be stored. Which of these solutions to adopt is purely a matter of the implementor’s taste in such affairs. My choice is to use an array of size n +1. Figure 4.26 shows an array-based queue implementation. listArray holds the queue elements, and as usual, the queue constructor allows an optional param- eter to set the maximum size of the queue. The array as created is actually large enough to hold one element more than the queue will allow, so that empty queues can be distinguished from full queues. Member maxSize is used to control the circular motion of the queue (it is the base for the modulus operator). Member rear is set to the position of the current rear element, while front is the position of the current front element. In this implementation, the front of the queue is defined to be toward the lower numbered positions in the array (in the counter-clockwise direction in Fig- ure 4.25), and the rear is defined to be toward the higher-numbered positions. Thus, enqueue increments the rear pointer (modulus size), and dequeue increments the front pointer. Implementation of all member functions is straightforward. Sec. 4.3 Queues 133 // Array-based queue implementation template class AQueue: public Queue { private: int maxSize; // Maximum size of queue int front; // Index of front element int rear; // Index of rear element E *listArray; // Array holding queue elements public: AQueue(int size =defaultSize) { // Constructor // Make list array one position larger for empty slot maxSize = size+1; rear = 0; front = 1; listArray = new E[maxSize]; } ˜AQueue() { delete [] listArray; } // Destructor void clear() { rear = 0; front = 1; } // Reinitialize void enqueue(const E& it) { // Put "it" in queue Assert(((rear+2) % maxSize) != front, "Queue is full"); rear = (rear+1) % maxSize; // Circular increment listArray[rear] = it; } E dequeue() { // Take element out Assert(length() != 0, "Queue is empty"); E it = listArray[front]; front = (front+1) % maxSize; // Circular increment return it; } const E& frontValue() const { // Get front value Assert(length() != 0, "Queue is empty"); return listArray[front]; } virtual int length() const // Return length { return ((rear+maxSize) - front + 1) % maxSize; } }; Figure 4.26 An array-based queue implementation. 134 Chap. 4 Lists, Stacks, and Queues 4.3.2 Linked Queues The linked queue implementation is a straightforward adaptation of the linked list. Figure 4.27 shows the linked queue class declaration. Methods front and rear are pointers to the front and rear queue elements, respectively. We will use a header link node, which allows for a simpler implementation of the enqueue operation by avoiding any special cases when the queue is empty. On initialization, the front and rear pointers will point to the header node, and front will always point to the header node while rear points to the true last link node in the queue. Method enqueue places the new element in a link node at the end of the linked list (i.e., the node that rear points to) and then advances rear to point to the new link node. Method dequeue removes and returns the first element of the list. 4.3.3 Comparison of Array-Based and Linked Queues All member functions for both the array-based and linked queue implementations require constant time. The space comparison issues are the same as for the equiva- lent stack implementations. Unlike the array-based stack implementation, there is no convenient way to store two queues in the same array, unless items are always transferred directly from one queue to the other. 4.4 Dictionaries The most common objective of computer programs is to store and retrieve data. Much of this book is about efficient ways to organize collections of data records so that they can be stored and retrieved quickly. In this section we describe a simple interface for such a collection, called a dictionary. The dictionary ADT provides operations for storing records, finding records, and removing records from the collection. This ADT gives us a standard basis for comparing various data structures. Before we can discuss the interface for a dictionary, we must first define the concepts of a key and comparable objects. If we want to search for a given record in a database, how should we describe what we are looking for? A database record could simply be a number, or it could be quite complicated, such as a payroll record with many fields of varying types. We do not want to describe what we are looking for by detailing and matching the entire contents of the record. If we knew every- thing about the record already, we probably would not need to look for it. Instead, we typically define what record we want in terms of a key value. For example, if searching for payroll records, we might wish to search for the record that matches a particular ID number. In this example the ID number is the search key. To implement the search function, we require that keys be comparable. At a minimum, we must be able to take two keys and reliably determine whether they Sec. 4.4 Dictionaries 135 // Linked queue implementation template class LQueue: public Queue { private: Link* front; // Pointer to front queue node Link* rear; // Pointer to rear queue node int size; // Number of elements in queue public: LQueue(int sz =defaultSize) // Constructor { front = rear = new Link(); size = 0; } ˜LQueue() { clear(); delete front; } // Destructor void clear() { // Clear queue while(front->next != NULL) { // Delete each link node rear = front; delete rear; } rear = front; size = 0; } void enqueue(const E& it) { // Put element on rear rear->next = new Link(it, NULL); rear = rear->next; size++; } E dequeue() { // Remove element from front Assert(size != 0, "Queue is empty"); E it = front->next->element; // Store dequeued value Link* ltemp = front->next; // Hold dequeued link front->next = ltemp->next; // Advance front if (rear == ltemp) rear = front; // Dequeue last element delete ltemp; // Delete link size --; return it; // Return element value } const E& frontValue() const { // Get front element Assert(size != 0, "Queue is empty"); return front->next->element; } virtual int length() const { return size; } }; Figure 4.27 Linked queue class implementation. 136 Chap. 4 Lists, Stacks, and Queues are equal or not. That is enough to enable a sequential search through a database of records and find one that matches a given key. However, we typically would like for the keys to define a total order (see Section 2.1), which means that we can tell which of two keys is greater than the other. Using key types with total orderings gives the database implementor the opportunity to organize a collection of records in a way that makes searching more efficient. An example is storing the records in sorted order in an array, which permits a binary search. Fortunately, in practice most fields of most records consist of simple data types with natural total orders. For example, integers, floats, doubles, and character strings all are totally ordered. Ordering fields that are naturally multi-dimensional, such as a point in two or three dimensions, present special opportunities if we wish to take advantage of their multidimensional nature. This problem is addressed in Section 13.3. Figure 4.28 shows the definition for a simple abstract dictionary class. The methods insert and find are the heart of the class. Method insert takes a record and inserts it into the dictionary. Method find takes a key value and returns some record from the dictionary whose key matches the one provided. If there are multiple records in the dictionary with that key value, there is no requirement as to which one is returned. Method clear simply re-initializes the dictionary. The remove method is similar to find, except that it also deletes the record returned from the dictionary. Once again, if there are multiple records in the dictionary that match the desired key, there is no requirement as to which one actually is removed and returned. Method size returns the number of elements in the dictionary. The remaining Method is removeAny. This is similar to remove, except that it does not take a key value. Instead, it removes an arbitrary record from the dictionary, if one exists. The purpose of this method is to allow a user the ability to iterate over all elements in the dictionary (of course, the dictionary will become empty in the process). Without the removeAny method, a dictionary user could not get at a record of the dictionary that he didn’t already know the key value for. With the removeAny method, the user can process all records in the dictionary as shown in the following code fragment. while (dict.size() > 0) { it = dict.removeAny(); doSomething(it); } There are other approaches that might seem more natural for iterating though a dictionary, such as using a “first” and a “next” function. But not all data structures that we want to use to implement a dictionary are able to do “first” efficiently. For example, a hash table implementation cannot efficiently locate the record in the table with the smallest key value. By using RemoveAny, we have a mechanism that provides generic access. Sec. 4.4 Dictionaries 137 // The Dictionary abstract class. template class Dictionary { private: void operator =(const Dictionary&) {} Dictionary(const Dictionary&) {} public: Dictionary() {} // Default constructor virtual ˜Dictionary() {} // Base destructor // Reinitialize dictionary virtual void clear() = 0; // Insert a record // k: The key for the record being inserted. // e: The record being inserted. virtual void insert(const Key& k, const E& e) = 0; // Remove and return a record. // k: The key of the record to be removed. // Return: A maching record. If multiple records match // "k", remove an arbitrary one. Return NULL if no record // with key "k" exists. virtual E remove(const Key& k) = 0; // Remove and return an arbitrary record from dictionary. // Return: The record removed, or NULL if none exists. virtual E removeAny() = 0; // Return: A record matching "k" (NULL if none exists). // If multiple records match, return an arbitrary one. // k: The key of the record to find virtual E find(const Key& k) const = 0; // Return the number of records in the dictionary. virtual int size() = 0; }; Figure 4.28 The ADT for a simple dictionary. 138 Chap. 4 Lists, Stacks, and Queues // A simple payroll entry with ID, name, address fields class Payroll { private: int ID; string name; string address; public: // Constructor Payroll(int inID, string inname, string inaddr) { ID = inID; name = inname; address = inaddr; } ˜Payroll() {} // Destructor // Local data member access functions int getID() { return ID; } string getname() { return name; } string getaddr() { return address; } }; Figure 4.29 A payroll record implementation. Given a database storing records of a particular type, we might want to search for records in multiple ways. For example, we might want to store payroll records in one dictionary that allows us to search by ID, and also store those same records in a second dictionary that allows us to search by name. Figure 4.29 shows an implementation for a payroll record. Class Payroll has multiple fields, each of which might be used as a search key. Simply by varying the type for the key, and using the appropriate field in each record as the key value, we can define a dictionary whose search key is the ID field, another whose search key is the name field, and a third whose search key is the address field. Figure 4.30 shows an example where Payroll objects are stored in two separate dictionaries, one using the ID field as the key and the other using the name field as the key. The fundamental operation for a dictionary is finding a record that matches a given key. This raises the issue of how to extract the key from a record. We would like any given dictionary implementation to support arbitrary record types, so we need some mechanism for extracting keys that is sufficiently general. One approach is to require all record types to support some particular method that returns the key value. For example, in Java the Comparable interface can be used to provide this effect. Unfortunately, this approach does not work when the same record type is meant to be stored in multiple dictionaries, each keyed by a different field of the record. This is typical in database applications. Another, more general approach is to supply a class whose job is to extract the key from the record. Unfortunately, Sec. 4.4 Dictionaries 139 int main() { // IDdict organizes Payroll records by ID UALdict IDdict; // namedict organizes Payroll records by name UALdict namedict; Payroll *foo1, *foo2, *findfoo1, *findfoo2; foo1 = new Payroll(5, "Joe", "Anytown"); foo2 = new Payroll(10, "John", "Mytown"); IDdict.insert(foo1->getID(), foo1); IDdict.insert(foo2->getID(), foo2); namedict.insert(foo1->getname(), foo1); namedict.insert(foo2->getname(), foo2); findfoo1 = IDdict.find(5); if (findfoo1 != NULL) cout << findfoo1; else cout << "NULL "; findfoo2 = namedict.find("John"); if (findfoo2 != NULL) cout << findfoo2; else cout << "NULL "; } Figure 4.30 A dictionary search example. Here, payroll records are stored in two dictionaries, one organized by ID and the other organized by name. Both dictionaries are implemented with an unsorted array-based list. this solution also does not work in all situations, because there are record types for which it is not possible to write a key extraction method.2 The fundamental issue is that the key value for a record is not an intrinsic prop- erty of the record’s class, or of any field within the class. The key for a record is actually a property of the context in which the record is used. A truly general alternative is to explicitly store the key associated with a given record, as a separate field in the dictionary. That is, each entry in the dictionary will contain both a record and its associated key. Such entries are known as key- value pairs. It is typical that storing the key explicitly duplicates some field in the record. However, keys tend to be much smaller than records, so this additional space overhead will not be great. A simple class for representing key-value pairs is shown in Figure 4.31. The insert method of the dictionary class supports the key-value pair implementation because it takes two parameters, a record and its associated key for that dictionary. 2One example of such a situation occurs when we have a collection of records that describe books in a library. One of the fields for such a record might be a list of subject keywords, where the typical record stores a few keywords. Our dictionary might be implemented as a list of records sorted by keyword. If a book contains three keywords, it would appear three times on the list, once for each associated keyword. However, given the record, there is no simple way to determine which keyword on the keyword list triggered this appearance of the record. Thus, we cannot write a function that extracts the key from such a record. 140 Chap. 4 Lists, Stacks, and Queues // Container for a key-value pair template class KVpair { private: Key k; E e; public: // Constructors KVpair() {} KVpair(Key kval, E eval) { k = kval; e = eval; } KVpair(const KVpair& o) // Copy constructor { k = o.k; e = o.e; } void operator =(const KVpair& o) // Assignment operator { k = o.k; e = o.e; } // Data member access functions Key key() { return k; } void setKey(Key ink) { k = ink; } E value() { return e; } }; Figure 4.31 Implementation for a class representing a key-value pair. Now that we have defined the dictionary ADT and settled on the design ap- proach of storing key-value pairs for our dictionary entries, we are ready to consider ways to implement it. Two possibilities would be to use an array-based or linked list. Figure 4.32 shows an implementation for the dictionary using an (unsorted) array-based list. Examining class UALdict (UAL stands for “unsorted array-based list), we can easily see that insert is a constant-time operation, because it simply inserts the new record at the end of the list. However, find, and remove both require ⇥(n) time in the average and worst cases, because we need to do a sequential search. Method remove in particular must touch every record in the list, because once the desired record is found, the remaining records must be shifted down in the list to fill the gap. Method removeAny removes the last record from the list, so this is a constant-time operation. As an alternative, we could implement the dictionary using a linked list. The implementation would be quite similar to that shown in Figure 4.32, and the cost of the functions should be the same asymptotically. Another alternative would be to implement the dictionary with a sorted list. The advantage of this approach would be that we might be able to speed up the find operation by using a binary search. To do so, first we must define a variation on the List ADT to support sorted lists.An implementation for the array-based sorted list is shown in Figure 4.33. A sorted list is somewhat different from an unsorted list in that it cannot permit the user to control where elements get inserted. Thus, Sec. 4.4 Dictionaries 141 // Dictionary implemented with an unsorted array-based list template class UALdict : public Dictionary { private: AList >* list; public: UALdict(int size=defaultSize) // Constructor { list = new AList >(size); } ˜UALdict() { delete list; } // Destructor void clear() { list->clear(); } // Reinitialize // Insert an element: append to list void insert(const Key&k, const E& e) { KVpair temp(k, e); list->append(temp); } // Use sequential search to find the element to remove E remove(const Key& k) { E temp = find(k); // "find" will set list position if(temp != NULL) list->remove(); return temp; } E removeAny() { // Remove the last element Assert(size() != 0, "Dictionary is empty"); list->moveToEnd(); list->prev(); KVpair e = list->remove(); return e.value(); } // Find "k" using sequential search E find(const Key& k) const { for(list->moveToStart(); list->currPos() < list->length(); list->next()) { KVpair temp = list->getValue(); if (k == temp.key()) return temp.value(); } return NULL; // "k" does not appear in dictionary } Figure 4.32 A dictionary implemented with an unsorted array-based list. int size() // Return list size { return list->length(); } }; Figure 4.32 (continued) 142 Chap. 4 Lists, Stacks, and Queues // Sorted array-based list // Inherit from AList as a protected base class template class SAList: protected AList > { public: SAList(int size=defaultSize) : AList >(size) {} ˜SAList() {} // Destructor // Redefine insert function to keep values sorted void insert(KVpair& it) { // Insert at right KVpair curr; for (moveToStart(); currPos() < length(); next()) { curr = getValue(); if(curr.key() > it.key()) break; } AList >::insert(it); // Do AList insert } // With the exception of append, all remaining methods are // exposed from AList. Append is not available to SAlist // class users since it has not been explicitly exposed. AList >::clear; AList >::remove; AList >::moveToStart; AList >::moveToEnd; AList >::prev; AList >::next; AList >::length; AList >::currPos; AList >::moveToPos; AList >::getValue; }; Figure 4.33 An implementation for a sorted array-based list. the insert method must be quite different in a sorted list than in an unsorted list. Likewise, the user cannot be permitted to append elements onto the list. For these reasons, a sorted list cannot be implemented with straightforward inheritance from the List ADT. Class SAList (SAL stands for “sorted array-based list”) does inherit from class AList; however it does so using class AList as a protected base class. This means that SAList has available for its use any member functions of AList, but those member functions are not necessarily available to the user of SAList. How- ever, many of the AList member functions are useful to the SALlist user. Thus, most of the AList member functions are passed along directly to the SAList user without change. For example, the line Sec. 4.4 Dictionaries 143 AList >::remove; provides SAList’s clients with access to the remove method of AList. How- ever, the original insert method from class AList is replaced, and the append method of AList is kept hidden. The dictionary ADT can easily be implemented from class SAList, as shown in Figure 4.34. Method insert for the dictionary simply calls the insert method of the sorted list. Method find uses a generalization of the binary search function originally shown in Section 3.5. The cost for find in a sorted list is ⇥(log n) for a list of length n. This is a great improvement over the cost of find in an unsorted list. Unfortunately, the cost of insert changes from constant time in the unsorted list to ⇥(n) time in the sorted list. Whether the sorted list imple- mentation for the dictionary ADT is more or less efficient than the unsorted list implementation depends on the relative number of insert and find operations to be performed. If many more find operations than insert operations are used, then it might be worth using a sorted list to implement the dictionary. In both cases, remove requires ⇥(n) time in the worst and average cases. Even if we used bi- nary search to cut down on the time to find the record prior to removal, we would still need to shift down the remaining records in the list to fill the gap left by the remove operation. Given two keys, we have not properly addressed the issue of how to compare them. One possibility would be to simply use the basic ==, <=, and >= operators built into C++. This is the approach taken by our implementations for dictionaries shown inFigures 4.32 and 4.34. If the key type is int, for example, this will work fine. However, if the key is a pointer to a string or any other type of object, then this will not give the desired result. When we compare two strings we probably want to know which comes first in alphabetical order, but what we will get from the standard comparison operators is simply which object appears first in memory. Unfortunately, the code will compile fine, but the answers probably will not be fine. In a language like C++ that supports operator overloading, we could require that the user of the dictionary overload the ==, <=, and >= operators for the given key type. This requirement then becomes an obligation on the user of the dictionary class. Unfortunately, this obligation is hidden within the code of the dictionary (and possibly in the user’s manual) rather than exposed in the dictionary’s interface. As a result, some users of the dictionary might neglect to implement the overloading, with unexpected results. Again, the compiler will not catch this problem. The most general solution is to have users supply their own definition for com- paring keys. The concept of a class that does comparison (called a comparator) is quite important. By making these operations be template parameters, the require- ment to supply the comparator class becomes part of the interface. This design is an example of the Strategy design pattern, because the “strategies” for comparing and getting keys from records are provided by the client. In some cases, it makes sense 144 Chap. 4 Lists, Stacks, and Queues // Dictionary implemented with a sorted array-based list template class SALdict : public Dictionary { private: SAList* list; public: SALdict(int size=defaultSize) // Constructor { list = new SAList(size); } ˜SALdict() { delete list; } // Destructor void clear() { list->clear(); } // Reinitialize // Insert an element: Keep elements sorted void insert(const Key&k, const E& e) { KVpair temp(k, e); list->insert(temp); } // Use sequential search to find the element to remove E remove(const Key& k) { E temp = find(k); if (temp != NULL) list->remove(); return temp; } E removeAny() { // Remove the last element Assert(size() != 0, "Dictionary is empty"); list->moveToEnd(); list->prev(); KVpair e = list->remove(); return e.value(); } // Find "K" using binary search E find(const Key& k) const { int l = -1; int r = list->length(); while (l+1 != r) { // Stop when l and r meet int i = (l+r)/2; // Check middle of remaining subarray list->moveToPos(i); KVpair temp = list->getValue(); if (k < temp.key()) r = i; // In left if (k == temp.key()) return temp.value(); // Found it if (k > temp.key()) l = i; // In right } return NULL; // "k" does not appear in dictionary } Figure 4.34 Dictionary implementation using a sorted array-based list. Sec. 4.5 Further Reading 145 int size() // Return list size { return list->length(); } }; Figure 4.34 (continued) for the comparator class to extract the key from the record type, as an alternative to storing key-value pairs. Here is an example of the required class for comparing two integers. class intintCompare { // Comparator class for integer keys public: static bool lt(int x, int y) { return x < y; } static bool eq(int x, int y) { return x == y; } static bool gt(int x, int y) { return x > y; } }; Class intintCompare provides methods for determining if two int variables are equal (eq), or if the first is less than the second (lt), or greater than the second (gt). Here is a class for comparing two C-style character strings. It makes use of the standard library function strcmp to do the actual comparison. class CCCompare { // Compare two character strings public: static bool lt(char* x, char* y) { return strcmp(x, y) < 0; } static bool eq(char* x, char* y) { return strcmp(x, y) == 0; } static bool gt(char* x, char* y) { return strcmp(x, y) > 0; } }; We will usea comparator in Section 5.5 to implement comparison in heaps, and in Chapter 7 to implement comparison in sorting algorithms. 4.5 Further Reading For more discussion on choice of functions used to define the List ADT, see the work of the Reusable Software Research Group from Ohio State. Their definition for the List ADT can be found in [SWH93]. More information about designing such classes can be found in [SW94]. 4.6 Exercises 4.1 Assume a list has the following configuration: h|2, 23, 15, 5, 9 i. 146 Chap. 4 Lists, Stacks, and Queues Write a series of C++ statements using the List ADT of Figure 4.1 to delete the element with value 15. 4.2 Show the list configuration resulting from each series of list operations using the List ADT of Figure 4.1. Assume that lists L1 and L2 are empty at the beginning of each series. Show where the current position is in the list. (a) L1.append(10); L1.append(20); L1.append(15); (b) L2.append(10); L2.append(20); L2.append(15); L2.moveToStart(); L2.insert(39);; L2.insert(12); 4.3 Write a series of C++ statements that uses the List ADT of Figure 4.1 to create a list capable of holding twenty elements and which actually stores the list with the following configuration: h 2, 23 | 15, 5, 9 i. 4.4 Using the list ADT of Figure 4.1, write a function to interchange the current element and the one following it. 4.5 In the linked list implementation presented in Section 4.1.2, the current po- sition is implemented using a pointer to the element ahead of the logical current node. The more “natural” approach might seem to be to have curr point directly to the node containing the current element. However, if this was done, then the pointer of the node preceding the current one cannot be updated properly because there is no access to this node from curr. An alternative is to add a new node after the current element, copy the value of the current element to this new node, and then insert the new value into the old current node. (a) What happens if curr is at the end of the list already? Is there still a way to make this work? Is the resulting code simpler or more complex than the implementation of Section 4.1.2? (b) Will deletion always work in constant time if curr points directly to the current node? In particular, can you make several deletions in a row? Sec. 4.6 Exercises 147 4.6 Add to the LList class implementation a member function to reverse the order of the elements on the list. Your algorithm should run in ⇥(n) time for a list of n elements. 4.7 Write a function to merge two linked lists. The input lists have their elements in sorted order, from lowest to highest. The output list should also be sorted from lowest to highest. Your algorithm should run in linear time on the length of the output list. 4.8 A circular linked list is one in which the next field for the last link node of the list points to the first link node of the list. This can be useful when you wish to have a relative positioning for elements, but no concept of an absolute first or last position. (a) Modify the code of Figure 4.8 to implement circular singly linked lists. (b) Modify the code of Figure 4.14 to implement circular doubly linked lists. 4.9 Section 4.1.3 states “the space required by the array-based list implementa- tion is ⌦(n), but can be greater.” Explain why this is so. 4.10 Section 4.1.3 presents an equation for determining the break-even point for the space requirements of two implementations of lists. The variables are D, E, P, and n. What are the dimensional units for each variable? Show that both sides of the equation balance in terms of their dimensional units. 4.11 Use the space equation of Section 4.1.3 to determine the break-even point for an array-based list and linked list implementation for lists when the sizes for the data field, a pointer, and the array-based list’s array are as specified. State when the linked list needs less space than the array. (a) The data field is eight bytes, a pointer is four bytes, and the array holds twenty elements. (b) The data field is two bytes, a pointer is four bytes, and the array holds thirty elements. (c) The data field is one byte, a pointer is four bytes, and the array holds thirty elements. (d) The data field is 32 bytes, a pointer is four bytes, and the array holds forty elements. 4.12 Determine the size of an int variable, a double variable, and a pointer on your computer. (The C++ operator sizeof might be useful here if you do not already know the answer.) (a) Calculate the break-even point, as a function of n, beyond which the array-based list is more space efficient than the linked list for lists whose elements are of type int. (b) Calculate the break-even point, as a function of n, beyond which the array-based list is more space efficient than the linked list for lists whose elements are of type double. 148 Chap. 4 Lists, Stacks, and Queues 4.13 Modify the code of Figure 4.18 to implement two stacks sharing the same array, as shown in Figure 4.20. 4.14 Modify the array-based queue definition of Figure 4.26 to use a separate Boolean member to keep track of whether the queue is empty, rather than require that one array position remain empty. 4.15 A palindrome is a string that reads the same forwards as backwards. Using only a fixed number of stacks and queues, the stack and queue ADT func- tions, and a fixed number of int and char variables, write an algorithm to determine if a string is a palindrome. Assume that the string is read from standard input one character at a time. The algorithm should output true or false as appropriate. 4.16 Re-implement function fibr from Exercise 2.11, using a stack to replace the recursive call as described in Section 4.2.4. 4.17 Write a recursive algorithm to compute the value of the recurrence relation T(n)=T(dn/2e)+T(bn/2c)+n; T(1) = 1. Then, rewrite your algorithm to simulate the recursive calls with a stack. 4.18 Let Q be a non-empty queue, and let S be an empty stack. Using only the stack and queue ADT functions and a single element variable X, write an algorithm to reverse the order of the elements in Q. 4.19 A common problem for compilers and text editors is to determine if the parentheses (or other brackets) in a string are balanced and properly nested. For example, the string “((())())()” contains properly nested pairs of paren- theses, but the string “)()(” does not, and the string “())” does not contain properly matching parentheses. (a) Give an algorithm that returns true if a string contains properly nested and balanced parentheses, and false otherwise. Use a stack to keep track of the number of left parentheses seen so far. Hint: At no time while scanning a legal string from left to right will you have encoun- tered more right parentheses than left parentheses. (b) Give an algorithm that returns the position in the string of the first of- fending parenthesis if the string is not properly nested and balanced. That is, if an excess right parenthesis is found, return its position; if there are too many left parentheses, return the position of the first ex- cess left parenthesis. Return 1 if the string is properly balanced and nested. Use a stack to keep track of the number and positions of left parentheses seen so far. 4.20 Imagine that you are designing an application where you need to perform the operations Insert, Delete Maximum, and Delete Minimum. For this application, the cost of inserting is not important, because it can be done Sec. 4.7 Projects 149 off-line prior to startup of the time-critical section, but the performance of the two deletion operations are critical. Repeated deletions of either kind must work as fast as possible. Suggest a data structure that can support this application, and justify your suggestion. What is the time complexity for each of the three key operations? 4.21 Write a function that reverses the order of an array of n items. 4.7 Projects 4.1 A deque (pronounced “deck”) is like a queue, except that items may be added and removed from both the front and the rear. Write either an array-based or linked implementation for the deque. 4.2 One solution to the problem of running out of space for an array-based list implementation is to replace the array with a larger array whenever the origi- nal array overflows. A good rule that leads to an implementation that is both space and time efficient is to double the current size of the array when there is an overflow. Re-implement the array-based List class of Figure 4.2 to support this array-doubling rule. 4.3 Use singly linked lists to implement integers of unlimited size. Each node of the list should store one digit of the integer. You should implement addition, subtraction, multiplication, and exponentiation operations. Limit exponents to be positive integers. What is the asymptotic running time for each of your operations, expressed in terms of the number of digits for the two operands of each function? 4.4 Implement doubly linked lists by storing the sum of the next and prev pointers in a single pointer variable as described in Example 4.1. 4.5 Implement a city database using unordered lists. Each database record con- tains the name of the city (a string of arbitrary length) and the coordinates of the city expressed as integer x and y coordinates. Your database should allow records to be inserted, deleted by name or coordinate, and searched by name or coordinate. Another operation that should be supported is to print all records within a given distance of a specified point. Implement the database using an array-based list implementation, and then a linked list im- plementation. Collect running time statistics for each operation in both im- plementations. What are your conclusions about the relative advantages and disadvantages of the two implementations? Would storing records on the list in alphabetical order by city name speed any of the operations? Would keeping the list in alphabetical order slow any of the operations? 4.6 Modify the code of Figure 4.18 to support storing variable-length strings of at most 255 characters. The stack array should have type char. A string is represented by a series of characters (one character per stack element), with 150 Chap. 4 Lists, Stacks, and Queues top = 10 ‘a’ ‘b’ ‘c’ 3 ‘h’ ‘e’ ‘l’ ‘o’ 5 0 1 2 3 4 5 6 7 8 9 10 ‘l’ Figure 4.35 An array-based stack storing variable-length strings. Each position stores either one character or the length of the string immediately to the left of it in the stack. the length of the string stored in the stack element immediately above the string itself, as illustrated by Figure 4.35. The push operation would store an element requiring i storage units in the i positions beginning with the current value of top and store the size in the position i storage units above top. The value of top would then be reset above the newly inserted element. The pop operation need only look at the size value stored in position top1 and then pop off the appropriate number of units. You may store the string on the stack in reverse order if you prefer, provided that when it is popped from the stack, it is returned in its proper order. 4.7 Define an ADT for a bag (see Section 2.1) and create an array-based imple- mentation for bags. Be sure that your bag ADT does not rely in any way on knowing or controlling the position of an element. Then, implement the dictionary ADT of Figure 4.28 using your bag implementation. 4.8 Implement the dictionary ADT of Figure 4.28 using an unsorted linked list as defined by class LList in Figure 4.8. Make the implementation as efficient as you can, given the restriction that your implementation must use the un- sorted linked list and its access operations to implement the dictionary. State the asymptotic time requirements for each function member of the dictionary ADT under your implementation. 4.9 Implement the dictionary ADT of Figure 4.28 based on stacks. Your imple- mentation should declare and use two stacks. 4.10 Implement the dictionary ADT of Figure 4.28 based on queues. Your imple- mentation should declare and use two queues. 5 Binary Trees The list representations of Chapter 4 have a fundamental limitation: Either search or insert can be made efficient, but not both at the same time. Tree structures permit both efficient access and update to large collections of data. Binary trees in particular are widely used and relatively easy to implement. But binary trees are useful for many things besides searching. Just a few examples of applications that trees can speed up include prioritizing jobs, describing mathematical expressions and the syntactic elements of computer programs, or organizing the information needed to drive data compression algorithms. This chapter begins by presenting definitions and some key properties of bi- nary trees. Section 5.2 discusses how to process all nodes of the binary tree in an organized manner. Section 5.3 presents various methods for implementing binary trees and their nodes. Sections 5.4 through 5.6 present three examples of binary trees used in specific applications: the Binary Search Tree (BST) for implementing dictionaries, heaps for implementing priority queues, and Huffman coding trees for text compression. The BST, heap, and Huffman coding tree each have distinctive structural features that affect their implementation and use. 5.1 Definitions and Properties A binary tree is made up of a finite set of elements called nodes. This set either is empty or consists of a node called the root together with two binary trees, called the left and right subtrees, which are disjoint from each other and from the root. (Disjoint means that they have no nodes in common.) The roots of these subtrees are children of the root. There is an edge from a node to each of its children, and a node is said to be the parent of its children. If n1, n2, ..., nk is a sequence of nodes in the tree such that ni is the parent of ni+1 for 1  i class BinNode { public: virtual ˜BinNode() {} // Base destructor // Return the node’s value virtual E& element() = 0; // Set the node’s value virtual void setElement(const E&) = 0; // Return the node’s left child virtual BinNode* left() const = 0; // Set the node’s left child virtual void setLeft(BinNode*) = 0; // Return the node’s right child virtual BinNode* right() const = 0; // Set the node’s right child virtual void setRight(BinNode*) = 0; // Return true if the node is a leaf, false otherwise virtual bool isLeaf() = 0; }; Figure 5.5 A binary tree node ADT. Example 5.1 The preorder enumeration for the tree of Figure 5.1 is ABDCEGFHI. The first node printed is the root. Then all nodes of the left subtree are printed (in preorder) before any node of the right subtree. Alternatively, we might wish to visit each node only after we visit its children (and their subtrees). For example, this would be necessary if we wish to return all nodes in the tree to free store. We would like to delete the children of a node before deleting the node itself. But to do that requires that the children’s children be deleted first, and so on. This is called a postorder traversal. Example 5.2 The postorder enumeration for the tree of Figure 5.1 is DBGEHIFCA. An inorder traversal first visits the left child (including its entire subtree), then visits the node, and finally visits the right child (including its entire subtree). The Sec. 5.2 Binary Tree Traversals 157 binary search tree of Section 5.4 makes use of this traversal to print all nodes in ascending order of value. Example 5.3 The inorder enumeration for the tree of Figure 5.1 is BDAGECHFI. A traversal routine is naturally written as a recursive function. Its input pa- rameter is a pointer to a node which we will call root because each node can be viewed as the root of a some subtree. The initial call to the traversal function passes in a pointer to the root node of the tree. The traversal function visits root and its children (if any) in the desired order. For example, a preorder traversal speci- fies that root be visited before its children. This can easily be implemented as follows. template void preorder(BinNode* root) { if (root == NULL) return; // Empty subtree, do nothing visit(root); // Perform desired action preorder(root->left()); preorder(root->right()); } Function preorder first checks that the tree is not empty (if it is, then the traversal is done and preorder simply returns). Otherwise, preorder makes a call to visit, which processes the root node (i.e., prints the value or performs whatever computation as required by the application). Function preorder is then called recursively on the left subtree, which will visit all nodes in that subtree. Finally, preorder is called on the right subtree, visiting all nodes in the right subtree. Postorder and inorder traversals are similar. They simply change the order in which the node and its children are visited, as appropriate. An important decision in the implementation of any recursive function on trees is when to check for an empty subtree. Function preorder first checks to see if the value for root is NULL. If not, it will recursively call itself on the left and right children of root. In other words, preorder makes no attempt to avoid calling itself on an empty child. Some programmers use an alternate design in which the left and right pointers of the current node are checked so that the recursive call is made only on non-empty children. Such a design typically looks as follows: template void preorder2(BinNode* root) { visit(root); // Perform whatever action is desired if (root->left() != NULL) preorder2(root->left()); if (root->right() != NULL) preorder2(root->right()); } 158 Chap. 5 Binary Trees At first it might appear that preorder2 is more efficient than preorder, because it makes only half as many recursive calls. (Why?) On the other hand, preorder2 must access the left and right child pointers twice as often. The net result is little or no performance improvement. In reality, the design of preorder2 is inferior to that of preorder for two reasons. First, while it is not apparent in this simple example, for more complex traversals it can become awkward to place the check for the NULL pointer in the calling code. Even here we had to write two tests for NULL, rather than the one needed by preorder. The more important concern with preorder2 is that it tends to be error prone. While preorder2 insures that no recursive calls will be made on empty subtrees, it will fail if the initial call passes in a NULL pointer. This would occur if the original tree is empty. To avoid the bug, either preorder2 needs an additional test for a NULL pointer at the beginning (making the subsequent tests redundant after all), or the caller of preorder2 has a hidden obligation to pass in a non-empty tree, which is unreliable design. The net result is that many programmers forget to test for the possibility that the empty tree is being traversed. By using the first design, which explicitly supports processing of empty subtrees, the problem is avoided. Another issue to consider when designing a traversal is how to define the visitor function that is to be executed on every node. One approach is simply to write a new version of the traversal for each such visitor function as needed. The disad- vantage to this is that whatever function does the traversal must have access to the BinNode class. It is probably better design to permit only the tree class to have access to the BinNode class. Another approach is for the tree class to supply a generic traversal function which takes the visitor either as a template parameter or as a function parameter. This is known as the visitor design pattern. A major constraint on this approach is that the signature for all visitor functions, that is, their return type and parameters, must be fixed in advance. Thus, the designer of the generic traversal function must be able to adequately judge what parameters and return type will likely be needed by potential visitor functions. Handling information flow between parts of a program can be a significant design challenge, especially when dealing with recursive functions such as tree traversals. In general, we can run into trouble either with passing in the correct information needed by the function to do its work, or with returning information to the recursive function’s caller. We will see many examples throughout the book that illustrate methods for passing information in and out of recursive functions as they traverse a tree structure. Here are a few simple examples. First we consider the simple case where a computation requires that we com- municate information back up the tree to the end user. Sec. 5.2 Binary Tree Traversals 159 20 50 40 75 20 to 40 Figure 5.6 To be a binary search tree, the left child of the node with value 40 must have a value between 20 and 40. Example 5.4 We wish to count the number of nodes in a binary tree. The key insight is that the total count for any (non-empty) subtree is one for the root plus the counts for the left and right subtrees. Where do left and right subtree counts come from? Calls to function count on the subtrees will compute this for us. Thus, we can implement count as follows. template int count(BinNode* root) { if (root == NULL) return 0; // Nothing to count return 1 + count(root->left()) + count(root->right()); } Another problem that occurs when recursively processing data collections is controlling which members of the collection will be visited. For example, some tree “traversals” might in fact visit only some tree nodes, while avoiding processing of others. Exercise 5.20 must solve exactly this problem in the context of a binary search tree. It must visit only those children of a given node that might possibly fall within a given range of values. Fortunately, it requires only a simple local calculation to determine which child(ren) to visit. A more difficult situation is illustrated by the following problem. Given an arbitrary binary tree we wish to determine if, for every node A, are all nodes in A’s left subtree less than the value of A, and are all nodes in A’s right subtree greater than the value of A? (This happens to be the definition for a binary search tree, described in Section 5.4.) Unfortunately, to make this decision we need to know some context that is not available just by looking at the node’s parent or children. As shown by Figure 5.6, it is not enough to verify that A’s left child has a value less than that of A, and that A’s right child has a greater value. Nor is it enough to verify that A has a value consistent with that of its parent. In fact, we need to know information about what range of values is legal for a given node. That information might come from any of the node’s ancestors. Thus, relevant range information must be passed down the tree. We can implement this function as follows. 160 Chap. 5 Binary Trees template bool checkBST(BSTNode* root, Key low, Key high) { if (root == NULL) return true; // Empty subtree Key rootkey = root->key(); if ((rootkey < low) || (rootkey > high)) return false; // Out of range if (!checkBST(root->left(), low, rootkey)) return false; // Left side failed return checkBST(root->right(), rootkey, high); } 5.3 Binary Tree Node Implementations In this section we examine ways to implement binary tree nodes. We begin with some options for pointer-based binary tree node implementations. Then comes a discussion on techniques for determining the space requirements for a given imple- mentation. The section concludes with an introduction to the array-based imple- mentation for complete binary trees. 5.3.1 Pointer-Based Node Implementations By definition, all binary tree nodes have two children, though one or both children can be empty. Binary tree nodes typically contain a value field, with the type of the field depending on the application. The most common node implementation includes a value field and pointers to the two children. Figure 5.7 shows a simple implementation for the BinNode abstract class, which we will name BSTNode. Class BSTNode includes a data member of type E, (which is the second template parameter) for the element type. To support search structures such as the Binary Search Tree, an additional field is included, with cor- responding access methods, to store a key value (whose purpose is explained in Section 4.4). Its type is determined by the first template parameter, named Key. Every BSTNode object also has two pointers, one to its left child and another to its right child. Overloaded new and delete operators could be added to support a freelist, as described in Section 4.1.2.Figure 5.8 illustrates the BSTNode imple- mentation. Some programmers find it convenient to add a pointer to the node’s parent, allowing easy upward movement in the tree. Using a parent pointer is somewhat analogous to adding a link to the previous node in a doubly linked list. In practice, the parent pointer is almost always unnecessary and adds to the space overhead for the tree implementation. It is not just a problem that parent pointers take space. More importantly, many uses of the parent pointer are driven by improper under- standing of recursion and so indicate poor programming. If you are inclined toward using a parent pointer, consider if there is a more efficient implementation possible. Sec. 5.3 Binary Tree Node Implementations 161 // Simple binary tree node implementation template class BSTNode : public BinNode { private: Key k; // The node’s key E it; // The node’s value BSTNode* lc; // Pointer to left child BSTNode* rc; // Pointer to right child public: // Two constructors -- with and without initial values BSTNode() { lc = rc = NULL; } BSTNode(Key K, E e, BSTNode* l =NULL, BSTNode* r =NULL) { k = K; it = e; lc = l; rc = r; } ˜BSTNode() {} // Destructor // Functions to set and return the value and key E& element() { return it; } void setElement(const E& e) { it = e; } Key& key() { return k; } void setKey(const Key& K) { k = K; } // Functions to set and return the children inline BSTNode* left() const { return lc; } void setLeft(BinNode* b) { lc = (BSTNode*)b; } inline BSTNode* right() const { return rc; } void setRight(BinNode* b) { rc = (BSTNode*)b; } // Return true if it is a leaf, false otherwise bool isLeaf() { return (lc == NULL) && (rc == NULL); } }; Figure 5.7 A binary tree node class implementation. A C GH ED B F I Figure 5.8 Illustration of a typical pointer-based binary tree implementation, where each node stores two child pointers and a value. 162 Chap. 5 Binary Trees 4 x x c a 2 * * * − + Figure 5.9 An expression tree for 4x(2x + a) c. An important decision in the design of a pointer-based node implementation is whether the same class definition will be used for leaves and internal nodes. Using the same class for both will simplify the implementation, but might be an inefficient use of space. Some applications require data values only for the leaves. Other applications require one type of value for the leaves and another for the in- ternal nodes. Examples include the binary trie of Section 13.1, the PR quadtree of Section 13.3, the Huffman coding tree of Section 5.6, and the expression tree illus- trated by Figure 5.9. By definition, only internal nodes have non-empty children. If we use the same node implementation for both internal and leaf nodes, then both must store the child pointers. But it seems wasteful to store child pointers in the leaf nodes. Thus, there are many reasons why it can save space to have separate implementations for internal and leaf nodes. As an example of a tree that stores different information at the leaf and inter- nal nodes, consider the expression tree illustrated by Figure 5.9. The expression tree represents an algebraic expression composed of binary operators such as ad- dition, subtraction, multiplication, and division. Internal nodes store operators, while the leaves store operands. The tree of Figure 5.9 represents the expression 4x(2x + a) c. The storage requirements for a leaf in an expression tree are quite different from those of an internal node. Internal nodes store one of a small set of operators, so internal nodes could store a small code identifying the operator such as a single byte for the operator’s character symbol. In contrast, leaves store vari- able names or numbers, which is considerably larger in order to handle the wider range of possible values. At the same time, leaf nodes need not store child pointers. C++ allows us to differentiate leaf from internal nodes through the use of class inheritance. A base class provides a general definition for an object, and a subclass modifies a base class to add more detail. A base class can be declared for binary tree nodes in general, with subclasses defined for the internal and leaf nodes. The base class of Figure 5.10 is named VarBinNode. It includes a virtual member function Sec. 5.3 Binary Tree Node Implementations 163 named isLeaf, which indicates the node type. Subclasses for the internal and leaf node types each implement isLeaf. Internal nodes store child pointers of the base class type; they do not distinguish their children’s actual subclass. Whenever a node is examined, its version of isLeaf indicates the node’s subclass. Figure 5.10 includes two subclasses derived from class VarBinNode, named LeafNode and IntlNode. Class IntlNode can access its children through pointers of type VarBinNode. Function traverse illustrates the use of these classes. When traverse calls method isLeaf, C++’s runtime environment determines which subclass this particular instance of rt happens to be and calls that subclass’s version of isLeaf. Method isLeaf then provides the actual node type to its caller. The other member functions for the derived subclasses are accessed by type-casting the base class pointer as appropriate, as shown in function traverse. There is another approach that we can take to represent separate leaf and inter- nal nodes, also using a virtual base class and separate node classes for the two types. This is to implement nodes using the composite design pattern. This approach is noticeably different from the one of Figure 5.10 in that the node classes themselves implement the functionality of traverse. Figure 5.11 shows the implementa- tion. Here, base class VarBinNode declares a member function traverse that each subclass must implement. Each subclass then implements its own appropriate behavior for its role in a traversal. The whole traversal process is called by invoking traverse on the root node, which in turn invokes traverse on its children. When comparing the implementations of Figures 5.10 and 5.11, each has ad- vantages and disadvantages. The first does not require that the node classes know about the traverse function. With this approach, it is easy to add new methods to the tree class that do other traversals or other operations on nodes of the tree. However, we see that traverse in Figure 5.10 does need to be familiar with each node subclass. Adding a new node subclass would therefore require modifications to the traverse function. In contrast, the approach of Figure 5.11 requires that any new operation on the tree that requires a traversal also be implemented in the node subclasses. On the other hand, the approach of Figure 5.11 avoids the need for the traverse function to know anything about the distinct abilities of the node subclasses. Those subclasses handle the responsibility of performing a traversal on themselves. A secondary benefit is that there is no need for traverse to explic- itly enumerate all of the different node subclasses, directing appropriate action for each. With only two node classes this is a minor point. But if there were many such subclasses, this could become a bigger problem. A disadvantage is that the traversal operation must not be called on a NULL pointer, because there is no object to catch the call. This problem could be avoided by using a flyweight (see Section 1.3.1) to implement empty nodes. Typically, the version of Figure 5.10 would be preferred in this example if traverse is a member function of the tree class, and if the node subclasses are 164 Chap. 5 Binary Trees // Node implementation with simple inheritance class VarBinNode { // Node abstract base class public: virtual ˜VarBinNode() {} virtual bool isLeaf() = 0; // Subclasses must implement }; class LeafNode : public VarBinNode { // Leaf node private: Operand var; // Operand value public: LeafNode(const Operand& val) { var = val; } // Constructor bool isLeaf() { return true; } // Version for LeafNode Operand value() { return var; } // Return node value }; class IntlNode : public VarBinNode { // Internal node private: VarBinNode* left; // Left child VarBinNode* right; // Right child Operator opx; // Operator value public: IntlNode(const Operator& op, VarBinNode* l, VarBinNode* r) { opx = op; left = l; right = r; } // Constructor bool isLeaf() { return false; } // Version for IntlNode VarBinNode* leftchild() { return left; } // Left child VarBinNode* rightchild() { return right; } // Right child Operator value() { return opx; } // Value }; void traverse(VarBinNode *root) { // Preorder traversal if (root == NULL) return; // Nothing to visit if (root->isLeaf()) // Do leaf node cout << "Leaf: " << ((LeafNode *)root)->value() << endl; else { // Do internal node cout << "Internal: " << ((IntlNode *)root)->value() << endl; traverse(((IntlNode *)root)->leftchild()); traverse(((IntlNode *)root)->rightchild()); } } Figure 5.10 An implementation for separate internal and leaf node representa- tions using C++ class inheritance and virtual functions. Sec. 5.3 Binary Tree Node Implementations 165 // Node implementation with the composite design pattern class VarBinNode { // Node abstract base class public: virtual ˜VarBinNode() {} // Generic destructor virtual bool isLeaf() = 0; virtual void traverse() = 0; }; class LeafNode : public VarBinNode { // Leaf node private: Operand var; // Operand value public: LeafNode(const Operand& val) { var = val; } // Constructor bool isLeaf() { return true; } // isLeaf for Leafnode Operand value() { return var; } // Return node value void traverse() { cout << "Leaf: " << value() << endl; } }; class IntlNode : public VarBinNode { // Internal node private: VarBinNode* lc; // Left child VarBinNode* rc; // Right child Operator opx; // Operator value public: IntlNode(const Operator& op, VarBinNode* l, VarBinNode* r) { opx = op; lc = l; rc = r; } // Constructor bool isLeaf() { return false; } // isLeaf for IntlNode VarBinNode* left() { return lc; } // Left child VarBinNode* right() { return rc; } // Right child Operator value() { return opx; } // Value void traverse() { // Traversal behavior for internal nodes cout << "Internal: " << value() << endl; if (left() != NULL) left()->traverse(); if (right() != NULL) right()->traverse(); } }; // Do a preorder traversal void traverse(VarBinNode *root) { if (root != NULL) root->traverse(); } Figure 5.11 A second implementation for separate internal and leaf node repre- sentations using C++ class inheritance and virtual functions using the composite design pattern. Here, the functionality of traverse is embedded into the node subclasses. 166 Chap. 5 Binary Trees hidden from users of that tree class. On the other hand, if the nodes are objects that have meaning to users of the tree separate from their existence as nodes in the tree, then the version of Figure 5.11 might be preferred because hiding the internal behavior of the nodes becomes more important. Another advantage of the composite design is that implementing each node type’s functionality might be easier. This is because you can focus solely on the information passing and other behavior needed by this node type to do its job. This breaks down the complexity that many programmers feel overwhelmed by when dealing with complex information flows related to recursive processing. 5.3.2 Space Requirements This section presents techniques for calculating the amount of overhead required by a binary tree implementation. Recall that overhead is the amount of space necessary to maintain the data structure. In other words, it is any space not used to store data records. The amount of overhead depends on several factors including which nodes store data values (all nodes, or just the leaves), whether the leaves store child pointers, and whether the tree is a full binary tree. In a simple pointer-based implementation for the binary tree such as that of Figure 5.7, every node has two pointers to its children (even when the children are NULL). This implementation requires total space amounting to n(2P + D) for a tree of n nodes. Here, P stands for the amount of space required by a pointer, and D stands for the amount of space required by a data value. The total overhead space will be 2Pnfor the entire tree. Thus, the overhead fraction will be 2P/(2P + D). The actual value for this expression depends on the relative size of pointers versus data fields. If we arbitrarily assume that P = D, then a full tree has about two thirds of its total space taken up in overhead. Worse yet, Theorem 5.2 tells us that about half of the pointers are “wasted” NULL values that serve only to indicate tree structure, but which do not provide access to new data. A common implementation is not to store any actual data in a node, but rather a pointer to the data record. In this case, each node will typically store three pointers, all of which are overhead, resulting in an overhead fraction of 3P/(3P + D). If only leaves store data values, then the fraction of total space devoted to over- head depends on whether the tree is full. If the tree is not full, then conceivably there might only be one leaf node at the end of a series of internal nodes. Thus, the overhead can be an arbitrarily high percentage for non-full binary trees. The overhead fraction drops as the tree becomes closer to full, being lowest when the tree is truly full. In this case, about one half of the nodes are internal. Great savings can be had by eliminating the pointers from leaf nodes in full binary trees. Again assume the tree stores a pointer to the data field. Because about half of the nodes are leaves and half internal nodes, and because only internal nodes Sec. 5.3 Binary Tree Node Implementations 167 now have child pointers, the overhead fraction in this case will be approximately n 2 (2P) n 2 (2P)+Dn = P P + D. If P = D, the overhead drops to about one half of the total space. However, if only leaf nodes store useful information, the overhead fraction for this implementation is actually three quarters of the total space, because half of the “data” space is unused. If a full binary tree needs to store data only at the leaf nodes, a better imple- mentation would have the internal nodes store two pointers and no data field while the leaf nodes store only a pointer to the data field. This implementation requires n 2 2P+ n 2 (p+d) units of space. If P = D, then the overhead is 3P/(3P+D)=3/4. It might seem counter-intuitive that the overhead ratio has gone up while the total amount of space has gone down. The reason is because we have changed our defini- tion of “data” to refer only to what is stored in the leaf nodes, so while the overhead fraction is higher, it is from a total storage requirement that is lower. There is one serious flaw with this analysis. When using separate implemen- tations for internal and leaf nodes, there must be a way to distinguish between the node types. When separate node types are implemented via C++ subclasses, the runtime environment stores information with each object allowing it to deter- mine, for example, the correct subclass to use when the isLeaf virtual function is called. Thus, each node requires additional space. Only one bit is truly necessary to distinguish the two possibilities. In rare applications where space is a critical resource, implementors can often find a spare bit within the node’s value field in which to store the node type indicator. An alternative is to use a spare bit within a node pointer to indicate node type. For example, this is often possible when the compiler requires that structures and objects start on word boundaries, leaving the last bit of a pointer value always zero. Thus, this bit can be used to store the node- type flag and is reset to zero before the pointer is dereferenced. Another alternative when the leaf value field is smaller than a pointer is to replace the pointer to a leaf with that leaf’s value. When space is limited, such techniques can make the differ- ence between success and failure. In any other situation, such “bit packing” tricks should be avoided because they are difficult to debug and understand at best, and are often machine dependent at worst.2 2In the early to mid 1980s, I worked on a Geographic Information System that stored spatial data in quadtrees (see Section 13.3). At the time space was a critical resource, so we used a bit-packing approach where we stored the nodetype flag as the last bit in the parent node’s pointer. This worked perfectly on various 32-bit workstations. Unfortunately, in those days IBM PC-compatibles used 16-bit pointers. We never did figure out how to port our code to the 16-bit machine. 168 Chap. 5 Binary Trees 5.3.3 Array Implementation for Complete Binary Trees The previous section points out that a large fraction of the space in a typical binary tree node implementation is devoted to structural overhead, not to storing data. This section presents a simple, compact implementation for complete binary trees. Recall that complete binary trees have all levels except the bottom filled out com- pletely, and the bottom level has all of its nodes filled in from left to right. Thus, a complete binary tree of n nodes has only one possible shape. You might think that a complete binary tree is such an unusual occurrence that there is no reason to develop a special implementation for it. However, the complete binary tree has practical uses, the most important being the heap data structure discussed in Sec- tion 5.5. Heaps are often used to implement priority queues (Section 5.5) and for external sorting algorithms (Section 8.5.2). We begin by assigning numbers to the node positions in the complete binary tree, level by level, from left to right as shown in Figure 5.12(a). An array can store the tree’s data values efficiently, placing each data value in the array position corresponding to that node’s position within the tree. Figure 5.12(b) lists the array indices for the children, parent, and siblings of each node in Figure 5.12(a). From Figure 5.12(b), you should see a pattern regarding the positions of a node’s relatives within the array. Simple formulas can be derived for calculating the array index for each relative of a node r from r’s index. No explicit pointers are necessary to reach a node’s left or right child. This means there is no overhead to the array implementation if the array is selected to be of size n for a tree of n nodes. The formulae for calculating the array indices of the various relatives of a node are as follows. The total number of nodes in the tree is n. The index of the node in question is r, which must fall in the range 0 to n 1. • Parent(r)=b(r 1)/2c if r 6=0. • Left child(r)=2r +1if 2r +1 class BST : public Dictionary { private: BSTNode* root; // Root of the BST int nodecount; // Number of nodes in the BST // Private "helper" functions void clearhelp(BSTNode*); BSTNode* inserthelp(BSTNode*, const Key&, const E&); BSTNode* deletemin(BSTNode*); BSTNode* getmin(BSTNode*); BSTNode* removehelp(BSTNode*, const Key&); E findhelp(BSTNode*, const Key&) const; void printhelp(BSTNode*, int) const; public: BST() { root = NULL; nodecount = 0; } // Constructor ˜BST() { clearhelp(root); } // Destructor void clear() // Reinitialize tree { clearhelp(root); root = NULL; nodecount = 0; } // Insert a record into the tree. // k Key value of the record. // e The record to insert. void insert(const Key& k, const E& e) { root = inserthelp(root, k, e); nodecount++; } // Remove a record from the tree. // k Key value of record to remove. // Return: The record removed, or NULL if there is none. E remove(const Key& k) { E temp = findhelp(root, k); // First find it if (temp != NULL) { root = removehelp(root, k); nodecount--; } return temp; } Figure 5.14 The binary search tree implementation. 172 Chap. 5 Binary Trees // Remove and return the root node from the dictionary. // Return: The record removed, null if tree is empty. E removeAny() { // Delete min value if (root != NULL) { E temp = root->element(); root = removehelp(root, root->key()); nodecount--; return temp; } else return NULL; } // Return Record with key value k, NULL if none exist. // k: The key value to find. */ // Return some record matching "k". // Return true if such exists, false otherwise. If // multiple records match "k", return an arbitrary one. E find(const Key& k) const { return findhelp(root, k); } // Return the number of records in the dictionary. int size() { return nodecount; } void print() const { // Print the contents of the BST if (root == NULL) cout << "The BST is empty.\n"; else printhelp(root, 0); } }; Figure 5.14 (continued) However, the find operation is most easily implemented as a recursive function whose parameters are the root of a subtree and the search key. Member findhelp has the desired form for this recursive subroutine and is implemented as follows. template E BST::findhelp(BSTNode* root, const Key& k) const { if (root == NULL) return NULL; // Empty tree if (k < root->key()) return findhelp(root->left(), k); // Check left else if (k > root->key()) return findhelp(root->right(), k); // Check right else return root->element(); // Found it } Once the desired record is found, it is passed through return values up the chain of recursive calls. If a suitable record is not found, null is returned. Inserting a record with key value k requires that we first find where that record would have been if it were in the tree. This takes us to either a leaf node, or to an Sec. 5.4 Binary Search Trees 173 37 24 2 32 35 42 40 42 120 7 Figure 5.15 An example of BST insertion. A record with value 35 is inserted into the BST of Figure 5.13(a). The node with value 32 becomes the parent of the new node containing 35. internal node with no child in the appropriate direction.3 Call this node R 0. We then add a new node containing the new record as a child of R 0. Figure 5.15 illustrates this operation. The value 35 is added as the right child of the node with value 32. Here is the implementation for inserthelp: template BSTNode* BST::inserthelp( BSTNode* root, const Key& k, const E& it) { if (root == NULL) // Empty tree: create node return new BSTNode(k, it, NULL, NULL); if (k < root->key()) root->setLeft(inserthelp(root->left(), k, it)); else root->setRight(inserthelp(root->right(), k, it)); return root; // Return tree with node inserted } You should pay careful attention to the implementation for inserthelp. Note that inserthelp returns a pointer to a BSTNode. What is being returned is a subtree identical to the old subtree, except that it has been modified to contain the new record being inserted. Each node along a path from the root to the parent of the new node added to the tree will have its appropriate child pointer assigned to it. Except for the last node in the path, none of these nodes will actually change their child’s pointer value. In that sense, many of the assignments seem redundant. However, the cost of these additional assignments is worth paying to keep the inser- tion process simple. The alternative is to check if a given assignment is necessary, which is probably more expensive than the assignment! The shape of a BST depends on the order in which elements are inserted. A new element is added to the BST as a new leaf node, potentially increasing the depth of the tree. Figure 5.13 illustrates two BSTs for a collection of values. It is possible 3This assumes that no node has a key value equal to the one being inserted. If we find a node that duplicates the key value to be inserted, we have two options. If the application does not allow nodes with equal keys, then this insertion should be treated as an error (or ignored). If duplicate keys are allowed, our convention will be to insert the duplicate in the right subtree. 174 Chap. 5 Binary Trees for the BST containing n nodes to be a chain of nodes with height n. This would happen if, for example, all elements were inserted in sorted order. In general, it is preferable for a BST to be as shallow as possible. This keeps the average cost of a BST operation low. Removing a node from a BST is a bit trickier than inserting a node, but it is not complicated if all of the possible cases are considered individually. Before tackling the general node removal process, let us first discuss how to remove from a given subtree the node with the smallest key value. This routine will be used later by the general node removal function. To remove the node with the minimum key value from a subtree, first find that node by continuously moving down the left link until there is no further left link to follow. Call this node S. To remove S, simply have the parent of S change its pointer to point to the right child of S. We know that S has no left child (because if S did have a left child, S would not be the node with minimum key value). Thus, changing the pointer as described will maintain a BST, with S removed. The code for this method, named deletemin, is as follows: template BSTNode* BST:: deletemin(BSTNode* rt) { if (rt->left() == NULL) // Found min return rt->right(); else { // Continue left rt->setLeft(deletemin(rt->left())); return rt; } } Example 5.6 Figure 5.16 illustrates the deletemin process. Beginning at the root node with value 10, deletemin follows the left link until there is no further left link, in this case reaching the node with value 5. The node with value 10 is changed to point to the right child of the node containing the minimum value. This is indicated in Figure 5.16 by a dashed line. A pointer to the node containing the minimum-valued element is stored in pa- rameter S. The return value of the deletemin method is the subtree of the cur- rent node with the minimum-valued node in the subtree removed. As with method inserthelp, each node on the path back to the root has its left child pointer reassigned to the subtree resulting from its call to the deletemin method. A useful companion method is getmin which returns a pointer to the node containing the minimum value in the subtree. Sec. 5.4 Binary Search Trees 175 9 5 20 5 10 subroot Figure 5.16 An example of deleting the node with minimum value. In this tree, the node with minimum value, 5, is the left child of the root. Thus, the root’s left pointer is changed to point to 5’s right child. template BSTNode* BST:: getmin(BSTNode* rt) { if (rt->left() == NULL) return rt; else return getmin(rt->left()); } Removing a node with given key value R from the BST requires that we first find R and then remove it from the tree. So, the first part of the remove operation is a search to find R. Once R is found, there are several possibilities. If R has no children, then R’s parent has its pointer set to NULL. If R has one child, then R’s parent has its pointer set to R’s child (similar to deletemin). The problem comes if R has two children. One simple approach, though expensive, is to set R’s parent to point to one of R’s subtrees, and then reinsert the remaining subtree’s nodes one at a time. A better alternative is to find a value in one of the subtrees that can replace the value in R. Thus, the question becomes: Which value can substitute for the one being re- moved? It cannot be any arbitrary value, because we must preserve the BST prop- erty without making major changes to the structure of the tree. Which value is most like the one being removed? The answer is the least key value greater than (or equal to) the one being removed, or else the greatest key value less than the one being removed. If either of these values replace the one being removed, then the BST property is maintained. Example 5.7 Assume that we wish to remove the value 37 from the BST of Figure 5.13(a). Instead of removing the root node, we remove the node with the least value in the right subtree (using the deletemin operation). This value can then replace the value in the root. In this example we first remove the node with value 40, because it contains the least value in the 176 Chap. 5 Binary Trees 37 40 24 7 32 42 40 42 1202 Figure 5.17 An example of removing the value 37 from the BST. The node containing this value has two children. We replace value 37 with the least value from the node’s right subtree, in this case 40. right subtree. We then substitute 40 as the new value for the root node. Figure 5.17 illustrates this process. When duplicate node values do not appear in the tree, it makes no difference whether the replacement is the greatest value from the left subtree or the least value from the right subtree. If duplicates are stored, then we must select the replacement from the right subtree. To see why, call the greatest value in the left subtree G. If multiple nodes in the left subtree have value G, selecting G as the replacement value for the root of the subtree will result in a tree with equal values to the left of the node now containing G. Precisely this situation occurs if we replace value 120 with the greatest value in the left subtree of Figure 5.13(b). Selecting the least value from the right subtree does not have a similar problem, because it does not violate the Binary Search Tree Property if equal values appear in the right subtree. From the above, we see that if we want to remove the record stored in a node with two children, then we simply call deletemin on the node’s right subtree and substitute the record returned for the record being removed. Figure 5.18 shows an implementation for removehelp. The cost for findhelp and inserthelp is the depth of the node found or inserted. The cost for removehelp is the depth of the node being removed, or in the case when this node has two children, the depth of the node with smallest value in its right subtree. Thus, in the worst case, the cost for any one of these operations is the depth of the deepest node in the tree. This is why it is desirable to keep BSTs balanced, that is, with least possible height. If a binary tree is balanced, then the height for a tree of n nodes is approximately log n. However, if the tree is completely unbalanced, for example in the shape of a linked list, then the height for a tree with n nodes can be as great as n. Thus, a balanced BST will in the average case have operations costing ⇥(log n), while a badly unbalanced BST can have operations in the worst case costing ⇥(n). Consider the situation where we construct a BST of n nodes by inserting records one at a time. If we are fortunate to have them arrive in an order that results in a balanced tree (a “random” order is Sec. 5.4 Binary Search Trees 177 // Remove a node with key value k // Return: The tree with the node removed template BSTNode* BST:: removehelp(BSTNode* rt, const Key& k) { if (rt == NULL) return NULL; // k is not in tree else if (k < rt->key()) rt->setLeft(removehelp(rt->left(), k)); else if (k > rt->key()) rt->setRight(removehelp(rt->right(), k)); else { // Found: remove it BSTNode* temp = rt; if (rt->left() == NULL) { // Only a right child rt = rt->right(); // so point to right delete temp; } else if (rt->right() == NULL) { // Only a left child rt = rt->left(); // so point to left delete temp; } else { // Both children are non-empty BSTNode* temp = getmin(rt->right()); rt->setElement(temp->element()); rt->setKey(temp->key()); rt->setRight(deletemin(rt->right())); delete temp; } } return rt; } Figure 5.18 Implementation for the BST removehelp method. likely to be good enough for this purpose), then each insertion will cost on average ⇥(log n), for a total cost of ⇥(n log n). However, if the records are inserted in order of increasing value, then the resulting tree will be a chain of height n. The cost of insertion in this case will be P n i=1 i =⇥(n2). Traversing a BST costs ⇥(n) regardless of the shape of the tree. Each node is visited exactly once, and each child pointer is followed exactly once. Below are two example traversals. The first is member clearhelp, which returns the nodes of the BST to the freelist. Because the children of a node must be freed before the node itself, this is a postorder traversal. template void BST:: clearhelp(BSTNode* root) { if (root == NULL) return; clearhelp(root->left()); clearhelp(root->right()); delete root; } 178 Chap. 5 Binary Trees The next example is printhelp, which performs an inorder traversal on the BST to print the node values in ascending order. Note that printhelp indents each line to indicate the depth of the corresponding node in the tree. Thus we pass in the current level of the tree in level, and increment this value each time that we make a recursive call. template void BST:: printhelp(BSTNode* root, int level) const { if (root == NULL) return; // Empty tree printhelp(root->left(), level+1); // Do left subtree for (int i=0; ikey() << "\n"; // Print node value printhelp(root->right(), level+1); // Do right subtree } While the BST is simple to implement and efficient when the tree is balanced, the possibility of its being unbalanced is a serious liability. There are techniques for organizing a BST to guarantee good performance. Two examples are the AVL tree and the splay tree of Section 13.2. Other search trees are guaranteed to remain balanced, such as the 2-3 tree of Section 10.4. 5.5 Heaps and Priority Queues There are many situations, both in real life and in computing applications, where we wish to choose the next “most important” from a collection of people, tasks, or objects. For example, doctors in a hospital emergency room often choose to see next the “most critical” patient rather than the one who arrived first. When scheduling programs for execution in a multitasking operating system, at any given moment there might be several programs (usually called jobs) ready to run. The next job selected is the one with the highest priority. Priority is indicated by a particular value associated with the job (and might change while the job remains in the wait list). When a collection of objects is organized by importance or priority, we call this a priority queue. A normal queue data structure will not implement a prior- ity queue efficiently because search for the element with highest priority will take ⇥(n) time. A list, whether sorted or not, will also require ⇥(n) time for either in- sertion or removal. A BST that organizes records by priority could be used, with the total of n inserts and n remove operations requiring ⇥(n log n) time in the average case. However, there is always the possibility that the BST will become unbal- anced, leading to bad performance. Instead, we would like to find a data structure that is guaranteed to have good performance for this special application. Sec. 5.5 Heaps and Priority Queues 179 This section presents the heap4 data structure. A heap is defined by two prop- erties. First, it is a complete binary tree, so heaps are nearly always implemented using the array representation for complete binary trees presented in Section 5.3.3. Second, the values stored in a heap are partially ordered. This means that there is a relationship between the value stored at any node and the values of its children. There are two variants of the heap, depending on the definition of this relationship. A max-heap has the property that every node stores a value that is greater than or equal to the value of either of its children. Because the root has a value greater than or equal to its children, which in turn have values greater than or equal to their children, the root stores the maximum of all values in the tree. A min-heap has the property that every node stores a value that is less than or equal to that of its children. Because the root has a value less than or equal to its children, which in turn have values less than or equal to their children, the root stores the minimum of all values in the tree. Note that there is no necessary relationship between the value of a node and that of its sibling in either the min-heap or the max-heap. For example, it is possible that the values for all nodes in the left subtree of the root are greater than the values for every node of the right subtree. We can contrast BSTs and heaps by the strength of their ordering relationships. A BST defines a total order on its nodes in that, given the positions for any two nodes in the tree, the one to the “left” (equivalently, the one appearing earlier in an inorder traversal) has a smaller key value than the one to the “right.” In contrast, a heap implements a partial order. Given their positions, we can determine the relative order for the key values of two nodes in the heap only if one is a descendant of the other. Min-heaps and max-heaps both have their uses. For example, the Heapsort of Section 7.6 uses the max-heap, while the Replacement Selection algorithm of Section 8.5.2 uses a min-heap. The examples in the rest of this section will use a max-heap. Be careful not to confuse the logical representation of a heap with its physical implementation by means of the array-based complete binary tree. The two are not synonymous because the logical view of the heap is actually a tree structure, while the typical physical implementation uses an array. Figure 5.19 shows an implementation for heaps.The class is a template with two parameters. E defines the type for the data elements stored in the heap, while Comp is the comparison class for comparing two elements. This class can implement ei- ther a min-heap or a max-heap by changing the definition for Comp. Comp defines method prior, a binary Boolean function that returns true if the first parameter should come before the second in the heap. This class definition makes two concessions to the fact that an array-based im- plementation is used. First, heap nodes are indicated by their logical position within 4The term “heap” is also sometimes used to refer to a memory pool. See Section 12.3. 180 Chap. 5 Binary Trees // Heap class template class heap { private: E* Heap; // Pointer to the heap array int maxsize; // Maximum size of the heap int n; // Number of elements now in the heap // Helper function to put element in its correct place void siftdown(int pos) { while (!isLeaf(pos)) { // Stop if pos is a leaf int j = leftchild(pos); int rc = rightchild(pos); if ((rc < n) && Comp::prior(Heap[rc], Heap[j])) j = rc; // Set j to greater child’s value if (Comp::prior(Heap[pos], Heap[j])) return; // Done swap(Heap, pos, j); pos = j; // Move down } } public: heap(E* h, int num, int max) // Constructor { Heap = h; n = num; maxsize = max; buildHeap(); } int size() const // Return current heap size { return n; } bool isLeaf(int pos) const // True if pos is a leaf { return (pos >= n/2) && (pos < n); } int leftchild(int pos) const { return 2*pos + 1; } // Return leftchild position int rightchild(int pos) const { return 2*pos + 2; } // Return rightchild position int parent(int pos) const // Return parent position { return (pos-1)/2; } void buildHeap() // Heapify contents of Heap { for (int i=n/2-1; i>=0; i--) siftdown(i); } // Insert "it" into the heap void insert(const E& it) { Assert(n < maxsize, "Heap is full"); int curr = n++; Heap[curr] = it; // Start at end of heap // Now sift up until curr’s parent > curr while ((curr!=0) && (Comp::prior(Heap[curr], Heap[parent(curr)]))) { swap(Heap, curr, parent(curr)); curr = parent(curr); } } Figure 5.19 An implementation for the heap. Sec. 5.5 Heaps and Priority Queues 181 // Remove first value E removefirst() { Assert (n > 0, "Heap is empty"); swap(Heap, 0, --n); // Swap first with last value if (n != 0) siftdown(0); // Siftdown new root val return Heap[n]; // Return deleted value } // Remove and return element at specified position E remove(int pos) { Assert((pos >= 0) && (pos < n), "Bad position"); if (pos == (n-1)) n--; // Last element, no work to do else { swap(Heap, pos, --n); // Swap with last value while ((pos != 0) && (Comp::prior(Heap[pos], Heap[parent(pos)]))) { swap(Heap, pos, parent(pos)); // Push up large key pos = parent(pos); } if (n != 0) siftdown(pos); // Push down small key } return Heap[n]; } }; Figure 5.19 (continued) the heap rather than by a pointer to the node. In practice, the logical heap position corresponds to the identically numbered physical position in the array. Second, the constructor takes as input a pointer to the array to be used. This approach provides the greatest flexibility for using the heap because all data values can be loaded into the array directly by the client. The advantage of this comes during the heap con- struction phase, as explained below. The constructor also takes an integer parame- ter indicating the initial size of the heap (based on the number of elements initially loaded into the array) and a second integer parameter indicating the maximum size allowed for the heap (the size of the array). Method heapsize returns the current size of the heap. H.isLeaf(pos) returns true if position pos is a leaf in heap H, and false otherwise. Members leftchild, rightchild, and parent return the position (actually, the array index) for the left child, right child, and parent of the position passed, respectively. One way to build a heap is to insert the elements one at a time. Method insert will insert a new element V into the heap. You might expect the heap insertion pro- cess to be similar to the insert function for a BST, starting at the root and working down through the heap. However, this approach is not likely to work because the heap must maintain the shape of a complete binary tree. Equivalently, if the heap takes up the first n positions of its array prior to the call to insert, it must take 182 Chap. 5 Binary Trees up the first n +1positions after. To accomplish this, insert first places V at po- sition n of the array. Of course, V is unlikely to be in the correct position. To move V to the right place, it is compared to its parent’s value. If the value of V is less than or equal to the value of its parent, then it is in the correct place and the insert routine is finished. If the value of V is greater than that of its parent, then the two elements swap positions. From here, the process of comparing V to its (current) parent continues until V reaches its correct position. Since the heap is a complete binary tree, its height is guaranteed to be the minimum possible. In particular, a heap containing n nodes will have a height of ⇥(n log n). Intuitively, we can see that this must be true because each level that we add will slightly more than double the number of nodes in the tree (the ith level has 2i nodes, and the sum of the first i levels is 2i+1 1). Starting at 1, we can double only log n times to reach a value of n. To be precise, the height of a heap with n nodes is dlog(n + 1)e. Each call to insert takes ⇥(log n) time in the worst case, because the value being inserted can move at most the distance from the bottom of the tree to the top of the tree. Thus, to insert n values into the heap, if we insert them one at a time, will take ⇥(n log n) time in the worst case. If all n values are available at the beginning of the building process, we can build the heap faster than just inserting the values into the heap one by one. Con- sider Figure 5.20(a), which shows one series of exchanges that could be used to build the heap. All exchanges are between a node and one of its children. The heap is formed as a result of this exchange process. The array for the right-hand tree of Figure 5.20(a) would appear as follows: 7 4 6 1 2 3 5 Figure 5.20(b) shows an alternate series of exchanges that also forms a heap, but much more efficiently. The equivalent array representation would be 7 5 6 4 2 1 3 From this example, it is clear that the heap for any given set of numbers is not unique, and we see that some rearrangements of the input values require fewer ex- changes than others to build the heap. So, how do we pick the best rearrangement? One good algorithm stems from induction. Suppose that the left and right sub- trees of the root are already heaps, and R is the name of the element at the root. This situation is illustrated by Figure 5.21. In this case there are two possibilities. (1) R has a value greater than or equal to its two children. In this case, construction is complete. (2) R has a value less than one or both of its children. In this case, R should be exchanged with the child that has greater value. The result will be a heap, except that R might still be less than one or both of its (new) children. In this case, we simply continue the process of “pushing down” R until it reaches a Sec. 5.5 Heaps and Priority Queues 183 (a) 6 (b) 4567 574 23 2 2 6 6 35 1 3 7 5 4213 7 4 1 1 Figure 5.20 Two series of exchanges to build a max-heap. (a) This heap is built by a series of nine exchanges in the order (4-2), (4-1), (2-1), (5-2), (5-4), (6-3), (6-5), (7-5), (7-6). (b) This heap is built by a series of four exchanges in the order (5-2), (7-3), (7-1), (6-1). R H1 H2 Figure 5.21 Final stage in the heap-building algorithm. Both subtrees of node R are heaps. All that remains is to push R down to its proper level in the heap. level where it is greater than its children, or is a leaf node. This process is imple- mented by the private method siftdown. The siftdown operation is illustrated by Figure 5.22. This approach assumes that the subtrees are already heaps, suggesting that a complete algorithm can be obtained by visiting the nodes in some order such that the children of a node are visited before the node itself. One simple way to do this is simply to work from the high index of the array to the low index. Actually, the build process need not visit the leaf nodes (they can never move down because they are already at the bottom), so the building algorithm can start in the middle of the array, with the first internal node. The exchanges shown in Figure 5.20(b) result from this process. Method buildHeap implements the building algorithm. 184 Chap. 5 Binary Trees (a) (b) (c) 5 1 7 7 51 7 56 42 4362634213 Figure 5.22 The siftdown operation. The subtrees of the root are assumed to be heaps. (a) The partially completed heap. (b) Values 1 and 7 are swapped. (c) Values 1 and 6 are swapped to form the final heap. What is the cost of buildHeap? Clearly it is the sum of the costs for the calls to siftdown. Each siftdown operation can cost at most the number of levels it takes for the node being sifted to reach the bottom of the tree. In any complete tree, approximately half of the nodes are leaves and so cannot be moved downward at all. One quarter of the nodes are one level above the leaves, and so their elements can move down at most one level. At each step up the tree we get half the number of nodes as were at the previous level, and an additional height of one. The maximum sum of total distances that elements can go is therefore log n Xi=1 (i 1) n 2i = n 2 log n Xi=1 i 1 2i1 . From Equation 2.9 we know that this summation has a closed-form solution of approximately 2, so this algorithm takes ⇥(n) time in the worst case. This is far better than building the heap one element at a time, which would cost ⇥(n log n) in the worst case. It is also faster than the ⇥(n log n) average-case time and ⇥(n2) worst-case time required to build the BST. Removing the maximum (root) value from a heap containing n elements re- quires that we maintain the complete binary tree shape, and that the remaining n 1 node values conform to the heap property. We can maintain the proper shape by moving the element in the last position in the heap (the current last element in the array) to the root position. We now consider the heap to be one element smaller. Unfortunately, the new root value is probably not the maximum value in the new heap. This problem is easily solved by using siftdown to reorder the heap. Be- cause the heap is log n levels deep, the cost of deleting the maximum element is ⇥(log n) in the average and worst cases. The heap is a natural implementation for the priority queue discussed at the beginning of this section. Jobs can be added to the heap (using their priority value as the ordering key) when needed. Method removemax can be called whenever a new job is to be executed. Sec. 5.6 Hu↵man Coding Trees 185 Some applications of priority queues require the ability to change the priority of an object already stored in the queue. This might require that the object’s position in the heap representation be updated. Unfortunately, a max-heap is not efficient when searching for an arbitrary value; it is only good for finding the maximum value. However, if we already know the index for an object within the heap, it is a simple matter to update its priority (including changing its position to maintain the heap property) or remove it. The remove method takes as input the position of the node to be removed from the heap. A typical implementation for priority queues requiring updating of priorities will need to use an auxiliary data structure that supports efficient search for objects (such as a BST). Records in the auxiliary data structure will store the object’s heap index, so that the object can be deleted from the heap and reinserted with its new priority (see Project 5.5). Sections 11.4.1 and 11.5.1 present applications for a priority queue with priority updating. 5.6 Hu↵man Coding Trees The space/time tradeoff principle from Section 3.9 states that one can often gain an improvement in space requirements in exchange for a penalty in running time. There are many situations where this is a desirable tradeoff. A typical example is storing files on disk. If the files are not actively used, the owner might wish to compress them to save space. Later, they can be uncompressed for use, which costs some time, but only once. We often represent a set of items in a computer program by assigning a unique code to each item. For example, the standard ASCII coding scheme assigns a unique eight-bit value to each character. It takes a certain minimum number of bits to provide unique codes for each character. For example, it takes dlog 128e or seven bits to provide the 128 unique codes needed to represent the 128 symbols of the ASCII character set.5 The requirement for dlog ne bits to represent n unique code values assumes that all codes will be the same length, as are ASCII codes. This is called a fixed-length coding scheme. If all characters were used equally often, then a fixed-length coding scheme is the most space efficient method. However, you are probably aware that not all characters are used equally often in many applications. For example, the various letters in an English language document have greatly different frequencies of use. Figure 5.23 shows the relative frequencies of the letters of the alphabet. From this table we can see that the letter ‘E’ appears about 60 times more often than the letter ‘Z.’ In normal ASCII, the words “DEED” and “MUCK” require the same 5The ASCII standard is eight bits, not seven, even though there are only 128 characters repre- sented. The eighth bit is used either to check for transmission errors, or to support “extended” ASCII codes with an additional 128 characters. 186 Chap. 5 Binary Trees Letter Frequency Letter Frequency A 77 N 67 B 17 O 67 C 32 P 20 D 42 Q 5 E 120 R 59 F 24 S 67 G 17 T 85 H 50 U 37 I 76 V 12 J 4 W 22 K 7 X 4 L 42 Y 22 M 24 Z 2 Figure 5.23 Relative frequencies for the 26 letters of the alphabet as they ap- pear in a selected set of English documents. “Frequency” represents the expected frequency of occurrence per 1000 letters, ignoring case. amount of space (four bytes). It would seem that words such as “DEED,” which are composed of relatively common letters, should be storable in less space than words such as “MUCK,” which are composed of relatively uncommon letters. If some characters are used more frequently than others, is it possible to take advantage of this fact and somehow assign them shorter codes? The price could be that other characters require longer codes, but this might be worthwhile if such characters appear rarely enough. This concept is at the heart of file compression techniques in common use today. The next section presents one such approach to assigning variable-length codes, called Huffman coding. While it is not commonly used in its simplest form for file compression (there are better methods), Huffman coding gives the flavor of such coding schemes. One motivation for studying Huff- man coding is because it provides our first opportunity to see a type of tree structure referred to as a search trie. 5.6.1 Building Hu↵man Coding Trees Huffman coding assigns codes to characters such that the length of the code de- pends on the relative frequency or weight of the corresponding character. Thus, it is a variable-length code. If the estimated frequencies for letters match the actual frequency found in an encoded message, then the length of that message will typi- cally be less than if a fixed-length code had been used. The Huffman code for each letter is derived from a full binary tree called the Huffman coding tree, or simply the Huffman tree. Each leaf of the Huffman tree corresponds to a letter, and we define the weight of the leaf node to be the weight (frequency) of its associated Sec. 5.6 Hu↵man Coding Trees 187 Letter CD E KLMUZ Frequency 32 42 120 7 42 24 37 2 Figure 5.24 The relative frequencies for eight selected letters. letter. The goal is to build a tree with the minimum external path weight. Define the weighted path length of a leaf to be its weight times its depth. The binary tree with minimum external path weight is the one with the minimum sum of weighted path lengths for the given set of leaves. A letter with high weight should have low depth, so that it will count the least against the total path length. As a result, another letter might be pushed deeper in the tree if it has less weight. The process of building the Huffman tree for n letters is quite simple. First, cre- ate a collection of n initial Huffman trees, each of which is a single leaf node con- taining one of the letters. Put the n partial trees onto a priority queue organized by weight (frequency). Next, remove the first two trees (the ones with lowest weight) from the priority queue. Join these two trees together to create a new tree whose root has the two trees as children, and whose weight is the sum of the weights of the two trees. Put this new tree back into the priority queue. This process is repeated until all of the partial Huffman trees have been combined into one. Example 5.8 Figure 5.25 illustrates part of the Huffman tree construction process for the eight letters of Figure 5.24. Ranking D and L arbitrarily by alphabetical order, the letters are ordered by frequency as Letter ZKM C U D L E Frequency 2 7 24 32 37 42 42 120 Because the first two letters on the list are Z and K, they are selected to be the first trees joined together.6 They become the children of a root node with weight 9. Thus, a tree whose root has weight 9 is placed back on the list, where it takes up the first position. The next step is to take values 9 and 24 off the list (corresponding to the partial tree with two leaf nodes built in the last step, and the partial tree storing the letter M, respectively) and join them together. The resulting root node has weight 33, and so this tree is placed back into the list. Its priority will be between the trees with values 32 (for letter C) and 37 (for letter U). This process continues until a tree whose root has weight 306 is built. This tree is shown in Figure 5.26. 6For clarity, the examples for building Huffman trees show a sorted list to keep the letters ordered by frequency. But a real implementation would use a heap to implement the priority queue for efficiency. 188 Chap. 5 Binary Trees Step 1: Step 2: 9 Step 3: Step 4: 65 Step 5: 42 32 C 65 33 9 E 79 L 24 L 120 37 42 C 42 32 24 UD E 27 KZ M 9 9 24 37 U 42 D 42 M 32 120 C LE MCUD 2 Z 7 24 32 37 42 42 L K 120 E 27 KMC 32 37 42 4224 LZ D 120 EU 120 2 Z 7 K 37 42 DU 2 Z 7 33 33 M K Figure 5.25 The first five steps of the building process for a sample Huffman tree. Sec. 5.6 Hu↵man Coding Trees 189 30601 E 0 79 01 37 U 42 1 1070 42 1 650 C 1 0 1 9 01 2 Z 7 DL M K 32 33 24 120 186 Figure 5.26 A Huffman tree for the letters of Figure 5.24. Figure 5.27 shows an implementation for Huffman tree nodes. This implemen- tation is similar to the VarBinNode implementation of Figure 5.10. There is an abstract base class, named HuffNode, and two subclasses, named LeafNode and IntlNode. This implementation reflects the fact that leaf and internal nodes contain distinctly different information. Figure 5.28 shows the implementation for the Huffman tree. Figure 5.29 shows the C++ code for the tree-building process. Huffman tree building is an example of a greedy algorithm. At each step, the algorithm makes a “greedy” decision to merge the two subtrees with least weight. This makes the algorithm simple, but does it give the desired result? This sec- tion concludes with a proof that the Huffman tree indeed gives the most efficient arrangement for the set of letters. The proof requires the following lemma. Lemma 5.1 For any Huffman tree built by function buildHuff containing at least two letters, the two letters with least frequency are stored in siblings nodes whose depth is at least as deep as any other leaf nodes in the tree. Proof: Call the two letters with least frequency l1 and l2. They must be siblings because buildHuff selects them in the first step of the construction process. Assume that l1 and l2 are not the deepest nodes in the tree. In this case, the Huffman tree must either look as shown in Figure 5.30, or in some sense be symmetrical to this. For this situation to occur, the parent of l1 and l2, labeled V, must have greater weight than the node labeled X. Otherwise, function buildHuff would have selected node V in place of node X as the child of node U. However, this is impossible because l1 and l2 are the letters with least frequency. 2 190 Chap. 5 Binary Trees // Huffman tree node abstract base class template class HuffNode { public: virtual ˜HuffNode() {} // Base destructor virtual int weight() = 0; // Return frequency virtual bool isLeaf() = 0; // Determine type }; template // Leaf node subclass class LeafNode : public HuffNode { private: E it; // Value int wgt; // Weight public: LeafNode(const E& val, int freq) // Constructor { it = val; wgt = freq; } int weight() { return wgt; } E val() { return it; } bool isLeaf() { return true; } }; template // Internal node subclass class IntlNode : public HuffNode { private: HuffNode* lc; // Left child HuffNode* rc; // Right child int wgt; // Subtree weight public: IntlNode(HuffNode* l, HuffNode* r) { wgt = l->weight() + r->weight(); lc = l; rc = r; } int weight() { return wgt; } bool isLeaf() { return false; } HuffNode* left() const { return lc; } void setLeft(HuffNode* b) { lc = (HuffNode*)b; } HuffNode* right() const { return rc; } void setRight(HuffNode* b) { rc = (HuffNode*)b; } }; Figure 5.27 Implementation for Huffman tree nodes. Internal nodes and leaf nodes are represented by separate classes, each derived from an abstract base class. Sec. 5.6 Hu↵man Coding Trees 191 // HuffTree is a template of two parameters: the element // type being coded and a comparator for two such elements. template class HuffTree { private: HuffNode* Root; // Tree root public: HuffTree(E& val, int freq) // Leaf constructor { Root = new LeafNode(val, freq); } // Internal node constructor HuffTree(HuffTree* l, HuffTree* r) { Root = new IntlNode(l->root(), r->root()); } ˜HuffTree() {} // Destructor HuffNode* root() { return Root; } // Get root int weight() { return Root->weight(); } // Root weight }; Figure 5.28 Class declarations for the Huffman tree. // Build a Huffman tree from a collection of frequencies template HuffTree* buildHuff(HuffTree** TreeArray, int count) { heap*,minTreeComp>* forest = new heap*, minTreeComp>(TreeArray, count, count); HuffTree *temp1, *temp2, *temp3 = NULL; while (forest->size() > 1) { temp1 = forest->removefirst(); // Pull first two trees temp2 = forest->removefirst(); // off the list temp3 = new HuffTree(temp1, temp2); forest->insert(temp3); // Put the new tree back on list delete temp1; // Must delete the remnants delete temp2; // of the trees we created } return temp3; } Figure 5.29 Implementation for the Huffman tree construction function. buildHuff takes as input fl, the min-heap of partial Huffman trees, which initially are single leaf nodes as shown in Step 1 of Figure 5.25. The body of function buildTree consists mainly of a for loop. On each iteration of the for loop, the first two partial trees are taken off the heap and placed in variables temp1 and temp2. A tree is created (temp3) such that the left and right subtrees are temp1 and temp2, respectively. Finally, temp3 is returned to fl. 192 Chap. 5 Binary Trees l1 X V l2 U Figure 5.30 An impossible Huffman tree, showing the situation where the two nodes with least weight, l1 and l2, are not the deepest nodes in the tree. Triangles represent subtrees. Theorem 5.3 Function buildHuff builds the Huffman tree with the minimum external path weight for the given set of letters. Proof: The proof is by induction on n, the number of letters. • Base Case: For n =2, the Huffman tree must have the minimum external path weight because there are only two possible trees, each with identical weighted path lengths for the two leaves. • Induction Hypothesis: Assume that any tree created by buildHuff that contains n 1 leaves has minimum external path length. • Induction Step: Given a Huffman tree T built by buildHuff with n leaves, n 2, suppose that w1  w2 ···wn where w1 to wn are the weights of the letters. Call V the parent of the letters with frequencies w1 and w2. From the lemma, we know that the leaf nodes containing the letters with frequencies w1 and w2 are as deep as any nodes in T. If any other leaf nodes in the tree were deeper, we could reduce their weighted path length by swapping them with w1 or w2. But the lemma tells us that no such deeper nodes exist. Call T0 the Huffman tree that is identical to T except that node V is replaced with a leaf node V 0 whose weight is w1 + w2. By the induction hypothesis, T0 has minimum external path length. Returning the children to V 0 restores tree T, which must also have minimum external path length. Thus by mathematical induction, function buildHuff creates the Huffman tree with minimum external path length. 2 5.6.2 Assigning and Using Hu↵man Codes Once the Huffman tree has been constructed, it is an easy matter to assign codes to individual letters. Beginning at the root, we assign either a ‘0’ or a ‘1’ to each edge in the tree. ‘0’ is assigned to edges connecting a node with its left child, and ‘1’ to edges connecting a node with its right child. This process is illustrated by Sec. 5.6 Hu↵man Coding Trees 193 Letter Freq Code Bits C 32 1110 4 D 42 101 3 E 120 0 1 K 7 111101 6 L 42 110 3 M 24 11111 5 U 37 100 3 Z 2 111100 6 Figure 5.31 The Huffman codes for the letters of Figure 5.24. Figure 5.26. The Huffman code for a letter is simply a binary number determined by the path from the root to the leaf corresponding to that letter. Thus, the code for E is ‘0’ because the path from the root to the leaf node for E takes a single left branch. The code for K is ‘111101’ because the path to the node for K takes four right branches, then a left, and finally one last right. Figure 5.31 lists the codes for all eight letters. Given codes for the letters, it is a simple matter to use these codes to encode a text message. We simply replace each letter in the string with its binary code. A lookup table can be used for this purpose. Example 5.9 Using the code generated by our example Huffman tree, the word “DEED” is represented by the bit string “10100101” and the word “MUCK” is represented by the bit string “111111001110111101.” Decoding the message is done by looking at the bits in the coded string from left to right until a letter is decoded. This can be done by using the Huffman tree in a reverse process from that used to generate the codes. Decoding a bit string begins at the root of the tree. We take branches depending on the bit value — left for ‘0’ and right for ‘1’ — until reaching a leaf node. This leaf contains the first character in the message. We then process the next bit in the code restarting at the root to begin the next character. Example 5.10 To decode the bit string “1011001110111101” we begin at the root of the tree and take a right branch for the first bit which is ‘1.’ Because the next bit is a ‘0’ we take a left branch. We then take another right branch (for the third bit ‘1’), arriving at the leaf node corresponding to the letter D. Thus, the first letter of the coded word is D. We then begin again at the root of the tree to process the fourth bit, which is a ‘1.’ Taking a right branch, then two left branches (for the next two bits which are ‘0’), we reach the leaf node corresponding to the letter U. Thus, the second letter 194 Chap. 5 Binary Trees is U. In similar manner we complete the decoding process to find that the last two letters are C and K, spelling the word “DUCK.” A set of codes is said to meet the prefix property if no code in the set is the prefix of another. The prefix property guarantees that there will be no ambiguity in how a bit string is decoded. In other words, once we reach the last bit of a code during the decoding process, we know which letter it is the code for. Huffman codes certainly have the prefix property because any prefix for a code would correspond to an internal node, while all codes correspond to leaf nodes. For example, the code for M is ‘11111.’ Taking five right branches in the Huffman tree of Figure 5.26 brings us to the leaf node containing M. We can be sure that no letter can have code ‘111’ because this corresponds to an internal node of the tree, and the tree-building process places letters only at the leaf nodes. How efficient is Huffman coding? In theory, it is an optimal coding method whenever the true frequencies are known, and the frequency of a letter is indepen- dent of the context of that letter in the message. In practice, the frequencies of letters in an English text document do change depending on context. For example, while E is the most commonly used letter of the alphabet in English documents, T is more common as the first letter of a word. This is why most commercial com- pression utilities do not use Huffman coding as their primary coding method, but instead use techniques that take advantage of the context for the letters. Another factor that affects the compression efficiency of Huffman coding is the relative frequencies of the letters. Some frequency patterns will save no space as compared to fixed-length codes; others can result in great compression. In general, Huffman coding does better when there is large variation in the frequencies of letters. In the particular case of the frequencies shown in Figure 5.31, we can determine the expected savings from Huffman coding if the actual frequencies of a coded message match the expected frequencies. Example 5.11 Because the sum of the frequencies in Figure 5.31 is 306 and E has frequency 120, we expect it to appear 120 times in a message containing 306 letters. An actual message might or might not meet this expectation. Letters D, L, and U have code lengths of three, and together are expected to appear 121 times in 306 letters. Letter C has a code length of four, and is expected to appear 32 times in 306 letters. Letter M has a code length of five, and is expected to appear 24 times in 306 letters. Finally, letters K and Z have code lengths of six, and together are expected to appear only 9 times in 306 letters. The average expected cost per character is simply the sum of the cost for each character (ci) times the probability of its occurring (pi), or c1p1 + c2p2 + ···+ cnpn. Sec. 5.6 Hu↵man Coding Trees 195 This can be reorganized as c1f1 + c2f2 + ···+ cnfn fT where fi is the (relative) frequency of letter i and fT is the total for all letter frequencies. For this set of frequencies, the expected cost per letter is [(1⇥120)+(3⇥121)+(4⇥32)+(5⇥24)+(6⇥9)]/306 = 785/306 ⇡ 2.57 A fixed-length code for these eight characters would require log 8 = 3 bits per letter as opposed to about 2.57 bits per letter for Huffman coding. Thus, Huffman coding is expected to save about 14% for this set of letters. Huffman coding for all ASCII symbols should do better than this. The letters of Figure 5.31 are atypical in that there are too many common letters compared to the number of rare letters. Huffman coding for all 26 letters would yield an expected cost of 4.29 bits per letter. The equivalent fixed-length code would require about five bits. This is somewhat unfair to fixed-length coding because there is actually room for 32 codes in five bits, but only 26 letters. More generally, Huffman coding of a typical text file will save around 40% over ASCII coding if we charge ASCII coding at eight bits per character. Huffman coding for a binary file (such as a compiled executable) would have a very different set of distribution frequencies and so would have a different space savings. Most commercial compression programs use two or three coding schemes to adjust to different types of files. In the preceding example, “DEED” was coded in 8 bits, a saving of 33% over the twelve bits required from a fixed-length coding. However, “MUCK” requires 18 bits, more space than required by the corresponding fixed-length coding. The problem is that “MUCK” is composed of letters that are not expected to occur often. If the message does not match the expected frequencies of the letters, than the length of the encoding will not be as expected either. 5.6.3 Search in Hu↵man Trees When we decode a character using the Huffman coding tree, we follow a path through the tree dictated by the bits in the code string. Each ‘0’ bit indicates a left branch while each ‘1’ bit indicates a right branch. Now look at Figure 5.26 and consider this structure in terms of searching for a given letter (whose key value is its Huffman code). We see that all letters with codes beginning with ’0’ are stored in the left branch, while all letters with codes beginning with ‘1’ are stored in the right branch. Contrast this with storing records in a BST. There, all records with key value less than the root value are stored in the left branch, while all records with key values greater than the root are stored in the right branch. 196 Chap. 5 Binary Trees If we view all records stored in either of these structures as appearing at some point on a number line representing the key space, we can see that the splitting behavior of these two structures is very different. The BST splits the space based on the key values as they are encountered when going down the tree. But the splits in the key space are predetermined for the Huffman tree. Search tree structures whose splitting points in the key space are predetermined are given the special name trie to distinguish them from the type of search tree (like the BST) whose splitting points are determined by the data. Tries are discussed in more detail in Chapter 13. 5.7 Further Reading See Shaffer and Brown [SB93] for an example of a tree implementation where an internal node pointer field stores the value of its child instead of a pointer to its child when the child is a leaf node. Many techniques exist for maintaining reasonably balanced BSTs in the face of an unfriendly series of insert and delete operations. One example is the AVL tree of Adelson-Velskii and Landis, which is discussed by Knuth [Knu98]. The AVL tree (see Section 13.2) is actually a BST whose insert and delete routines reorganize the tree structure so as to guarantee that the subtrees rooted by the children of any node will differ in height by at most one. Another example is the splay tree [ST85], also discussed in Section 13.2. See Bentley’s Programming Pearl “Thanks, Heaps” [Ben85, Ben88] for a good discussion on the heap data structure and its uses. The proof of Section 5.6.1 that the Huffman coding tree has minimum external path weight is from Knuth [Knu97]. For more information on data compression techniques, see Managing Gigabytes by Witten, Moffat, and Bell [WMB99], and Codes and Cryptography by Dominic Welsh [Wel88]. Tables 5.23 and 5.24 are derived from Welsh [Wel88]. 5.8 Exercises 5.1 Section 5.1.1 claims that a full binary tree has the highest number of leaf nodes among all trees with n internal nodes. Prove that this is true. 5.2 Define the degree of a node as the number of its non-empty children. Prove by induction that the number of degree 2 nodes in any binary tree is one less than the number of leaves. 5.3 Define the internal path length for a tree as the sum of the depths of all internal nodes, while the external path length is the sum of the depths of all leaf nodes in the tree. Prove by induction that if tree T is a full binary tree with n internal nodes, I is T’s internal path length, and E is T’s external path length, then E = I +2n for n 0. Sec. 5.8 Exercises 197 5.4 Explain why function preorder2 from Section 5.2 makes half as many recursive calls as function preorder. Explain why it makes twice as many accesses to left and right children. 5.5 (a) Modify the preorder traversal of Section 5.2 to perform an inorder traversal of a binary tree. (b) Modify the preorder traversal of Section 5.2 to perform a postorder traversal of a binary tree. 5.6 Write a recursive function named search that takes as input the pointer to the root of a binary tree (not a BST!) and a value K, and returns true if value K appears in the tree and false otherwise. 5.7 Write an algorithm that takes as input the pointer to the root of a binary tree and prints the node values of the tree in level order. Level order first prints the root, then all nodes of level 1, then all nodes of level 2, and so on. Hint: Preorder traversals make use of a stack through recursive calls. Consider making use of another data structure to help implement the level- order traversal. 5.8 Write a recursive function that returns the height of a binary tree. 5.9 Write a recursive function that returns a count of the number of leaf nodes in a binary tree. 5.10 Assume that a given BST stores integer values in its nodes. Write a recursive function that sums the values of all nodes in the tree. 5.11 Assume that a given BST stores integer values in its nodes. Write a recursive function that traverses a binary tree, and prints the value of every node who’s grandparent has a value that is a multiple of five. 5.12 Write a recursive function that traverses a binary tree, and prints the value of every node which has at least four great-grandchildren. 5.13 Compute the overhead fraction for each of the following full binary tree im- plementations. (a) All nodes store data, two child pointers, and a parent pointer. The data field requires four bytes and each pointer requires four bytes. (b) All nodes store data and two child pointers. The data field requires sixteen bytes and each pointer requires four bytes. (c) All nodes store data and a parent pointer, and internal nodes store two child pointers. The data field requires eight bytes and each pointer re- quires four bytes. (d) Only leaf nodes store data; internal nodes store two child pointers. The data field requires eight bytes and each pointer requires four bytes. 5.14 Why is the BST Property defined so that nodes with values equal to the value of the root appear only in the right subtree, rather than allow equal-valued nodes to appear in either subtree? 198 Chap. 5 Binary Trees 5.15 (a) Show the BST that results from inserting the values 15, 20, 25, 18, 16, 5, and 7 (in that order). (b) Show the enumerations for the tree of (a) that result from doing a pre- order traversal, an inorder traversal, and a postorder traversal. 5.16 Draw the BST that results from adding the value 5 to the BST shown in Figure 5.13(a). 5.17 Draw the BST that results from deleting the value 7 from the BST of Fig- ure 5.13(b). 5.18 Write a function that prints out the node values for a BST in sorted order from highest to lowest. 5.19 Write a recursive function named smallcount that, given the pointer to the root of a BST and a key K, returns the number of nodes having key values less than or equal to K. Function smallcount should visit as few nodes in the BST as possible. 5.20 Write a recursive function named printRange that, given the pointer to the root of a BST, a low key value, and a high key value, prints in sorted order all records whose key values fall between the two given keys. Function printRange should visit as few nodes in the BST as possible. 5.21 Write a recursive function named checkBST that, given the pointer to the root of a binary tree, will return true if the tree is a BST, and false if it is not. 5.22 Describe a simple modification to the BST that will allow it to easily support finding the Kth smallest value in ⇥(log n) average case time. Then write a pseudo-code function for finding the Kth smallest value in your modified BST. 5.23 What are the minimum and maximum number of elements in a heap of height h? 5.24 Where in a max-heap might the smallest element reside? 5.25 Show the max-heap that results from running buildHeap on the following values stored in an array: 10 5 12 3 2 1 8 7 9 4 5.26 (a) Show the heap that results from deleting the maximum value from the max-heap of Figure 5.20b. (b) Show the heap that results from deleting the element with value 5 from the max-heap of Figure 5.20b. 5.27 Revise the heap definition of Figure 5.19 to implement a min-heap. The member function removemax should be replaced by a new function called removemin. 5.28 Build the Huffman coding tree and determine the codes for the following set of letters and weights: Sec. 5.8 Exercises 199 Letter ABCD E F G H I J K L Frequency 2 3 5 7 11 13 17 19 23 31 37 41 What is the expected length in bits of a message containing n characters for this frequency distribution? 5.29 What will the Huffman coding tree look like for a set of sixteen characters all with equal weight? What is the average code length for a letter in this case? How does this differ from the smallest possible fixed length code for sixteen characters? 5.30 A set of characters with varying weights is assigned Huffman codes. If one of the characters is assigned code 001, then, (a) Describe all codes that cannot have been assigned. (b) Describe all codes that must have been assigned. 5.31 Assume that a sample alphabet has the following weights: Letter QZ F M T S O E Frequency 2 3 10 10 10 15 20 30 (a) For this alphabet, what is the worst-case number of bits required by the Huffman code for a string of n letters? What string(s) have the worst- case performance? (b) For this alphabet, what is the best-case number of bits required by the Huffman code for a string of n letters? What string(s) have the best- case performance? (c) What is the average number of bits required by a character using the Huffman code for this alphabet? 5.32 You must keep track of some data. Your options are: (1) A linked-list maintained in sorted order. (2) A linked-list of unsorted records. (3) A binary search tree. (4) An array-based list maintained in sorted order. (5) An array-based list of unsorted records. For each of the following scenarios, which of these choices would be best? Explain your answer. (a) The records are guaranteed to arrive already sorted from lowest to high- est (i.e., whenever a record is inserted, its key value will always be greater than that of the last record inserted). A total of 1000 inserts will be interspersed with 1000 searches. (b) The records arrive with values having a uniform random distribution (so the BST is likely to be well balanced). 1,000,000 insertions are performed, followed by 10 searches. 200 Chap. 5 Binary Trees (c) The records arrive with values having a uniform random distribution (so the BST is likely to be well balanced). 1000 insertions are interspersed with 1000 searches. (d) The records arrive with values having a uniform random distribution (so the BST is likely to be well balanced). 1000 insertions are performed, followed by 1,000,000 searches. 5.9 Projects 5.1 Re-implement the composite design for the binary tree node class of Fig- ure 5.11 using a flyweight in place of NULL pointers to empty nodes. 5.2 One way to deal with the “problem” of NULL pointers in binary trees is to use that space for some other purpose. One example is the threaded binary tree. Extending the node implementation of Figure 5.7, the threaded binary tree stores with each node two additional bit fields that indicate if the child pointers lc and rc are regular pointers to child nodes or threads. If lc is not a pointer to a non-empty child (i.e., if it would be NULL in a regular binary tree), then it instead stores a pointer to the inorder predecessor of that node. The inorder predecessor is the node that would be printed immediately before the current node in an inorder traversal. If rc is not a pointer to a child, then it instead stores a pointer to the node’s inorder successor. The inorder successor is the node that would be printed immediately after the current node in an inorder traversal. The main advantage of threaded binary trees is that operations such as inorder traversal can be implemented without using recursion or a stack. Re-implement the BST as a threaded binary tree, and include a non-recursive version of the preorder traversal 5.3 Implement a city database using a BST to store the database records. Each database record contains the name of the city (a string of arbitrary length) and the coordinates of the city expressed as integer x- and y-coordinates. The BST should be organized by city name. Your database should allow records to be inserted, deleted by name or coordinate, and searched by name or coordinate. Another operation that should be supported is to print all records within a given distance of a specified point. Collect running-time statistics for each operation. Which operations can be implemented reason- ably efficiently (i.e., in ⇥(log n) time in the average case) using a BST? Can the database system be made more efficient by using one or more additional BSTs to organize the records by location? 5.4 Create a binary tree ADT that includes generic traversal methods that take a visitor, as described in Section 5.2. Write functions count and BSTcheck of Section 5.2 as visitors to be used with the generic traversal method. Sec. 5.9 Projects 201 5.5 Implement a priority queue class based on the max-heap class implementa- tion of Figure 5.19. The following methods should be supported for manip- ulating the priority queue: void enqueue(int ObjectID, int priority); int dequeue(); void changeweight(int ObjectID, int newPriority); Method enqueue inserts a new object into the priority queue with ID num- ber ObjectID and priority priority. Method dequeue removes the object with highest priority from the priority queue and returns its object ID. Method changeweight changes the priority of the object with ID number ObjectID to be newPriority. The type for E should be a class that stores the object ID and the priority for that object. You will need a mech- anism for finding the position of the desired object within the heap. Use an array, storing the object with ObjectID i in position i. (Be sure in your testing to keep the ObjectIDs within the array bounds.) You must also modify the heap implementation to store the object’s position in the auxil- iary array so that updates to objects in the heap can be updated as well in the array. 5.6 The Huffman coding tree function buildHuff of Figure 5.29 manipulates a sorted list. This could result in a ⇥(n2) algorithm, because placing an inter- mediate Huffman tree on the list could take ⇥(n) time. Revise this algorithm to use a priority queue based on a min-heap instead of a list. 5.7 Complete the implementation of the Huffman coding tree, building on the code presented in Section 5.6. Include a function to compute and store in a table the codes for each letter, and functions to encode and decode messages. This project can be further extended to support file compression. To do so requires adding two steps: (1) Read through the input file to generate actual frequencies for all letters in the file; and (2) store a representation for the Huffman tree at the beginning of the encoded output file to be used by the decoding function. If you have trouble with devising such a representation, see Section 6.5. 6 Non-Binary Trees Many organizations are hierarchical in nature, such as the military and most busi- nesses. Consider a company with a president and some number of vice presidents who report to the president. Each vice president has some number of direct sub- ordinates, and so on. If we wanted to model this company with a data structure, it would be natural to think of the president in the root node of a tree, the vice presi- dents at level 1, and their subordinates at lower levels in the tree as we go down the organizational hierarchy. Because the number of vice presidents is likely to be more than two, this com- pany’s organization cannot easily be represented by a binary tree. We need instead to use a tree whose nodes have an arbitrary number of children. Unfortunately, when we permit trees to have nodes with an arbitrary number of children, they be- come much harder to implement than binary trees. We consider such trees in this chapter. To distinguish them from binary trees, we use the term general tree. Section 6.1 presents general tree terminology. Section 6.2 presents a simple representation for solving the important problem of processing equivalence classes. Several pointer-based implementations for general trees are covered in Section 6.3. Aside from general trees and binary trees, there are also uses for trees whose in- ternal nodes have a fixed number K of children where K is something other than two. Such trees are known as K-ary trees. Section 6.4 generalizes the properties of binary trees to K-ary trees. Sequential representations, useful for applications such as storing trees on disk, are covered in Section 6.5. 6.1 General Tree Definitions and Terminology A tree T is a finite set of one or more nodes such that there is one designated node R, called the root of T. If the set (T{R}) is not empty, these nodes are partitioned into n>0 disjoint subsets T0, T1, ..., Tn1, each of which is a tree, and whose roots R1, R2, ..., Rn, respectively, are children of R. The subsets Ti (0  i class GTNode { public: E value(); // Return node’s value bool isLeaf(); // True if node is a leaf GTNode* parent(); // Return parent GTNode* leftmostChild(); // Return first child GTNode* rightSibling(); // Return right sibling void setValue(E&); // Set node’s value void insertFirst(GTNode*); // Insert first child void insertNext(GTNode*); // Insert next sibling void removeFirst(); // Remove first child void removeNext(); // Remove right sibling }; // General tree ADT template class GenTree { public: void clear(); // Send all nodes to free store GTNode* root(); // Return the root of the tree // Combine two subtrees void newroot(E&, GTNode*, GTNode*); void print(); // Print a tree }; Figure 6.2 Definitions for the general tree and general tree node One choice would be to provide a function that takes as its parameter the index for the desired child. That combined with a function that returns the number of children for a given node would support the ability to access any node or process all children of a node. Unfortunately, this view of access tends to bias the choice for node implementations in favor of an array-based approach, because these functions favor random access to a list of children. In practice, an implementation based on a linked list is often preferred. An alternative is to provide access to the first (or leftmost) child of a node, and to provide access to the next (or right) sibling of a node. Figure 6.2 shows class declarations for general trees and their nodes. Based on these two access functions, the children of a node can be traversed like a list. Trying to find the next sibling of the rightmost sibling would return NULL. 6.1.2 General Tree Traversals In Section 5.2, three tree traversals were presented for binary trees: preorder, pos- torder, and inorder. For general trees, preorder and postorder traversals are defined with meanings similar to their binary tree counterparts. Preorder traversal of a gen- eral tree first visits the root of the tree, then performs a preorder traversal of each subtree from left to right. A postorder traversal of a general tree performs a pos- torder traversal of the root’s subtrees from left to right, then visits the root. Inorder 206 Chap. 6 Non-Binary Trees B DE FC A R Figure 6.3 An example of a general tree. traversal does not have a natural definition for the general tree, because there is no particular number of children for an internal node. An arbitrary definition — such as visit the leftmost subtree in inorder, then the root, then visit the remaining sub- trees in inorder — can be invented. However, inorder traversals are generally not useful with general trees. Example 6.1 A preorder traversal of the tree in Figure 6.3 visits the nodes in order RACDEBF. A postorder traversal of this tree visits the nodes in order CDEAF BR. To perform a preorder traversal, it is necessary to visit each of the children for a given node (say R) from left to right. This is accomplished by starting at R’s leftmost child (call it T). From T, we can move to T’s right sibling, and then to that node’s right sibling, and so on. Using the ADT of Figure 6.2, here is a C++ implementation to print the nodes of a general tree in preorder. Note the for loop at the end, which processes the list of children by beginning with the leftmost child, then repeatedly moving to the next child until calling next returns NULL. // Print using a preorder traversal void printhelp(GTNode* root) { if (root->isLeaf()) cout << "Leaf: "; else cout << "Internal: "; cout << root->value() << "\n"; // Now process the children of "root" for (GTNode* temp = root->leftmostChild(); temp != NULL; temp = temp->rightSibling()) printhelp(temp); } Sec. 6.2 The Parent Pointer Implementation 207 6.2 The Parent Pointer Implementation Perhaps the simplest general tree implementation is to store for each node only a pointer to that node’s parent. We will call this the parent pointer implementation. Clearly this implementation is not general purpose, because it is inadequate for such important operations as finding the leftmost child or the right sibling for a node. Thus, it may seem to be a poor idea to implement a general tree in this way. However, the parent pointer implementation stores precisely the information required to answer the following, useful question: “Given two nodes, are they in the same tree?” To answer the question, we need only follow the series of parent pointers from each node to its respective root. If both nodes reach the same root, then they must be in the same tree. If the roots are different, then the two nodes are not in the same tree. The process of finding the ultimate root for a given node we will call FIND. The parent pointer representation is most often used to maintain a collection of disjoint sets. Two disjoint sets share no members in common (their intersection is empty). A collection of disjoint sets partitions some objects such that every object is in exactly one of the disjoint sets. There are two basic operations that we wish to support: (1) determine if two objects are in the same set, and (2) merge two sets together. Because two merged sets are united, the merging operation is called UNION and the whole process of determining if two objects are in the same set and then merging the sets goes by the name “UNION/FIND.” To implement UNION/FIND, we represent each disjoint set with a separate general tree. Two objects are in the same disjoint set if they are in the same tree. Every node of the tree (except for the root) has precisely one parent. Thus, each node requires the same space to represent it. The collection of objects is typically stored in an array, where each element of the array corresponds to one object, and each element stores the object’s value. The objects also correspond to nodes in the various disjoint trees (one tree for each disjoint set), so we also store the parent value with each object in the array. Those nodes that are the roots of their respective trees store an appropriate indicator. Note that this representation means that a single array is being used to implement a collection of trees. This makes it easy to merge trees together with UNION operations. Figure 6.4 shows the parent pointer implementation for the general tree, called ParPtrTree. This class is greatly simplified from the declarations of Figure 6.2 because we need only a subset of the general tree operations. Instead of implement- ing a separate node class, ParPtrTree simply stores an array where each array element corresponds to a node of the tree. Each position i of the array stores the value for node i and the array position for the parent of node i. Class ParPtrTree 208 Chap. 6 Non-Binary Trees // General tree representation for UNION/FIND class ParPtrTree { private: int* array; // Node array int size; // Size of node array int FIND(int) const; // Find root public: ParPtrTree(int); // Constructor ˜ParPtrTree() { delete [] array; } // Destructor void UNION(int, int); // Merge equivalences bool differ(int, int); // True if not in same tree }; int ParPtrTree::FIND(int curr) const { // Find root while (array[curr] != ROOT) curr = array[curr]; return curr; // At root } Figure 6.4 General tree implementation using parent pointers for the UNION/ FIND algorithm. is given two new methods, differ and UNION. Method differ checks if two objects are in different sets, and method UNION merges two sets together. A private method FIND is used to find the ultimate root for an object. An application using the UNION/FIND operations should store a set of n ob- jects, where each object is assigned a unique index in the range 0 to n 1. The indices refer to the corresponding parent pointers in the array. Class ParPtrTree creates and initializes the UNION/FIND array, and methods differ and UNION take array indices as inputs. Figure 6.5 illustrates the parent pointer implementation. Note that the nodes can appear in any order within the array, and the array can store up to n separate trees. For example, Figure 6.5 shows two trees stored in the same array. Thus, a single array can store a collection of items distributed among an arbitrary (and changing) number of disjoint subsets. Consider the problem of assigning the members of a set to disjoint subsets called equivalence classes. Recall from Section 2.1 that an equivalence relation is reflexive, symmetric, and transitive. Thus, if objects A and B are equivalent, and objects B and C are equivalent, we must be able to recognize that objects A and C are also equivalent. There are many practical uses for disjoint sets and representing equivalences. For example, consider Figure 6.6 which shows a graph of ten nodes labeled A through J. Notice that for nodes A through I, there is some series of edges that connects any pair of the nodes, but node J is disconnected from the rest of the nodes. Such a graph might be used to represent connections such as wires be- tween components on a circuit board, or roads between cities. We can consider two nodes of the graph to be equivalent if there is a path between them. Thus, Sec. 6.2 The Parent Pointer Implementation 209 CD F W XY Parent’s Index 1 1 1 2 EDCBARLabel Z WZYXF 00777 BA E R Node Index 0 1 2 3 4 5 6 7 8 9 10 Figure 6.5 The parent pointer array implementation. Each node corresponds to a position in the node array, which stores its value and a pointer to its parent. The parent pointers are represented by the position in the array of the parent. The root of any tree stores ROOT, represented graphically by a slash in the “Parent’s Index” box. This figure shows two trees stored in the same parent pointer array, one rooted at R, and the other rooted at W. B E D G FJA C I H Figure 6.6 A graph with two connected components. nodes A, H, and E would be equivalent in Figure 6.6, but J is not equivalent to any other. A subset of equivalent (connected) edges in a graph is called a connected component. The goal is to quickly classify the objects into disjoint sets that corre- spond to the connected components. Another application for UNION/FIND occurs in Kruskal’s algorithm for computing the minimal cost spanning tree for a graph (Section 11.5.2). The input to the UNION/FIND algorithm is typically a series of equivalence pairs. In the case of the connected components example, the equivalence pairs would simply be the set of edges in the graph. An equivalence pair might say that object C is equivalent to object A. If so, C and A are placed in the same subset. If a later equivalence relates A and B, then by implication C is also equivalent to B. Thus, an equivalence pair may cause two subsets to merge, each of which contains several objects. 210 Chap. 6 Non-Binary Trees Equivalence classes can be managed efficiently with the UNION/FIND alg- orithm. Initially, each object is at the root of its own tree. An equivalence pair is processed by checking to see if both objects of the pair are in the same tree us- ing method differ. If they are in the same tree, then no change need be made because the objects are already in the same equivalence class. Otherwise, the two equivalence classes should be merged by the UNION method. Example 6.2 As an example of solving the equivalence class problem, consider the graph of Figure 6.6. Initially, we assume that each node of the graph is in a distinct equivalence class. This is represented by storing each as the root of its own tree. Figure 6.7(a) shows this initial configuration using the parent pointer array representation. Now, consider what happens when equivalence relationship (A, B) is processed. The root of the tree containing A is A, and the root of the tree containing B is B. To make them equivalent, one of these two roots is set to be the parent of the other. In this case it is irrelevant which points to which, so we arbitrarily select the first in alphabetical order to be the root. This is represented in the parent pointer array by setting the parent field of B (the node in array position 1 of the array) to store a pointer to A. Equivalence pairs (C, H), (G, F), and (D, E) are processed in similar fashion. When processing the equivalence pair (I, F), because I and F are both their own roots, I is set to point to F. Note that this also makes G equivalent to I. The result of processing these five equivalences is shown in Figure 6.7(b). The parent pointer representation places no limit on the number of nodes that can share a parent. To make equivalence processing as efficient as possible, the distance from each node to the root of its respective tree should be as small as possible. Thus, we would like to keep the height of the trees small when merging two equivalence classes together. Ideally, each tree would have all nodes pointing directly to the root. Achieving this goal all the time would require too much ad- ditional processing to be worth the effort, so we must settle for getting as close as possible. A low-cost approach to reducing the height is to be smart about how two trees are joined together. One simple technique, called the weighted union rule, joins the tree with fewer nodes to the tree with more nodes by making the smaller tree’s root point to the root of the bigger tree. This will limit the total depth of the tree to O(log n), because the depth of nodes only in the smaller tree will now increase by one, and the depth of the deepest node in the combined tree can only be at most one deeper than the deepest node before the trees were combined. The total number of nodes in the combined tree is therefore at least twice the number in the smaller Sec. 6.2 The Parent Pointer Implementation 211 0123456789 0123456789 0123456789 (a) (b) (c) (d) 0123456789 BHG ACF AF CG H F A BC GD E H J JD E DB E J BCDEFGH BCD FGH G BCDE GH A A A E F I BCD J I I E H J F A I I I I J J 005 5 0352 32 0053 255 AB DEFGHIC 5 5 5 J Figure 6.7 An example of equivalence processing. (a) Initial configuration for the ten nodes of the graph in Figure 6.6. The nodes are placed into ten independent equivalence classes. (b) The result of processing five edges: (A, B), (C, H), (G, F), (D, E), and (I, F). (c) The result of processing two more edges: (H, A) and (E, G). (d) The result of processing edge (H, E). 212 Chap. 6 Non-Binary Trees subtree. Thus, the depth of any node can be increased at most log n times when n equivalences are processed. Example 6.3 When processing equivalence pair (I, F) in Figure 6.7(b), F is the root of a tree with two nodes while I is the root of a tree with only one node. Thus, I is set to point to F rather than the other way around. Figure 6.7(c) shows the result of processing two more equivalence pairs: (H, A) and (E, G). For the first pair, the root for H is C while the root for A is itself. Both trees contain two nodes, so it is an arbitrary decision as to which node is set to be the root for the combined tree. In the case of equivalence pair (E, G), the root of E is D while the root of G is F. Because F is the root of the larger tree, node D is set to point to F. Not all equivalences will combine two trees. If equivalence (F, G) is processed when the representation is in the state shown in Figure 6.7(c), no change will be made because F is already the root for G. The weighted union rule helps to minimize the depth of the tree, but we can do better than this. Path compression is a method that tends to create extremely shal- low trees. Path compression takes place while finding the root for a given node X. Call this root R. Path compression resets the parent of every node on the path from X to R to point directly to R. This can be implemented by first finding R. A second pass is then made along the path from X to R, assigning the parent field of each node encountered to R. Alternatively, a recursive algorithm can be implemented as follows. This version of FIND not only returns the root of the current node, but also makes all ancestors of the current node point to the root. // FIND with path compression int ParPtrTree::FIND(int curr) const { if (array[curr] == ROOT) return curr; // At root array[curr] = FIND(array[curr]); return array[curr]; } Example 6.4 Figure 6.7(d) shows the result of processing equivalence pair (H, E) on the the representation shown in Figure 6.7(c) using the stan- dard weighted union rule without path compression. Figure 6.8 illustrates the path compression process for the same equivalence pair. After locating the root for node H, we can perform path compression to make H point directly to root object A. Likewise, E is set to point directly to its root, F. Finally, object A is set to point to root object F. Note that path compression takes place during the FIND operation, not during the UNION operation. In Figure 6.8, this means that nodes B, C, and H have node A remain as their parent, rather than changing their parent to Sec. 6.3 General Tree Implementations 213 50055 505 ABCD J 9876543210 A B GE EFGHI J CH ID F Figure 6.8 An example of path compression, showing the result of processing equivalence pair (H, E) on the representation of Figure 6.7(c). be F. While we might prefer to have these nodes point to F, to accomplish this would require that additional information from the FIND operation be passed back to the UNION operation. This would not be practical. Path compression keeps the cost of each FIND operation very close to constant. To be more precise about what is meant by “very close to constant,” the cost of path compression for n FIND operations on n nodes (when combined with the weighted union rule for joining sets) is approximately1 ⇥(n log⇤ n). The notation “log⇤ n” means the number of times that the log of n must be taken before n  1. For example, log⇤ 65536 is 4 because log 65536 = 16, log 16 = 4, log 4 = 2, and finally log 2 = 1. Thus, log⇤ n grows very slowly, so the cost for a series of n FIND operations is very close to n. Note that this does not mean that the tree resulting from processing n equiva- lence pairs necessarily has depth ⇥(log⇤ n). One can devise a series of equivalence operations that yields ⇥(log n) depth for the resulting tree. However, many of the equivalences in such a series will look only at the roots of the trees being merged, requiring little processing time. The total amount of processing time required for n operations will be ⇥(n log⇤ n), yielding nearly constant time for each equiva- lence operation. This is an example of amortized analysis, discussed further in Section 14.3. 6.3 General Tree Implementations We now tackle the problem of devising an implementation for general trees that allows efficient processing for all member functions of the ADTs shown in Fig- ure 6.2. This section presents several approaches to implementing general trees. Each implementation yields advantages and disadvantages in the amount of space required to store a node and the relative ease with which key operations can be performed. General tree implementations should place no restriction on how many 1To be more precise, this cost has been found to grow in time proportional to the inverse of Ackermann’s function. See Section 6.6. 214 Chap. 6 Non-Binary Trees R BA CDE F Index Val Par 0 1 2 3 4 5 6 7 R A C B D F E 0 1 0 1 3 1 3 246 5 1 Figure 6.9 The “list of children” implementation for general trees. The col- umn of numbers to the left of the node array labels the array indices. The column labeled “Val” stores node values. The column labeled “Par” stores indices (or pointers) to the parents. The last column stores pointers to the linked list of chil- dren for each internal node. Each element of the linked list stores a pointer to one of the node’s children (shown as the array index of the target node). children a node may have. In some applications, once a node is created the number of children never changes. In such cases, a fixed amount of space can be allocated for the node when it is created, based on the number of children for the node. Mat- ters become more complicated if children can be added to or deleted from a node, requiring that the node’s space allocation be adjusted accordingly. 6.3.1 List of Children Our first attempt to create a general tree implementation is called the “list of chil- dren” implementation for general trees. It simply stores with each internal node a linked list of its children. This is illustrated by Figure 6.9. The “list of children” implementation stores the tree nodes in an array. Each node contains a value, a pointer (or index) to its parent, and a pointer to a linked list of the node’s children, stored in order from left to right. Each linked list element contains a pointer to one child. Thus, the leftmost child of a node can be found directly because it is the first element in the linked list. However, to find the right sibling for a node is more difficult. Consider the case of a node M and its parent P. To find M’s right sibling, we must move down the child list of P until the linked list element storing the pointer to M has been found. Going one step further takes us to the linked list element that stores a pointer to M’s right sibling. Thus, in the worst case, to find M’s right sibling requires that all children of M’s parent be searched. Sec. 6.3 General Tree Implementations 215 R’ Left Val ParRight R BA CDE F X X7 1 1 1 0 2 1 3 4 R A B C D E F 5 R’8 6 20 Figure 6.10 The “left-child/right-sibling” implementation. Combining trees using this representation is difficult if each tree is stored in a separate node array. If the nodes of both trees are stored in a single node array, then adding tree T as a subtree of node R is done by simply adding the root of T to R’s list of children. 6.3.2 The Left-Child/Right-Sibling Implementation With the “list of children” implementation, it is difficult to access a node’s right sibling. Figure 6.10 presents an improvement. Here, each node stores its value and pointers to its parent, leftmost child, and right sibling. Thus, each of the basic ADT operations can be implemented by reading a value directly from the node. If two trees are stored within the same node array, then adding one as the subtree of the other simply requires setting three pointers. Combining trees in this way is illustrated by Figure 6.11. This implementation is more space efficient than the “list of children” implementation, and each node requires a fixed amount of space in the node array. 6.3.3 Dynamic Node Implementations The two general tree implementations just described use an array to store the col- lection of nodes. In contrast, our standard implementation for binary trees stores each node as a separate dynamic object containing its value and pointers to its two children. Unfortunately, nodes of a general tree can have any number of children, and this number may change during the life of the node. A general tree node imple- mentation must support these properties. One solution is simply to limit the number of children permitted for any node and allocate pointers for exactly that number of 216 Chap. 6 Non-Binary Trees 0 1 1 1 7 2 0R’ XR BA DE F R Left Val ParRight C 18 3A 2 6B C4 D5 E F X 0 7 R’ Figure 6.11 Combining two trees that use the “left-child/right-sibling” imple- mentation. The subtree rooted at R in Figure 6.10 now becomes the first child of R0. Three pointers are adjusted in the node array: The left-child field of R0 now points to node R, while the right-sibling field for R points to node X. The parent field of node R points to node R0. children. There are two major objections to this. First, it places an undesirable limit on the number of children, which makes certain trees unrepresentable by this implementation. Second, this might be extremely wasteful of space because most nodes will have far fewer children and thus leave some pointer positions empty. The alternative is to allocate variable space for each node. There are two basic approaches. One is to allocate an array of child pointers as part of the node. In essence, each node stores an array-based list of child pointers. Figure 6.12 illus- trates the concept. This approach assumes that the number of children is known when the node is created, which is true for some applications but not for others. It also works best if the number of children does not change. If the number of children does change (especially if it increases), then some special recovery mech- anism must be provided to support a change in the size of the child pointer array. One possibility is to allocate a new node of the correct size from free store and re- turn the old copy of the node to free store for later reuse. This works especially well in a language with built-in garbage collection such as Java. For example, assume that a node M initially has two children, and that space for two child pointers is al- located when M is created. If a third child is added to M, space for a new node with three child pointers can be allocated, the contents of M is copied over to the new space, and the old space is then returned to free store. As an alternative to relying on the system’s garbage collector, a memory manager for variable size storage units can be implemented, as described in Section 12.3. Another possibility is to use a collection of free lists, one for each array size, as described in Section 4.1.2. Note Sec. 6.3 General Tree Implementations 217 Val Size (b)(a) F B DEC A R R2 A3 B1 C0 D0 E0 F0 Figure 6.12 A dynamic general tree representation with fixed-size arrays for the child pointers. (a) The general tree. (b) The tree representation. For each node, the first field stores the node value while the second field stores the size of the child pointer array. (b)(a) B FED R C A R BA CDE F Figure 6.13 A dynamic general tree representation with linked lists of child pointers. (a) The general tree. (b) The tree representation. in Figure 6.12 that the current number of children for each node is stored explicitly in a size field. The child pointers are stored in an array with size elements. Another approach that is more flexible, but which requires more space, is to store a linked list of child pointers with each node as illustrated by Figure 6.13. This implementation is essentially the same as the “list of children” implementation of Section 6.3.1, but with dynamically allocated nodes rather than storing the nodes in an array. 218 Chap. 6 Non-Binary Trees (a) root (b) Figure 6.14 Converting from a forest of general trees to a single binary tree. Each node stores pointers to its left child and right sibling. The tree roots are assumed to be siblings for the purpose of converting. 6.3.4 Dynamic “Left-Child/Right-Sibling” Implementation The “left-child/right-sibling” implementation of Section 6.3.2 stores a fixed number of pointers with each node. This can be readily adapted to a dynamic implemen- tation. In essence, we substitute a binary tree for a general tree. Each node of the “left-child/right-sibling” implementation points to two “children” in a new binary tree structure. The left child of this new structure is the node’s first child in the general tree. The right child is the node’s right sibling. We can easily extend this conversion to a forest of general trees, because the roots of the trees can be con- sidered siblings. Converting from a forest of general trees to a single binary tree is illustrated by Figure 6.14. Here we simply include links from each node to its right sibling and remove links to all children except the leftmost child. Figure 6.15 shows how this might look in an implementation with two pointers at each node. Compared with the implementation illustrated by Figure 6.13 which requires over- head of three pointers/node, the implementation of Figure 6.15 only requires two pointers per node. The representation of Figure 6.15 is likely to be easier to imple- ment, space efficient, and more flexible than the other implementations presented in this section. 6.4 K-ary Trees K-ary trees are trees whose internal nodes all have exactly K children. Thus, a full binary tree is a 2-ary tree. The PR quadtree discussed in Section 13.3 is an example of a 4-ary tree. Because K-ary tree nodes have a fixed number of children, unlike general trees, they are relatively easy to implement. In general, K-ary trees bear many similarities to binary trees, and similar implementations can be used for K-ary tree nodes. Note that as K becomes large, the potential number of NULL pointers grows, and the difference between the required sizes for internal nodes and leaf nodes increases. Thus, as K becomes larger, the need to choose separate implementations for the internal and leaf nodes becomes more pressing. Sec. 6.5 Sequential Tree Implementations 219 (a) B FED R C A A R BC DF E (b) Figure 6.15 A general tree converted to the dynamic “left-child/right-sibling” representation. Compared to the representation of Figure 6.13, this representation requires less space. (a) (b) Figure 6.16 Full and complete 3-ary trees. (a) This tree is full (but not complete). (b) This tree is complete (but not full). Full and complete K-ary trees are analogous to full and complete binary trees, respectively. Figure 6.16 shows full and complete K-ary trees for K =3. In practice, most applications of K-ary trees limit them to be either full or complete. Many of the properties of binary trees extend to K-ary trees. Equivalent theo- rems to those in Section 5.1.1 regarding the number of NULL pointers in a K-ary tree and the relationship between the number of leaves and the number of internal nodes in a K-ary tree can be derived. We can also store a complete K-ary tree in an array, using simple formulas to compute a node’s relations in a manner similar to that used in Section 5.3.3. 6.5 Sequential Tree Implementations Next we consider a fundamentally different approach to implementing trees. The goal is to store a series of node values with the minimum information needed to reconstruct the tree structure. This approach, known as a sequential tree imple- mentation, has the advantage of saving space because no pointers are stored. It has 220 Chap. 6 Non-Binary Trees the disadvantage that accessing any node in the tree requires sequentially process- ing all nodes that appear before it in the node list. In other words, node access must start at the beginning of the node list, processing nodes sequentially in whatever order they are stored until the desired node is reached. Thus, one primary virtue of the other implementations discussed in this section is lost: efficient access (typi- cally ⇥(log n) time) to arbitrary nodes in the tree. Sequential tree implementations are ideal for archiving trees on disk for later use because they save space, and the tree structure can be reconstructed as needed for later processing. Sequential tree implementations can be used to serialize a tree structure. Seri- alization is the process of storing an object as a series of bytes, typically so that the data structure can be transmitted between computers. This capability is important when using data structures in a distributed processing environment. A sequential tree implementation typically stores the node values as they would be enumerated by a preorder traversal, along with sufficient information to describe the tree’s shape. If the tree has restricted form, for example if it is a full binary tree, then less information about structure typically needs to be stored. A general tree, because it has the most flexible shape, tends to require the most additional shape information. There are many possible sequential tree implementation schemes. We will begin by describing methods appropriate to binary trees, then generalize to an implementation appropriate to a general tree structure. Because every node of a binary tree is either a leaf or has two (possibly empty) children, we can take advantage of this fact to implicitly represent the tree’s struc- ture. The most straightforward sequential tree implementation lists every node value as it would be enumerated by a preorder traversal. Unfortunately, the node values alone do not provide enough information to recover the shape of the tree. In particular, as we read the series of node values, we do not know when a leaf node has been reached. However, we can treat all non-empty nodes as internal nodes with two (possibly empty) children. Only NULL values will be interpreted as leaf nodes, and these can be listed explicitly. Such an augmented node list provides enough information to recover the tree structure. Example 6.5 For the binary tree of Figure 6.17, the corresponding se- quential representation would be as follows (assuming that ‘/’ stands for NULL): AB/D//CEG///FH//I// (6.1) To reconstruct the tree structure from this node list, we begin by setting node A to be the root. A’s left child will be node B. Node B’s left child is a NULL pointer, so node D must be B’s right child. Node D has two NULL children, so node C must be the right child of node A. Sec. 6.5 Sequential Tree Implementations 221 GI EF A CB D H Figure 6.17 Sample binary tree for sequential tree implementation examples. To illustrate the difficulty involved in using the sequential tree representation for processing, consider searching for the right child of the root node. We must first move sequentially through the node list of the left subtree. Only at this point do we reach the value of the root’s right child. Clearly the sequential representation is space efficient, but not time efficient for descending through the tree along some arbitrary path. Assume that each node value takes a constant amount of space. An example would be if the node value is a positive integer and NULL is indicated by the value zero. From the Full Binary Tree Theorem of Section 5.1.1, we know that the size of the node list will be about twice the number of nodes (i.e., the overhead fraction is 1/2). The extra space is required by the NULL pointers. We should be able to store the node list more compactly. However, any sequential implementation must recognize when a leaf node has been reached, that is, a leaf node indicates the end of a subtree. One way to do this is to explicitly list with each node whether it is an internal node or a leaf. If a node X is an internal node, then we know that its two children (which may be subtrees) immediately follow X in the node list. If X is a leaf node, then the next node in the list is the right child of some ancestor of X, not the right child of X. In particular, the next node will be the child of X’s most recent ancestor that has not yet seen its right child. However, this assumes that each internal node does in fact have two children, in other words, that the tree is full. Empty children must be indicated in the node list explicitly. Assume that internal nodes are marked with a prime (0) and that leaf nodes show no mark. Empty children of internal nodes are indicated by ‘/’, but the (empty) children of leaf nodes are not represented at all. Note that a full binary tree stores no NULL values with this implementation, and so requires less overhead. Example 6.6 We can represent the tree of Figure 6.17 as follows: A0B0/DC0E0G/F0HI (6.2) 222 Chap. 6 Non-Binary Trees Note that slashes are needed for the empty children because this is not a full binary tree. Storing n extra bits can be a considerable savings over storing n NULL values. In Example 6.6, each node is shown with a mark if it is internal, or no mark if it is a leaf. This requires that each node value has space to store the mark bit. This might be true if, for example, the node value were stored as a 4-byte integer but the range of the values sored was small enough so that not all bits are used. An example would be if all node values must be positive. Then the high-order (sign) bit of the integer value could be used as the mark bit. Another approach is to store a separate bit vector to represent the status of each node. In this case, each node of the tree corresponds to one bit in the bit vector. A value of ‘1’ could indicate an internal node, and ‘0’ could indicate a leaf node. Example 6.7 The bit vector for the tree if Figure 6.17 (including positions for the null children of nodes B and E) would be 11001100100 (6.3) Storing general trees by means of a sequential implementation requires that more explicit structural information be included with the node list. Not only must the general tree implementation indicate whether a node is leaf or internal, it must also indicate how many children the node has. Alternatively, the implementation can indicate when a node’s child list has come to an end. The next example dis- penses with marks for internal or leaf nodes. Instead it includes a special mark (we will use the “)” symbol) to indicate the end of a child list. All leaf nodes are fol- lowed by a “)” symbol because they have no children. A leaf node that is also the last child for its parent would indicate this by two or more successive “)” symbols. Example 6.8 For the general tree of Figure 6.3, we get the sequential representation RAC)D)E))BF))) (6.4) Note that F is followed by three “)” marks, because it is a leaf, the last node of B’s rightmost subtree, and the last node of R’s rightmost subtree. Note that this representation for serializing general trees cannot be used for bi- nary trees. This is because a binary tree is not merely a restricted form of general tree with at most two children. Every binary tree node has a left and a right child, though either or both might be empty. For example, the representation of Exam- ple 6.8 cannot let us distinguish whether node D in Figure 6.17 is the left or right child of node B. Sec. 6.6 Further Reading 223 6.6 Further Reading The expression log⇤ n cited in Section 6.2 is closely related to the inverse of Ack- ermann’s function. For more information about Ackermann’s function and the cost of path compression for UNION/FIND, see Robert E. Tarjan’s paper “On the effi- ciency of a good but not linear set merging algorithm” [Tar75]. The article “Data Structures and Algorithms for Disjoint Set Union Problems” by Galil and Italiano [GI91] covers many aspects of the equivalence class problem. Foundations of Multidimensional and Metric Data Structures by Hanan Samet [Sam06] treats various implementations of tree structures in detail within the con- text of K-ary trees. Samet covers sequential implementations as well as the linked and array implementations such as those described in this chapter and Chapter 5. While these books are ostensibly concerned with spatial data structures, many of the concepts treated are relevant to anyone who must implement tree structures. 6.7 Exercises 6.1 Write an algorithm to determine if two general trees are identical. Make the algorithm as efficient as you can. Analyze your algorithm’s running time. 6.2 Write an algorithm to determine if two binary trees are identical when the ordering of the subtrees for a node is ignored. For example, if a tree has root node with value R, left child with value A and right child with value B, this would be considered identical to another tree with root node value R, left child value B, and right child value A. Make the algorithm as efficient as you can. Analyze your algorithm’s running time. How much harder would it be to make this algorithm work on a general tree? 6.3 Write a postorder traversal function for general trees, similar to the preorder traversal function named preorder given in Section 6.1.2. 6.4 Write a function that takes as input a general tree and returns the number of nodes in that tree. Write your function to use the GenTree and GTNode ADTs of Figure 6.2. 6.5 Describe how to implement the weighted union rule efficiently. In particular, describe what information must be stored with each node and how this infor- mation is updated when two trees are merged. Modify the implementation of Figure 6.4 to support the weighted union rule. 6.6 A potential alternative to the weighted union rule for combining two trees is the height union rule. The height union rule requires that the root of the tree with greater height become the root of the union. Explain why the height union rule can lead to worse average time behavior than the weighted union rule. 6.7 Using the weighted union rule and path compression, show the array for the parent pointer implementation that results from the following series of 224 Chap. 6 Non-Binary Trees equivalences on a set of objects indexed by the values 0 through 15. Initially, each element in the set should be in a separate equivalence class. When two trees to be merged are the same size, make the root with greater index value be the child of the root with lesser index value. (0, 2) (1, 2) (3, 4) (3, 1) (3, 5) (9, 11) (12, 14) (3, 9) (4, 14) (6, 7) (8, 10) (8, 7) (7, 0) (10, 15) (10, 13) 6.8 Using the weighted union rule and path compression, show the array for the parent pointer implementation that results from the following series of equivalences on a set of objects indexed by the values 0 through 15. Initially, each element in the set should be in a separate equivalence class. When two trees to be merged are the same size, make the root with greater index value be the child of the root with lesser index value. (2, 3) (4, 5) (6, 5) (3, 5) (1, 0) (7, 8) (1, 8) (3, 8) (9, 10) (11, 14) (11, 10) (12, 13) (11, 13) (14, 1) 6.9 Devise a series of equivalence statements for a collection of sixteen items that yields a tree of height 5 when both the weighted union rule and path compression are used. What is the total number of parent pointers followed to perform this series? 6.10 One alternative to path compression that gives similar performance gains is called path halving. In path halving, when the path is traversed from the node to the root, we make the grandparent of every other node i on the path the new parent of i. Write a version of FIND that implements path halving. Your FIND operation should work as you move up the tree, rather than require the two passes needed by path compression. 6.11 Analyze the fraction of overhead required by the “list of children” imple- mentation, the “left-child/right-sibling” implementation, and the two linked implementations of Section 6.3.3. How do these implementations compare in space efficiency? 6.12 Using the general tree ADT of Figure 6.2, write a function that takes as input the root of a general tree and returns a binary tree generated by the conversion process illustrated by Figure 6.14. 6.13 Use mathematical induction to prove that the number of leaves in a non- empty full K-ary tree is (K 1)n +1, where n is the number of internal nodes. 6.14 Derive the formulas for computing the relatives of a non-empty complete K-ary tree node stored in the complete tree representation of Section 5.3.3. 6.15 Find the overhead fraction for a full K-ary tree implementation with space requirements as follows: (a) All nodes store data, K child pointers, and a parent pointer. The data field requires four bytes and each pointer requires four bytes. Sec. 6.7 Exercises 225 C A B F EH IDG Figure 6.18 A sample tree for Exercise 6.16. (b) All nodes store data and K child pointers. The data field requires six- teen bytes and each pointer requires four bytes. (c) All nodes store data and a parent pointer, and internal nodes store K child pointers. The data field requires eight bytes and each pointer re- quires four bytes. (d) Only leaf nodes store data; only internal nodes store K child pointers. The data field requires four bytes and each pointer requires two bytes. 6.16 (a) Write out the sequential representation for Figure 6.18 using the coding illustrated by Example 6.5. (b) Write out the sequential representation for Figure 6.18 using the coding illustrated by Example 6.6. 6.17 Draw the binary tree representing the following sequential representation for binary trees illustrated by Example 6.5: ABD//E//C/F// 6.18 Draw the binary tree representing the following sequential representation for binary trees illustrated by Example 6.6: A0/B0/C0D0G/E Show the bit vector for leaf and internal nodes (as illustrated by Example 6.7) for this tree. 6.19 Draw the general tree represented by the following sequential representation for general trees illustrated by Example 6.8: XPC)Q)RV)M)))) 6.20 (a) Write a function to decode the sequential representation for binary trees illustrated by Example 6.5. The input should be the sequential repre- sentation and the output should be a pointer to the root of the resulting binary tree. 226 Chap. 6 Non-Binary Trees (b) Write a function to decode the sequential representation for full binary trees illustrated by Example 6.6. The input should be the sequential representation and the output should be a pointer to the root of the re- sulting binary tree. (c) Write a function to decode the sequential representation for general trees illustrated by Example 6.8. The input should be the sequential representation and the output should be a pointer to the root of the re- sulting general tree. 6.21 Devise a sequential representation for Huffman coding trees suitable for use as part of a file compression utility (see Project 5.7). 6.8 Projects 6.1 Write classes that implement the general tree class declarations of Figure 6.2 using the dynamic “left-child/right-sibling” representation described in Sec- tion 6.3.4. 6.2 Write classes that implement the general tree class declarations of Figure 6.2 using the linked general tree implementation with child pointer arrays of Fig- ure 6.12. Your implementation should support only fixed-size nodes that do not change their number of children once they are created. Then, re- implement these classes with the linked list of children representation of Figure 6.13. How do the two implementations compare in space and time efficiency and ease of implementation? 6.3 Write classes that implement the general tree class declarations of Figure 6.2 using the linked general tree implementation with child pointer arrays of Fig- ure 6.12. Your implementation must be able to support changes in the num- ber of children for a node. When created, a node should be allocated with only enough space to store its initial set of children. Whenever a new child is added to a node such that the array overflows, allocate a new array from free store that can store twice as many children. 6.4 Implement a BST file archiver. Your program should take a BST created in main memory using the implementation of Figure 5.14 and write it out to disk using one of the sequential representations of Section 6.5. It should also be able to read in disk files using your sequential representation and create the equivalent main memory representation. 6.5 Use the UNION/FIND algorithm to implement a solution to the following problem. Given a set of points represented by their xy-coordinates, assign the points to clusters. Any two points are defined to be in the same cluster if they are within a specified distance d of each other. For the purpose of this problem, clustering is an equivalence relationship. In other words, points A, B, and C are defined to be in the same cluster if the distance between A and B Sec. 6.8 Projects 227 is less than d and the distance between A and C is also less than d, even if the distance between B and C is greater than d. To solve the problem, compute the distance between each pair of points, using the equivalence processing algorithm to merge clusters whenever two points are within the specified distance. What is the asymptotic complexity of this algorithm? Where is the bottleneck in processing? 6.6 In this project, you will run some empirical tests to determine if some vari- ations on path compression in the UNION/FIND algorithm will lead to im- proved performance. You should compare the following five implementa- tions: (a) Standard UNION/FIND with path compression and weighted union. (b) Path compression and weighted union, except that path compression is done after the UNION, instead of during the FIND operation. That is, make all nodes along the paths traversed in both trees point directly to the root of the larger tree. (c) Weighted union and path halving as described in Exercise 6.10. (d) Weighted union and a simplified form of path compression. At the end of every FIND operation, make the node point to its tree’s root (but don’t change the pointers for other nodes along the path). (e) Weighted union and a simplified form of path compression. Both nodes in the equivalence will be set to point directly to the root of the larger tree after the UNION operation. For example, consider processing the equivalence (A, B) where A0 is the root of A and B0 is the root of B. Assume the tree with root A0 is bigger than the tree with root B0. At the end of the UNION/FIND operation, nodes A, B, and B0 will all point directly to A0. PART III Sorting and Searching 229 7 Internal Sorting We sort many things in our everyday lives: A handful of cards when playing Bridge; bills and other piles of paper; jars of spices; and so on. And we have many intuitive strategies that we can use to do the sorting, depending on how many objects we have to sort and how hard they are to move around. Sorting is also one of the most frequently performed computing tasks. We might sort the records in a database so that we can search the collection efficiently. We might sort the records by zip code so that we can print and mail them more cheaply. We might use sorting as an intrinsic part of an algorithm to solve some other problem, such as when computing the minimum-cost spanning tree (see Section 11.5). Because sorting is so important, naturally it has been studied intensively and many algorithms have been devised. Some of these algorithms are straightforward adaptations of schemes we use in everyday life. Others are totally alien to how hu- mans do things, having been invented to sort thousands or even millions of records stored on the computer. After years of study, there are still unsolved problems related to sorting. New algorithms are still being developed and refined for special- purpose applications. While introducing this central problem in computer science, this chapter has a secondary purpose of illustrating issues in algorithm design and analysis. For example, this collection of sorting algorithms shows multiple approaches to us- ing divide-and-conquer. In particular, there are multiple ways to do the dividing: Mergesort divides a list in half; Quicksort divides a list into big values and small values; and Radix Sort divides the problem by working on one digit of the key at a time. Sorting algorithms can also illustrate a wide variety of analysis techniques. We’ll find that it is possible for an algorithm to have an average case whose growth rate is significantly smaller than its worse case (Quicksort). We’ll see how it is possible to speed up sorting algorithms (both Shellsort and Quicksort) by taking advantage of the best case behavior of another algorithm (Insertion sort). We’ll see several examples of how we can tune an algorithm for better performance. We’ll see that special case behavior by some algorithms makes them a good solution for 231 232 Chap. 7 Internal Sorting special niche applications (Heapsort). Sorting provides an example of a significant technique for analyzing the lower bound for a problem. Sorting will also be used to motivate the introduction to file processing presented in Chapter 8. The present chapter covers several standard algorithms appropriate for sorting a collection of records that fit in the computer’s main memory. It begins with a dis- cussion of three simple, but relatively slow, algorithms requiring ⇥(n2) time in the average and worst cases. Several algorithms with considerably better performance are then presented, some with ⇥(n log n) worst-case running time. The final sort- ing method presented requires only ⇥(n) worst-case time under special conditions. The chapter concludes with a proof that sorting in general requires ⌦(n log n) time in the worst case. 7.1 Sorting Terminology and Notation Except where noted otherwise, input to the sorting algorithms presented in this chapter is a collection of records stored in an array. Records are compared to one another by means of a comparator class, as introduced in Section 4.4. To simplify the discussion we will assume that each record has a key field whose value is ex- tracted from the record by the comparator. The key method of the comparator class is prior, which returns true when its first argument should appear prior to its sec- ond argument in the sorted list. We also assume that for every record type there is a swap function that can interchange the contents of two records in the array(see the Appendix). Given a set of records r1, r2, ..., rn with key values k1, k2, ..., kn, the Sorting Problem is to arrange the records into any order s such that records rs1 , rs2 , ..., rsn have keys obeying the property ks1  ks2  ...  ksn . In other words, the sorting problem is to arrange a set of records so that the values of their key fields are in non-decreasing order. As defined, the Sorting Problem allows input with two or more records that have the same key value. Certain applications require that input not contain duplicate key values. The sorting algorithms presented in this chapter and in Chapter 8 can handle duplicate key values unless noted otherwise. When duplicate key values are allowed, there might be an implicit ordering to the duplicates, typically based on their order of occurrence within the input. It might be desirable to maintain this initial ordering among duplicates. A sorting algorithm is said to be stable if it does not change the relative ordering of records with identical key values. Many, but not all, of the sorting algorithms presented in this chapter are stable, or can be made stable with minor changes. When comparing two sorting algorithms, the most straightforward approach would seem to be simply program both and measure their running times. An ex- ample of such timings is presented in Figure 7.20. However, such a comparison Sec. 7.2 Three ⇥(n2) Sorting Algorithms 233 can be misleading because the running time for many sorting algorithms depends on specifics of the input values. In particular, the number of records, the size of the keys and the records, the allowable range of the key values, and the amount by which the input records are “out of order” can all greatly affect the relative running times for sorting algorithms. When analyzing sorting algorithms, it is traditional to measure the number of comparisons made between keys. This measure is usually closely related to the running time for the algorithm and has the advantage of being machine and data- type independent. However, in some cases records might be so large that their physical movement might take a significant fraction of the total running time. If so, it might be appropriate to measure the number of swap operations performed by the algorithm. In most applications we can assume that all records and keys are of fixed length, and that a single comparison or a single swap operation requires a constant amount of time regardless of which keys are involved. Some special situations “change the rules” for comparing sorting algorithms. For example, an application with records or keys having widely varying length (such as sorting a sequence of variable length strings) will benefit from a special-purpose sorting technique. Some applications require that a small number of records be sorted, but that the sort be performed frequently. An example would be an application that repeatedly sorts groups of five numbers. In such cases, the constants in the runtime equations that are usually ignored in an asymptotic analysis now become crucial. Finally, some situations require that a sorting algorithm use as little memory as possible. We will note which sorting algorithms require significant extra memory beyond the input array. 7.2 Three ⇥(n2) Sorting Algorithms This section presents three simple sorting algorithms. While easy to understand and implement, we will soon see that they are unacceptably slow when there are many records to sort. Nonetheless, there are situations where one of these simple algorithms is the best tool for the job. 7.2.1 Insertion Sort Imagine that you have a stack of phone bills from the past two years and that you wish to organize them by date. A fairly natural way to do this might be to look at the first two bills and put them in order. Then take the third bill and put it into the right order with respect to the first two, and so on. As you take each bill, you would add it to the sorted pile that you have already made. This naturally intuitive process is the inspiration for our first sorting algorithm, called Insertion Sort. Insertion Sort iterates through a list of records. Each record is inserted in turn at the correct position within a sorted list composed of those records already processed. The 234 Chap. 7 Internal Sorting i=1 3 4 5 6 42 20 17 13 28 14 23 15 20 42 17 13 28 14 23 15 2 17 20 42 13 28 14 23 15 13 17 20 42 28 14 23 13 17 20 28 42 14 23 13 14 17 20 28 42 23 13 14 17 20 23 28 42 13 14 15 17 20 23 28 42 7 15 15 1515 Figure 7.1 An illustration of Insertion Sort. Each column shows the array after the iteration with the indicated value of i in the outer for loop. Values above the line in each column have been sorted. Arrows indicate the upward motions of records through the array. following is a C++ implementation. The input is an array of n records stored in array A. template void inssort(E A[], int n) { // Insertion Sort for (int i=1; i0) && (Comp::prior(A[j], A[j-1])); j--) swap(A, j, j-1); } Consider the case where inssort is processing the ith record, which has key value X. The record is moved upward in the array as long as X is less than the key value immediately above it. As soon as a key value less than or equal to X is encountered, inssort is done with that record because all records above it in the array must have smaller keys. Figure 7.1 illustrates how Insertion Sort works. The body of inssort is made up of two nested for loops. The outer for loop is executed n 1 times. The inner for loop is harder to analyze because the number of times it executes depends on how many keys in positions 1 to i 1 have a value less than that of the key in position i. In the worst case, each record must make its way to the top of the array. This would occur if the keys are initially arranged from highest to lowest, in the reverse of sorted order. In this case, the number of comparisons will be one the first time through the for loop, two the second time, and so on. Thus, the total number of comparisons will be n Xi=2 i ⇡ n2/2=⇥(n2). In contrast, consider the best-case cost. This occurs when the keys begin in sorted order from lowest to highest. In this case, every pass through the inner for loop will fail immediately, and no values will be moved. The total number Sec. 7.2 Three ⇥(n2) Sorting Algorithms 235 of comparisons will be n 1, which is the number of times the outer for loop executes. Thus, the cost for Insertion Sort in the best case is ⇥(n). While the best case is significantly faster than the worst case, the worst case is usually a more reliable indication of the “typical” running time. However, there are situations where we can expect the input to be in sorted or nearly sorted order. One example is when an already sorted list is slightly disordered by a small number of additions to the list; restoring sorted order using Insertion Sort might be a good idea if we know that the disordering is slight. Examples of algorithms that take ad- vantage of Insertion Sort’s near-best-case running time are the Shellsort algorithm of Section 7.3 and the Quicksort algorithm of Section 7.5. What is the average-case cost of Insertion Sort? When record i is processed, the number of times through the inner for loop depends on how far “out of order” the record is. In particular, the inner for loop is executed once for each key greater than the key of record i that appears in array positions 0 through i1. For example, in the leftmost column of Figure 7.1 the value 15 is preceded by five values greater than 15. Each such occurrence is called an inversion. The number of inversions (i.e., the number of values greater than a given value that occur prior to it in the array) will determine the number of comparisons and swaps that must take place. We need to determine what the average number of inversions will be for the record in position i. We expect on average that half of the keys in the first i 1 array positions will have a value greater than that of the key at position i. Thus, the average case should be about half the cost of the worst case, or around n2/4, which is still ⇥(n2). So, the average case is no better than the worst case in asymptotic complexity. Counting comparisons or swaps yields similar results. Each time through the inner for loop yields both a comparison and a swap, except the last (i.e., the comparison that fails the inner for loop’s test), which has no swap. Thus, the number of swaps for the entire sort operation is n 1 less than the number of comparisons. This is 0 in the best case, and ⇥(n2) in the average and worst cases. 7.2.2 Bubble Sort Our next sorting algorithm is called Bubble Sort. Bubble Sort is often taught to novice programmers in introductory computer science courses. This is unfortunate, because Bubble Sort has no redeeming features whatsoever. It is a relatively slow sort, it is no easier to understand than Insertion Sort, it does not correspond to any intuitive counterpart in “everyday” use, and it has a poor best-case running time. However, Bubble Sort can serve as the inspiration for a better sorting algorithm that will be presented in Section 7.2.3. Bubble Sort consists of a simple double for loop. The first iteration of the inner for loop moves through the record array from bottom to top, comparing adjacent keys. If the lower-indexed key’s value is greater than its higher-indexed 236 Chap. 7 Internal Sorting i=0 1 2 3 4 5 6 42 20 17 13 28 14 23 13 42 20 17 14 28 15 13 14 42 20 17 15 28 23 13 14 15 42 20 17 23 28 13 14 15 17 42 20 23 28 13 14 15 17 20 42 23 28 13 14 15 17 20 23 42 13 14 15 17 20 23 28 4223 2815 Figure 7.2 An illustration of Bubble Sort. Each column shows the array after the iteration with the indicated value of i in the outer for loop. Values above the line in each column have been sorted. Arrows indicate the swaps that take place during a given iteration. neighbor, then the two values are swapped. Once the smallest value is encountered, this process will cause it to “bubble” up to the top of the array. The second pass through the array repeats this process. However, because we know that the smallest value reached the top of the array on the first pass, there is no need to compare the top two elements on the second pass. Likewise, each succeeding pass through the array compares adjacent elements, looking at one less value than the preceding pass. Figure 7.2 illustrates Bubble Sort. A C++ implementation is as follows: template void bubsort(E A[], int n) { // Bubble Sort for (int i=0; ii; j--) if (Comp::prior(A[j], A[j-1])) swap(A, j, j-1); } Determining Bubble Sort’s number of comparisons is easy. Regardless of the arrangement of the values in the array, the number of comparisons made by the inner for loop is always i, leading to a total cost of n Xi=1 i ⇡ n2/2=⇥(n2). Bubble Sort’s running time is roughly the same in the best, average, and worst cases. The number of swaps required depends on how often a value is less than the one immediately preceding it in the array. We can expect this to occur for about half the comparisons in the average case, leading to ⇥(n2) for the expected number of swaps. The actual number of swaps performed by Bubble Sort will be identical to that performed by Insertion Sort. Sec. 7.2 Three ⇥(n2) Sorting Algorithms 237 i=0 1 2 3 4 5 6 42 20 17 13 28 14 23 15 13 20 17 42 28 14 23 15 13 14 17 42 28 20 23 15 13 14 15 42 28 20 23 17 13 14 15 17 28 20 23 42 13 14 15 17 20 28 23 42 13 14 15 17 20 23 28 42 13 14 15 17 20 23 28 42 Figure 7.3 An example of Selection Sort. Each column shows the array after the iteration with the indicated value of i in the outer for loop. Numbers above the line in each column have been sorted and are in their final positions. 7.2.3 Selection Sort Consider again the problem of sorting a pile of phone bills for the past year. An- other intuitive approach might be to look through the pile until you find the bill for January, and pull that out. Then look through the remaining pile until you find the bill for February, and add that behind January. Proceed through the ever-shrinking pile of bills to select the next one in order until you are done. This is the inspiration for our last ⇥(n2) sort, called Selection Sort. The ith pass of Selection Sort “se- lects” the ith smallest key in the array, placing that record into position i. In other words, Selection Sort first finds the smallest key in an unsorted list, then the second smallest, and so on. Its unique feature is that there are few record swaps. To find the next smallest key value requires searching through the entire unsorted portion of the array, but only one swap is required to put the record in place. Thus, the total number of swaps required will be n 1 (we get the last record in place “for free”). Figure 7.3 illustrates Selection Sort. Below is a C++ implementation. template void selsort(E A[], int n) { // Selection Sort for (int i=0; ii; j--) // Find the least value if (Comp::prior(A[j], A[lowindex])) lowindex = j; // Put it in place swap(A, i, lowindex); } } Selection Sort (as written here) is essentially a Bubble Sort, except that rather than repeatedly swapping adjacent values to get the next smallest record into place, we instead remember the position of the element to be selected and do one swap at the end. Thus, the number of comparisons is still ⇥(n2), but the number of swaps is much less than that required by bubble sort. Selection Sort is particularly 238 Chap. 7 Internal Sorting Key = 42 Key = 5 Key = 42 Key = 5 (a) (b) Key = 23 Key = 10 Key = 23 Key = 10 Figure 7.4 An example of swapping pointers to records. (a) A series of four records. The record with key value 42 comes before the record with key value 5. (b) The four records after the top two pointers have been swapped. Now the record with key value 5 comes before the record with key value 42. advantageous when the cost to do a swap is high, for example, when the elements are long strings or other large records. Selection Sort is more efficient than Bubble Sort (by a constant factor) in most other situations as well. There is another approach to keeping the cost of swapping records low that can be used by any sorting algorithm even when the records are large. This is to have each element of the array store a pointer to a record rather than store the record itself. In this implementation, a swap operation need only exchange the pointer values; the records themselves do not move. This technique is illustrated by Figure 7.4. Additional space is needed to store the pointers, but the return is a faster swap operation. 7.2.4 The Cost of Exchange Sorting Figure 7.5 summarizes the cost of Insertion, Bubble, and Selection Sort in terms of their required number of comparisons and swaps1 in the best, average, and worst cases. The running time for each of these sorts is ⇥(n2) in the average and worst cases. The remaining sorting algorithms presented in this chapter are significantly bet- ter than these three under typical conditions. But before continuing on, it is instruc- tive to investigate what makes these three sorts so slow. The crucial bottleneck is that only adjacent records are compared. Thus, comparisons and moves (in all but Selection Sort) are by single steps. Swapping adjacent records is called an ex- change. Thus, these sorts are sometimes referred to as exchange sorts. The cost of any exchange sort can be at best the total number of steps that the records in the 1There is a slight anomaly with Selection Sort. The supposed advantage for Selection Sort is its low number of swaps required, yet Selection Sort’s best-case number of swaps is worse than that for Insertion Sort or Bubble Sort. This is because the implementation given for Selection Sort does not avoid a swap in the case where record i is already in position i. One could put in a test to avoid swapping in this situation. But it usually takes more time to do the tests than would be saved by avoiding such swaps. Sec. 7.3 Shellsort 239 Insertion Bubble Selection Comparisons: Best Case ⇥(n) ⇥(n2) ⇥(n2) Average Case ⇥(n2) ⇥(n2) ⇥(n2) Worst Case ⇥(n2) ⇥(n2) ⇥(n2) Swaps: Best Case 0 0 ⇥(n) Average Case ⇥(n2) ⇥(n2) ⇥(n) Worst Case ⇥(n2) ⇥(n2) ⇥(n) Figure 7.5 A comparison of the asymptotic complexities for three simple sorting algorithms. array must move to reach their “correct” location (i.e., the number of inversions for each record). What is the average number of inversions? Consider a list L containing n val- ues. Define LR to be L in reverse. L has n(n1)/2 distinct pairs of values, each of which could potentially be an inversion. Each such pair must either be an inversion in L or in LR. Thus, the total number of inversions in L and LR together is exactly n(n1)/2 for an average of n(n1)/4 per list. We therefore know with certainty that any sorting algorithm which limits comparisons to adjacent items will cost at least n(n 1)/4=⌦(n2) in the average case. 7.3 Shellsort The next sorting algorithm that we consider is called Shellsort, named after its inventor, D.L. Shell. It is also sometimes called the diminishing increment sort. Unlike Insertion and Selection Sort, there is no real life intuitive equivalent to Shell- sort. Unlike the exchange sorts, Shellsort makes comparisons and swaps between non-adjacent elements. Shellsort also exploits the best-case performance of Inser- tion Sort. Shellsort’s strategy is to make the list “mostly sorted” so that a final Insertion Sort can finish the job. When properly implemented, Shellsort will give substantially better performance than ⇥(n2) in the worst case. Shellsort uses a process that forms the basis for many of the sorts presented in the following sections: Break the list into sublists, sort them, then recombine the sublists. Shellsort breaks the array of elements into “virtual” sublists. Each sublist is sorted using an Insertion Sort. Another group of sublists is then chosen and sorted, and so on. During each iteration, Shellsort breaks the list into disjoint sublists so that each element in a sublist is a fixed number of positions apart. For example, let us as- sume for convenience that n, the number of values to be sorted, is a power of two. One possible implementation of Shellsort will begin by breaking the list into n/2 240 Chap. 7 Internal Sorting 59 20 17 13 28 14 23 83 36 98 591523142813112036 28 14 11 13 36 20 17 15 98362028152314171311 11 13 14 15 17 20 23 28 36 41 42 59 65 70 83 98 11 70 65 41 42 15 83424165701798 98 42 8359 41 23 70 65 658359704241 Figure 7.6 An example of Shellsort. Sixteen items are sorted in four passes. The first pass sorts 8 sublists of size 2 and increment 8. The second pass sorts 4 sublists of size 4 and increment 4. The third pass sorts 2 sublists of size 8 and increment 2. The fourth pass sorts 1 list of size 16 and increment 1 (a regular Insertion Sort). sublists of 2 elements each, where the array index of the 2 elements in each sublist differs by n/2. If there are 16 elements in the array indexed from 0 to 15, there would initially be 8 sublists of 2 elements each. The first sublist would be the ele- ments in positions 0 and 8, the second in positions 1 and 9, and so on. Each list of two elements is sorted using Insertion Sort. The second pass of Shellsort looks at fewer, bigger lists. For our example the second pass would have n/4 lists of size 4, with the elements in the list being n/4 positions apart. Thus, the second pass would have as its first sublist the 4 elements in positions 0, 4, 8, and 12; the second sublist would have elements in positions 1, 5, 9, and 13; and so on. Each sublist of four elements would also be sorted using an Insertion Sort. The third pass would be made on two lists, one consisting of the odd positions and the other consisting of the even positions. The culminating pass in this example would be a “normal” Insertion Sort of all elements. Figure 7.6 illustrates the process for an array of 16 values where the sizes of the increments (the distances between elements on the successive passes) are 8, 4, 2, and 1. Figure 7.7 presents a C++ implementation for Shellsort. Shellsort will work correctly regardless of the size of the increments, provided that the final pass has increment 1 (i.e., provided the final pass is a regular Insertion Sort). If Shellsort will always conclude with a regular Insertion Sort, then how can it be any improvement on Insertion Sort? The expectation is that each of the (relatively cheap) sublist sorts will make the list “more sorted” than it was before. Sec. 7.4 Mergesort 241 // Modified version of Insertion Sort for varying increments template void inssort2(E A[], int n, int incr) { for (int i=incr; i=incr) && (Comp::prior(A[j], A[j-incr])); j-=incr) swap(A, j, j-incr); } template void shellsort(E A[], int n) { // Shellsort for (int i=n/2; i>2; i/=2) // For each increment for (int j=0; j(&A[j], n-j, i); inssort2(A, n, 1); } Figure 7.7 An implementation for Shell Sort. It is not necessarily the case that this will be true, but it is almost always true in practice. When the final Insertion Sort is conducted, the list should be “almost sorted,” yielding a relatively cheap final Insertion Sort pass. Some choices for increments will make Shellsort run more efficiently than oth- ers. In particular, the choice of increments described above (2k, 2k1, ..., 2, 1) turns out to be relatively inefficient. A better choice is the following series based on division by three: (..., 121, 40, 13, 4, 1). The analysis of Shellsort is difficult, so we must accept without proof that the average-case performance of Shellsort (for “divisions by three” increments) is O(n1.5). Other choices for the increment series can reduce this upper bound somewhat. Thus, Shellsort is substantially better than Insertion Sort, or any of the ⇥(n2) sorts presented in Section 7.2. In fact, Shellsort is not terrible when com- pared with the asymptotically better sorts to be presented whenever n is of medium size (thought is tends to be a little slower than these other algorithms when they are well implemented). Shellsort illustrates how we can sometimes exploit the spe- cial properties of an algorithm (in this case Insertion Sort) even if in general that algorithm is unacceptably slow. 7.4 Mergesort A natural approach to problem solving is divide and conquer. In terms of sorting, we might consider breaking the list to be sorted into pieces, process the pieces, and then put them back together somehow. A simple way to do this would be to split the list in half, sort the halves, and then merge the sorted halves together. This is the idea behind Mergesort. 242 Chap. 7 Internal Sorting 36 20 17 13 28 14 23 15 2823151436201713 20 36 13 17 14 28 15 23 13 14 15 17 20 23 28 36 Figure 7.8 An illustration of Mergesort. The first row shows eight numbers that are to be sorted. Mergesort will recursively subdivide the list into sublists of one element each, then recombine the sublists. The second row shows the four sublists of size 2 created by the first merging pass. The third row shows the two sublists of size 4 created by the next merging pass on the sublists of row 2. The last row shows the final sorted list created by merging the two sublists of row 3. Mergesort is one of the simplest sorting algorithms conceptually, and has good performance both in the asymptotic sense and in empirical running time. Surpris- ingly, even though it is based on a simple concept, it is relatively difficult to im- plement in practice. Figure 7.8 illustrates Mergesort. A pseudocode sketch of Mergesort is as follows: List mergesort(List inlist) { if (inlist.length() <= 1) return inlist;; List L1 = half of the items from inlist; List L2 = other half of the items from inlist; return merge(mergesort(L1), mergesort(L2)); } Before discussing how to implement Mergesort, we will first examine the merge function. Merging two sorted sublists is quite simple. Function merge examines the first element of each sublist and picks the smaller value as the smallest element overall. This smaller value is removed from its sublist and placed into the output list. Merging continues in this way, comparing the front elements of the sublists and continually appending the smaller to the output list until no more input elements remain. Implementing Mergesort presents a number of technical difficulties. The first decision is how to represent the lists. Mergesort lends itself well to sorting a singly linked list because merging does not require random access to the list elements. Thus, Mergesort is the method of choice when the input is in the form of a linked list. Implementing merge for linked lists is straightforward, because we need only remove items from the front of the input lists and append items to the output list. Breaking the input list into two equal halves presents some difficulty. Ideally we would just break the lists into front and back halves. However, even if we know the length of the list in advance, it would still be necessary to traverse halfway down the linked list to reach the beginning of the second half. A simpler method, which does not rely on knowing the length of the list in advance, assigns elements of the Sec. 7.4 Mergesort 243 template void mergesort(E A[], E temp[], int left, int right) { if (left == right) return; // List of one element int mid = (left+right)/2; mergesort(A, temp, left, mid); mergesort(A, temp, mid+1, right); for (int i=left; i<=right; i++) // Copy subarray to temp temp[i] = A[i]; // Do the merge operation back to A int i1 = left; int i2 = mid + 1; for (int curr=left; curr<=right; curr++) { if (i1 == mid+1) // Left sublist exhausted A[curr] = temp[i2++]; else if (i2 > right) // Right sublist exhausted A[curr] = temp[i1++]; else if (Comp::prior(temp[i1], temp[i2])) A[curr] = temp[i1++]; else A[curr] = temp[i2++]; } } Figure 7.9 Standard implementation for Mergesort. input list alternating between the two sublists. The first element is assigned to the first sublist, the second element to the second sublist, the third to first sublist, the fourth to the second sublist, and so on. This requires one complete pass through the input list to build the sublists. When the input to Mergesort is an array, splitting input into two subarrays is easy if we know the array bounds. Merging is also easy if we merge the subarrays into a second array. Note that this approach requires twice the amount of space as any of the sorting methods presented so far, which is a serious disadvantage for Mergesort. It is possible to merge the subarrays without using a second array, but this is extremely difficult to do efficiently and is not really practical. Merging the two subarrays into a second array, while simple to implement, presents another dif- ficulty. The merge process ends with the sorted list in the auxiliary array. Consider how the recursive nature of Mergesort breaks the original array into subarrays, as shown in Figure 7.8. Mergesort is recursively called until subarrays of size 1 have been created, requiring log n levels of recursion. These subarrays are merged into subarrays of size 2, which are in turn merged into subarrays of size 4, and so on. We need to avoid having each merge operation require a new array. With some difficulty, an algorithm can be devised that alternates between two arrays. A much simpler approach is to copy the sorted sublists to the auxiliary array first, and then merge them back to the original array. Figure 7.9 shows a complete implementation for mergesort following this approach. An optimized Mergesort implementation is shown in Figure 7.10. It reverses the order of the second subarray during the initial copy. Now the current positions of the two subarrays work inwards from the ends, allowing the end of each subarray 244 Chap. 7 Internal Sorting template void mergesort(E A[], E temp[], int left, int right) { if ((right-left) <= THRESHOLD) { // Small list inssort(&A[left], right-left+1); return; } int i, j, k, mid = (left+right)/2; mergesort(A, temp, left, mid); mergesort(A, temp, mid+1, right); // Do the merge operation. First, copy 2 halves to temp. for (i=mid; i>=left; i--) temp[i] = A[i]; for (j=1; j<=right-mid; j++) temp[right-j+1] = A[j+mid]; // Merge sublists back to A for (i=left,j=right,k=left; k<=right; k++) if (Comp::prior(temp[i], temp[j])) A[k] = temp[i++]; else A[k] = temp[j--]; } Figure 7.10 Optimized implementation for Mergesort. to act as a sentinel for the other. Unlike the previous implementation, no test is needed to check for when one of the two subarrays becomes empty. This version also uses Insertion Sort to sort small subarrays. Analysis of Mergesort is straightforward, despite the fact that it is a recursive algorithm. The merging part takes time ⇥(i) where i is the total length of the two subarrays being merged. The array to be sorted is repeatedly split in half until subarrays of size 1 are reached, at which time they are merged to be of size 2, these merged to subarrays of size 4, and so on as shown in Figure 7.8. Thus, the depth of the recursion is log n for n elements (assume for simplicity that n is a power of two). The first level of recursion can be thought of as working on one array of size n, the next level working on two arrays of size n/2, the next on four arrays of size n/4, and so on. The bottom of the recursion has n arrays of size 1. Thus, n arrays of size 1 are merged (requiring ⇥(n) total steps), n/2 arrays of size 2 (again requiring ⇥(n) total steps), n/4 arrays of size 4, and so on. At each of the log n levels of recursion, ⇥(n) work is done, for a total cost of ⇥(n log n). This cost is unaffected by the relative order of the values being sorted, thus this analysis holds for the best, average, and worst cases. 7.5 Quicksort While Mergesort uses the most obvious form of divide and conquer (split the list in half then sort the halves), it is not the only way that we can break down the sorting problem. And we saw that doing the merge step for Mergesort when using an array implementation is not so easy. So perhaps a different divide and conquer strategy might turn out to be more efficient? Sec. 7.5 Quicksort 245 Quicksort is aptly named because, when properly implemented, it is the fastest known general-purpose in-memory sorting algorithm in the average case. It does not require the extra array needed by Mergesort, so it is space efficient as well. Quicksort is widely used, and is typically the algorithm implemented in a library sort routine such as the UNIX qsort function. Interestingly, Quicksort is ham- pered by exceedingly poor worst-case performance, thus making it inappropriate for certain applications. Before we get to Quicksort, consider for a moment the practicality of using a Binary Search Tree for sorting. You could insert all of the values to be sorted into the BST one by one, then traverse the completed tree using an inorder traversal. The output would form a sorted list. This approach has a number of drawbacks, including the extra space required by BST pointers and the amount of time required to insert nodes into the tree. However, this method introduces some interesting ideas. First, the root of the BST (i.e., the first node inserted) splits the list into two sublists: The left subtree contains those values in the list less than the root value while the right subtree contains those values in the list greater than or equal to the root value. Thus, the BST implicitly implements a “divide and conquer” approach to sorting the left and right subtrees. Quicksort implements this concept in a much more efficient way. Quicksort first selects a value called the pivot. (This is conceptually like the root node’s value in the BST.) Assume that the input array contains k values less than the pivot. The records are then rearranged in such a way that the k values less than the pivot are placed in the first, or leftmost, k positions in the array, and the values greater than or equal to the pivot are placed in the last, or rightmost, nk positions. This is called a partition of the array. The values placed in a given partition need not (and typically will not) be sorted with respect to each other. All that is required is that all values end up in the correct partition. The pivot value itself is placed in position k. Quicksort then proceeds to sort the resulting subarrays now on either side of the pivot, one of size k and the other of size n k 1. How are these values sorted? Because Quicksort is such a good algorithm, using Quicksort on the subarrays would be appropriate. Unlike some of the sorts that we have seen earlier in this chapter, Quicksort might not seem very “natural” in that it is not an approach that a person is likely to use to sort real objects. But it should not be too surprising that a really efficient sort for huge numbers of abstract objects on a computer would be rather different from our experiences with sorting a relatively few physical objects. The C++ code for Quicksort is shown in Figure 7.11. Parameters i and j define the left and right indices, respectively, for the subarray being sorted. The initial call to Quicksort would be qsort(array, 0, n-1). Function partition will move records to the appropriate partition and then return k, the first position in the right partition. Note that the pivot value is initially 246 Chap. 7 Internal Sorting template void qsort(E A[], int i, int j) { // Quicksort if (j <= i) return; // Don’t sort 0 or 1 element int pivotindex = findpivot(A, i, j); swap(A, pivotindex, j); // Put pivot at end // k will be the first position in the right subarray int k = partition(A, i-1, j, A[j]); swap(A, k, j); // Put pivot in place qsort(A, i, k-1); qsort(A, k+1, j); } Figure 7.11 Implementation for Quicksort. placed at the end of the array (position j). Thus, partition must not affect the value of array position j. After partitioning, the pivot value is placed in position k, which is its correct position in the final, sorted array. By doing so, we guarantee that at least one value (the pivot) will not be processed in the recursive calls to qsort. Even if a bad pivot is selected, yielding a completely empty partition to one side of the pivot, the larger partition will contain at most n 1 elements. Selecting a pivot can be done in many ways. The simplest is to use the first key. However, if the input is sorted or reverse sorted, this will produce a poor partitioning with all values to one side of the pivot. It is better to pick a value at random, thereby reducing the chance of a bad input order affecting the sort. Unfortunately, using a random number generator is relatively expensive, and we can do nearly as well by selecting the middle position in the array. Here is a simple findpivot function: template inline int findpivot(E A[], int i, int j) { return (i+j)/2; } We now turn to function partition. If we knew in advance how many keys are less than the pivot, partition could simply copy elements with key values less than the pivot to the low end of the array, and elements with larger keys to the high end. Because we do not know in advance how many keys are less than the pivot, we use a clever algorithm that moves indices inwards from the ends of the subarray, swapping values as necessary until the two indices meet. Figure 7.12 shows a C++ implementation for the partition step. Figure 7.13 illustrates partition. Initially, variables l and r are immedi- ately outside the actual bounds of the subarray being partitioned. Each pass through the outer do loop moves the counters l and r inwards, until eventually they meet. Note that at each iteration of the inner while loops, the bounds are moved prior to checking against the pivot value. This ensures that progress is made by each while loop, even when the two values swapped on the last iteration of the do loop were equal to the pivot. Also note the check that r>lin the second while Sec. 7.5 Quicksort 247 template inline int partition(E A[], int l, int r, E& pivot) { do { // Move the bounds inward until they meet while (Comp::prior(A[++l], pivot)); // Move l right and while ((l < r) && Comp::prior(pivot, A[--r])); // r left swap(A, l, r); // Swap out-of-place values } while (l < r); // Stop when they cross return l; // Return first position in right partition } Figure 7.12 The Quicksort partition implementation. Pass 1 Swap 1 Pass 2 Swap 2 Pass 3 72 6 57 88 85 42 83 73 48 60 l r 72 6 57 88 85 42 83 73 48 60 48 6 57 88 85 42 83 73 72 60 r 48 6 57 88 85 42 83 73 72 60 l 48 6 57 42 85 88 83 73 72 60 rl 48 6 57 42 88 83 73 72 60 Initial l l r r 85 l,r Figure 7.13 The Quicksort partition step. The first row shows the initial po- sitions for a collection of ten key values. The pivot value is 60, which has been swapped to the end of the array. The do loop makes three iterations, each time moving counters l and r inwards until they meet in the third pass. In the end, the left partition contains four values and the right partition contains six values. Function qsort will place the pivot value into position 4. loop. This ensures that r does not run off the low end of the partition in the case where the pivot is the least value in that partition. Function partition returns the first index of the right partition so that the subarray bound for the recursive calls to qsort can be determined. Figure 7.14 illustrates the complete Quicksort algorithm. To analyze Quicksort, we first analyze the findpivot and partition functions operating on a subarray of length k. Clearly, findpivot takes con- stant time. Function partition contains a do loop with two nested while loops. The total cost of the partition operation is constrained by how far l and r can move inwards. In particular, these two bounds variables together can move a 248 Chap. 7 Internal Sorting Pivot = 6 Pivot = 73 Pivot = 57 Final Sorted Array Pivot = 60 Pivot = 88 42 57 48 57 6 42 48 57 60 72 73 83 85 88 Pivot = 42 Pivot = 85 6 57 88 60 42 83 73 48 85 8572738388604257648 6 4842 42 48 85 83 88 8583 72 73 85 88 83 72 Figure 7.14 An illustration of Quicksort. total of s steps for a subarray of length s. However, this does not directly tell us how much work is done by the nested while loops. The do loop as a whole is guaranteed to move both l and r inward at least one position on each first pass. Each while loop moves its variable at least once (except in the special case where r is at the left edge of the array, but this can happen only once). Thus, we see that the do loop can be executed at most s times, the total amount of work done moving l and r is s, and each while loop can fail its test at most s times. The total work for the entire partition function is therefore ⇥(s). Knowing the cost of findpivot and partition, we can determine the cost of Quicksort. We begin with a worst-case analysis. The worst case will occur when the pivot does a poor job of breaking the array, that is, when there are no elements in one partition, and n 1 elements in the other. In this case, the divide and conquer strategy has done a poor job of dividing, so the conquer phase will work on a subproblem only one less than the size of the original problem. If this happens at each partition step, then the total cost of the algorithm will be n Xk=1 k =⇥(n2). In the worst case, Quicksort is ⇥(n2). This is terrible, no better than Bubble Sort.2 When will this worst case occur? Only when each pivot yields a bad parti- tioning of the array. If the pivot values are selected at random, then this is extremely unlikely to happen. When selecting the middle position of the current subarray, it 2The worst insult that I can think of for a sorting algorithm. Sec. 7.5 Quicksort 249 is still unlikely to happen. It does not take many good partitionings for Quicksort to work fairly well. Quicksort’s best case occurs when findpivot always breaks the array into two equal halves. Quicksort repeatedly splits the array into smaller partitions, as shown in Figure 7.14. In the best case, the result will be log n levels of partitions, with the top level having one array of size n, the second level two arrays of size n/2, the next with four arrays of size n/4, and so on. Thus, at each level, all partition steps for that level do a total of n work, for an overall cost of n log n work when Quicksort finds perfect pivots. Quicksort’s average-case behavior falls somewhere between the extremes of worst and best case. Average-case analysis considers the cost for all possible ar- rangements of input, summing the costs and dividing by the number of cases. We make one reasonable simplifying assumption: At each partition step, the pivot is equally likely to end in any position in the (sorted) array. In other words, the pivot is equally likely to break an array into partitions of sizes 0 and n1, or 1 and n2, and so on. Given this assumption, the average-case cost is computed from the following equation: T(n)=cn + 1 n n1 Xk=0 [T(k)+T(n 1 k)], T(0) = T(1) = c. This equation is in the form of a recurrence relation. Recurrence relations are discussed in Chapters 2 and 14, and this one is solved in Section 14.2.4. This equation says that there is one chance in n that the pivot breaks the array into subarrays of size 0 and n 1, one chance in n that the pivot breaks the array into subarrays of size 1 and n 2, and so on. The expression “T(k)+T(n 1 k)” is the cost for the two recursive calls to Quicksort on two arrays of size k and n1k. The initial cn term is the cost of doing the findpivot and partition steps, for some constant c. The closed-form solution to this recurrence relation is ⇥(n log n). Thus, Quicksort has average-case cost ⇥(n log n). This is an unusual situation that the average case cost and the worst case cost have asymptotically different growth rates. Consider what “average case” actually means. We compute an average cost for inputs of size n by summing up for every possible input of size n the product of the running time cost of that input times the probability that that input will occur. To simplify things, we assumed that every permutation is equally likely to occur. Thus, finding the average means summing up the cost for every permutation and dividing by the number of inputs (n!). We know that some of these n! inputs cost O(n2). But the sum of all the permutation costs has to be (n!)(O(n log n)). Given the extremely high cost of the worst inputs, there must be very few of them. In fact, there cannot be a constant fraction of the inputs with cost O(n2). Even, say, 1% of the inputs with cost O(n2) would lead to 250 Chap. 7 Internal Sorting an average cost of O(n2). Thus, as n grows, the fraction of inputs with high cost must be going toward a limit of zero. We can conclude that Quicksort will have good behavior if we can avoid those very few bad input permutations. The running time for Quicksort can be improved (by a constant factor), and much study has gone into optimizing this algorithm. The most obvious place for improvement is the findpivot function. Quicksort’s worst case arises when the pivot does a poor job of splitting the array into equal size subarrays. If we are willing to do more work searching for a better pivot, the effects of a bad pivot can be decreased or even eliminated. One good choice is to use the “median of three” algorithm, which uses as a pivot the middle of three randomly selected values. Using a random number generator to choose the positions is relatively expensive, so a common compromise is to look at the first, middle, and last positions of the current subarray. However, our simple findpivot function that takes the middle value as its pivot has the virtue of making it highly unlikely to get a bad input by chance, and it is quite cheap to implement. This is in sharp contrast to selecting the first or last element as the pivot, which would yield bad performance for many permutations that are nearly sorted or nearly reverse sorted. A significant improvement can be gained by recognizing that Quicksort is rel- atively slow when n is small. This might not seem to be relevant if most of the time we sort large arrays, nor should it matter how long Quicksort takes in the rare instance when a small array is sorted because it will be fast anyway. But you should notice that Quicksort itself sorts many, many small arrays! This happens as a natural by-product of the divide and conquer approach. A simple improvement might then be to replace Quicksort with a faster sort for small numbers, say Insertion Sort or Selection Sort. However, there is an even better — and still simpler — optimization. When Quicksort partitions are below a certain size, do nothing! The values within that partition will be out of order. However, we do know that all values in the array to the left of the partition are smaller than all values in the partition. All values in the array to the right of the partition are greater than all values in the partition. Thus, even if Quicksort only gets the values to “nearly” the right locations, the array will be close to sorted. This is an ideal situation in which to take advantage of the best-case performance of Insertion Sort. The final step is a single call to Insertion Sort to process the entire array, putting the elements into final sorted order. Empirical testing shows that the subarrays should be left unordered whenever they get down to nine or fewer elements. The last speedup to be considered reduces the cost of making recursive calls. Quicksort is inherently recursive, because each Quicksort operation must sort two sublists. Thus, there is no simple way to turn Quicksort into an iterative algorithm. However, Quicksort can be implemented using a stack to imitate recursion, as the amount of information that must be stored is small. We need not store copies of a Sec. 7.6 Heapsort 251 subarray, only the subarray bounds. Furthermore, the stack depth can be kept small if care is taken on the order in which Quicksort’s recursive calls are executed. We can also place the code for findpivot and partition inline to eliminate the remaining function calls. Note however that by not processing sublists of size nine or less as suggested above, about three quarters of the function calls will already have been eliminated. Thus, eliminating the remaining function calls will yield only a modest speedup. 7.6 Heapsort Our discussion of Quicksort began by considering the practicality of using a binary search tree for sorting. The BST requires more space than the other sorting meth- ods and will be slower than Quicksort or Mergesort due to the relative expense of inserting values into the tree. There is also the possibility that the BST might be un- balanced, leading to a ⇥(n2) worst-case running time. Subtree balance in the BST is closely related to Quicksort’s partition step. Quicksort’s pivot serves roughly the same purpose as the BST root value in that the left partition (subtree) stores val- ues less than the pivot (root) value, while the right partition (subtree) stores values greater than or equal to the pivot (root). A good sorting algorithm can be devised based on a tree structure more suited to the purpose. In particular, we would like the tree to be balanced, space efficient, and fast. The algorithm should take advantage of the fact that sorting is a special- purpose application in that all of the values to be stored are available at the start. This means that we do not necessarily need to insert one value at a time into the tree structure. Heapsort is based on the heap data structure presented in Section 5.5. Heapsort has all of the advantages just listed. The complete binary tree is balanced, its array representation is space efficient, and we can load all values into the tree at once, taking advantage of the efficient buildheap function. The asymptotic perfor- mance of Heapsort is ⇥(n log n) in the best, average, and worst cases. It is not as fast as Quicksort in the average case (by a constant factor), but Heapsort has special properties that will make it particularly useful when sorting data sets too large to fit in main memory, as discussed in Chapter 8. A sorting algorithm based on max-heaps is quite straightforward. First we use the heap building algorithm of Section 5.5 to convert the array into max-heap order. Then we repeatedly remove the maximum value from the heap, restoring the heap property each time that we do so, until the heap is empty. Note that each time we remove the maximum element from the heap, it is placed at the end of the array. Assume the n elements are stored in array positions 0 through n 1. After removing the maximum value from the heap and readjusting, the maximum value will now be placed in position n 1 of the array. The heap is now considered to be 252 Chap. 7 Internal Sorting of size n 1. Removing the new maximum (root) value places the second largest value in position n2 of the array. After removing each of the remaining values in turn, the array will be properly sorted from least to greatest. This is why Heapsort uses a max-heap rather than a min-heap as might have been expected. Figure 7.15 illustrates Heapsort. The complete C++ implementation is as follows: template void heapsort(E A[], int n) { // Heapsort E maxval; heap H(A, n, n); // Build the heap for (int i=0; i void binsort(E A[], int n) { List B[MaxKeyValue]; E item; for (int i=0; i void radix(E A[], E B[], int n, int k, int r, int cnt[]) { // cnt[i] stores number of records in bin[i] int j; for (int i=0, rtoi=1; i=0; j--) B[--cnt[(getKey::key(A[j])/rtoi)%r]] = A[j]; for (j=0; ji; j--) Consider the effect of replacing this with the following statement: for (int j=n-1; j>0; j--) Would the new implementation work correctly? Would the change affect the asymptotic complexity of the algorithm? How would the change affect the running time of the algorithm? 7.4 When implementing Insertion Sort, a binary search could be used to locate the position within the first i 1 elements of the array into which element i should be inserted. How would this affect the number of comparisons re- quired? How would using such a binary search affect the asymptotic running time for Insertion Sort? 7.5 Figure 7.5 shows the best-case number of swaps for Selection Sort as ⇥(n). This is because the algorithm does not check to see if the ith record is already in the ith position; that is, it might perform unnecessary swaps. (a) Modify the algorithm so that it does not make unnecessary swaps. (b) What is your prediction regarding whether this modification actually improves the running time? (c) Write two programs to compare the actual running times of the origi- nal Selection Sort and the modified algorithm. Which one is actually faster? 7.6 Recall that a sorting algorithm is said to be stable if the original ordering for duplicate keys is preserved. Of the sorting algorithms Insertion Sort, Bub- ble Sort, Selection Sort, Shellsort, Mergesort, Quicksort, Heapsort, Binsort, and Radix Sort, which of these are stable, and which are not? For each one, describe either why it is or is not stable. If a minor change to the implemen- tation would make it stable, describe the change. 7.7 Recall that a sorting algorithm is said to be stable if the original ordering for duplicate keys is preserved. We can make any algorithm stable if we alter the input keys so that (potentially) duplicate key values are made unique in a way that the first occurrence of the original duplicate value is less than the second occurrence, which in turn is less than the third, and so on. In the worst case, it is possible that all n input records have the same key value. Give an Sec. 7.11 Exercises 267 algorithm to modify the key values such that every modified key value is unique, the resulting key values give the same sort order as the original keys, the result is stable (in that the duplicate original key values remain in their original order), and the process of altering the keys is done in linear time using only a constant amount of additional space. 7.8 The discussion of Quicksort in Section 7.5 described using a stack instead of recursion to reduce the number of function calls made. (a) How deep can the stack get in the worst case? (b) Quicksort makes two recursive calls. The algorithm could be changed to make these two calls in a specific order. In what order should the two calls be made, and how does this affect how deep the stack can become? 7.9 Give a permutation for the values 0 through 7 that will cause Quicksort (as implemented in Section 7.5) to have its worst case behavior. 7.10 Assume L is an array, length(L) returns the number of records in the array, and qsort(L, i, j) sorts the records of L from i to j (leaving the records sorted in L) using the Quicksort algorithm. What is the average- case time complexity for each of the following code fragments? (a) for (i=0; i 1) { SPLITk(L, sub); // SPLITk places sublists into sub for (i=0; i L[dpne] we will step to the right by pn until we reach a value in L that is greater than K. We are now within pn positions of K. Assume (for now) that it takes a constant number of comparisons to bracket K within a sublist of size pn. We then take this sublist and repeat the process recursively. That is, at the next level we compute an interpolation to start somewhere in the subarray. We then step to the left or right (as appropriate) by steps of size p pn. What is the cost for QBS? Note that pcn = cn/2, and we will be repeatedly taking square roots of the current sublist size until we find the item that we are looking for. Because n =2log n and we can cut log n in half only log log n times, the cost is ⇥(log log n) if the number of probes on jump search is constant. Say that the number of comparisons needed is i, in which case the cost is i (since we have to do i comparisons). If Pi is the probability of needing exactly i probes, then pn Xi=1 iP(need exactly i probes) =1P1 +2P2 +3P3 + ···+ pnPpn 316 Chap. 9 Searching We now show that this is the same as pn Xi=1 P(need at least i probes) =1+(1 P1)+(1 P1 P2)+···+ Ppn =(P1 + ... + Ppn)+(P2 + ... + Ppn)+ (P3 + ... + Ppn)+··· =1P1 +2P2 +3P3 + ···+ pnPpn We require at least two probes to set the bounds, so the cost is 2+ pn Xi=3 P(need at least i probes). We now make take advantage of a useful fact known as ˇCebyˇsev’s Inequality. ˇCebyˇsev’s inequality states that P(need exactly i probes), or Pi, is Pi  p(1 p)n (i 2)2n  1 4(i 2)2 because p(1 p)  1/4 for any probability p. This assumes uniformly distributed data. Thus, the expected number of probes is 2+ pn Xi=3 1 4(i 2)2 < 2+1 4 1 Xi=1 1 i2 =2+1 4 ⇡ 6 ⇡ 2.4112 Is QBS better than binary search? Theoretically yes, because O(log log n) grows slower than O(log n). However, we have a situation here which illustrates the limits to the model of asymptotic complexity in some practical situations. Yes, c1 log n does grow faster than c2 log log n. In fact, it is exponentially faster! But even so, for practical input sizes, the absolute cost difference is fairly small. Thus, the constant factors might play a role. First we compare lg lg n to lg n. Factor n lg n lg lg n Di↵erence 16 4 2 2 256 8 3 2.7 216 16 4 4 232 32 5 6.4 Sec. 9.2 Self-Organizing Lists 317 It is not always practical to reduce an algorithm’s growth rate. There is a “prac- ticality window” for every problem, in that we have a practical limit to how big an input we wish to solve for. If our problem size never grows too big, it might not matter if we can reduce the cost by an extra log factor, because the constant factors in the two algorithms might differ by more than the log of the log of the input size. For our two algorithms, let us look further and check the actual number of comparisons used. For binary search, we need about log n 1 total comparisons. Quadratic binary search requires about 2.4lglgn comparisons. If we incorporate this observation into our table, we get a different picture about the relative differ- ences. Factor n lg n 12.4lglgn Di↵erence 16 3 4.8 worse 256 7 7.2 ⇡ same 64K 15 9.61.6 232 31 12 2.6 But we still are not done. This is only a count of raw comparisons. Bi- nary search is inherently much simpler than QBS, because binary search only needs to calculate the midpoint position of the array before each comparison, while quadratic binary search must calculate an interpolation point which is more expen- sive. So the constant factors for QBS are even higher. Not only are the constant factors worse on average, but QBS is far more depen- dent than binary search on good data distribution to perform well. For example, imagine that you are searching a telephone directory for the name “Young.” Nor- mally you would look near the back of the book. If you found a name beginning with ‘Z,’ you might look just a little ways toward the front. If the next name you find also begins with ’Z,‘ you would look a little further toward the front. If this particular telephone directory were unusual in that half of the entries begin with ‘Z,’ then you would need to move toward the front many times, each time eliminating relatively few records from the search. In the extreme, the performance of interpo- lation search might not be much better than sequential search if the distribution of key values is badly calculated. While it turns out that QBS is not a practical algorithm, this is not a typical situation. Fortunately, algorithm growth rates are usually well behaved, so that as- ymptotic algorithm analysis nearly always gives us a practical indication for which of two algorithms is better. 9.2 Self-Organizing Lists While ordering of lists is most commonly done by key value, this is not the only viable option. Another approach to organizing lists to speed search is to order the 318 Chap. 9 Searching records by expected frequency of access. While the benefits might not be as great as when organized by key value, the cost to organize (at least approximately) by frequency of access can be much cheaper, and thus can speed up sequential search in some situations. Assume that we know, for each key ki, the probability pi that the record with key ki will be requested. Assume also that the list is ordered so that the most frequently requested record is first, then the next most frequently requested record, and so on. Search in the list will be done sequentially, beginning with the first position. Over the course of many searches, the expected number of comparisons required for one search is Cn =1p0 +2p1 + ... + npn1. In other words, the cost to access the record in L[0] is 1 (because one key value is looked at), and the probability of this occurring is p0. The cost to access the record in L[1] is 2 (because we must look at the first and the second records’ key values), with probability p1, and so on. For n records, assuming that all searches are for records that actually exist, the probabilities p0 through pn1 must sum to one. Certain probability distributions give easily computed results. Example 9.1 Calculate the expected cost to search a list when each record has equal chance of being accessed (the classic sequential search through an unsorted list). Setting pi =1/n yields Cn = n Xi=1 i/n =(n + 1)/2. This result matches our expectation that half the records will be accessed on average by normal sequential search. If the records truly have equal access probabilities, then ordering records by frequency yields no benefit. We saw in Section 9.1 the more general case where we must consider the probability (labeled pn) that the search key does not match that for any record in the array. In that case, in accordance with our general formula, we get (1pn)n +1 2 +pnn = n +1 npnn pn +2pn 2 = n +1+p0(n 1) 2 . Thus, n+1 2  Cn  n, depending on the value of p0. A geometric probability distribution can yield quite different results. Sec. 9.2 Self-Organizing Lists 319 Example 9.2 Calculate the expected cost for searching a list ordered by frequency when the probabilities are defined as pi = ⇢ 1/2i if 0  i  n 2 1/2n if i = n 1. Then, Cn ⇡ n1 Xi=0 (i + 1)/2i+1 = n Xi=1 (i/2i) ⇡ 2. For this example, the expected number of accesses is a constant. This is because the probability for accessing the first record is high (one half), the second is much lower (one quarter) but still much higher than for the third record, and so on. This shows that for some probability distributions, or- dering the list by frequency can yield an efficient search technique. In many search applications, real access patterns follow a rule of thumb called the 80/20 rule. The 80/20 rule says that 80% of the record accesses are to 20% of the records. The values of 80 and 20 are only estimates; every data access pat- tern has its own values. However, behavior of this nature occurs surprisingly often in practice (which explains the success of caching techniques widely used by web browsers for speeding access to web pages, and by disk drive and CPU manufac- turers for speeding access to data stored in slower memory; see the discussion on buffer pools in Section 8.3). When the 80/20 rule applies, we can expect consid- erable improvements to search performance from a list ordered by frequency of access over standard sequential search in an unordered list. Example 9.3 The 80/20 rule is an example of a Zipf distribution. Nat- urally occurring distributions often follow a Zipf distribution. Examples include the observed frequency for the use of words in a natural language such as English, and the size of the population for cities (i.e., view the relative proportions for the populations as equivalent to the “frequency of use”). Zipf distributions are related to the Harmonic Series defined in Equa- tion 2.10. Define the Zipf frequency for item i in the distribution for n records as 1/(iHn) (see Exercise 9.4). The expected cost for the series whose members follow this Zipf distribution will be Cn = n Xi=1 i/iHn = n/Hn ⇡ n/ loge n. When a frequency distribution follows the 80/20 rule, the average search looks at about 10-15% of the records in a list ordered by frequency. 320 Chap. 9 Searching This is potentially a useful observation that typical “real-life” distributions of record accesses, if the records were ordered by frequency, would require that we visit on average only 10-15% of the list when doing sequential search. This means that if we had an application that used sequential search, and we wanted to make it go a bit faster (by a constant amount), we could do so without a major rewrite to the system to implement something like a search tree. But that is only true if there is an easy way to (at least approximately) order the records by frequency. In most applications, we have no means of knowing in advance the frequencies of access for the data records. To complicate matters further, certain records might be accessed frequently for a brief period of time, and then rarely thereafter. Thus, the probability of access for records might change over time (in most database systems, this is to be expected). Self-organizing lists seek to solve both of these problems. Self-organizing lists modify the order of records within the list based on the actual pattern of record access. Self-organizing lists use a heuristic for deciding how to to reorder the list. These heuristics are similar to the rules for managing buffer pools (see Section 8.3). In fact, a buffer pool is a form of self-organizing list. Ordering the buffer pool by expected frequency of access is a good strategy, because typically we must search the contents of the buffers to determine if the desired information is already in main memory. When ordered by frequency of access, the buffer at the end of the list will be the one most appropriate for reuse when a new page of information must be read. Below are three traditional heuristics for managing self-organizing lists: 1. The most obvious way to keep a list ordered by frequency would be to store a count of accesses to each record and always maintain records in this or- der. This method will be referred to as count. Count is similar to the least frequently used buffer replacement strategy. Whenever a record is accessed, it might move toward the front of the list if its number of accesses becomes greater than a record preceding it. Thus, count will store the records in the order of frequency that has actually occurred so far. Besides requiring space for the access counts, count does not react well to changing frequency of access over time. Once a record has been accessed a large number of times under the frequency count system, it will remain near the front of the list regardless of further access history. 2. Bring a record to the front of the list when it is found, pushing all the other records back one position. This is analogous to the least recently used buffer replacement strategy and is called move-to-front. This heuristic is easy to implement if the records are stored using a linked list. When records are stored in an array, bringing a record forward from near the end of the array will result in a large number of records (slightly) changing position. Move- to-front’s cost is bounded in the sense that it requires at most twice the num- Sec. 9.2 Self-Organizing Lists 321 ber of accesses required by the optimal static ordering for n records when at least n searches are performed. In other words, if we had known the se- ries of (at least n) searches in advance and had stored the records in order of frequency so as to minimize the total cost for these accesses, this cost would be at least half the cost required by the move-to-front heuristic. (This will be proved using amortized analysis in Section 14.3.) Finally, move-to-front responds well to local changes in frequency of access, in that if a record is frequently accessed for a brief period of time it will be near the front of the list during that period of access. Move-to-front does poorly when the records are processed in sequential order, especially if that sequential order is then repeated multiple times. 3. Swap any record found with the record immediately preceding it in the list. This heuristic is called transpose. Transpose is good for list implementations based on either linked lists or arrays. Frequently used records will, over time, move to the front of the list. Records that were once frequently accessed but are no longer used will slowly drift toward the back. Thus, it appears to have good properties with respect to changing frequency of access. Unfortunately, there are some pathological sequences of access that can make transpose perform poorly. Consider the case where the last record of the list (call it X) is accessed. This record is then swapped with the next-to-last record (call it Y), making Y the last record. If Y is now accessed, it swaps with X. A repeated series of accesses alternating between X and Y will continually search to the end of the list, because neither record will ever make progress toward the front. However, such pathological cases are unusual in practice. A variation on transpose would be to move the accessed record forward in the list by some fixed number of steps. Example 9.4 Assume that we have eight records, with key values A to H, and that they are initially placed in alphabetical order. Now, consider the result of applying the following access pattern: FDFGEGFADFGE. Assume that when a record’s frequency count goes up, it moves forward in the list to become the last record with that value for its frequency count. After the first two accesses, F will be the first record and D will be the second. The final list resulting from these accesses will be FGDEABCH, and the total cost for the twelve accesses will be 45 comparisons. If the list is organized by the move-to-front heuristic, then the final list will be EGFDABCH, 322 Chap. 9 Searching and the total number of comparisons required is 54. Finally, if the list is organized by the transpose heuristic, then the final list will be ABF DGECH, and the total number of comparisons required is 62. While self-organizing lists do not generally perform as well as search trees or a sorted list, both of which require O(log n) search time, there are many situations in which self-organizing lists prove a valuable tool. Obviously they have an advantage over sorted lists in that they need not be sorted. This means that the cost to insert a new record is low, which could more than make up for the higher search cost when insertions are frequent. Self-organizing lists are simpler to implement than search trees and are likely to be more efficient for small lists. Nor do they require additional space. Finally, in the case of an application where sequential search is “almost” fast enough, changing an unsorted list to a self-organizing list might speed the application enough at a minor cost in additional code. As an example of applying self-organizing lists, consider an algorithm for com- pressing and transmitting messages. The list is self-organized by the move-to-front rule. Transmission is in the form of words and numbers, by the following rules: 1. If the word has been seen before, transmit the current position of the word in the list. Move the word to the front of the list. 2. If the word is seen for the first time, transmit the word. Place the word at the front of the list. Both the sender and the receiver keep track of the position of words in the list in the same way (using the move-to-front rule), so they agree on the meaning of the numbers that encode repeated occurrences of words. Consider the following example message to be transmitted (for simplicity, ignore case in letters). The car on the left hit the car I left. The first three words have not been seen before, so they must be sent as full words. The fourth word is the second appearance of “the,” which at this point is the third word in the list. Thus, we only need to transmit the position value “3.” The next two words have not yet been seen, so must be sent as full words. The seventh word is the third appearance of “the,” which coincidentally is again in the third position. The eighth word is the second appearance of “car,” which is now in the fifth position of the list. “I” is a new word, and the last word “left” is now in the fifth position. Thus the entire transmission would be The car on 3 left hit 3 5 I 5. Sec. 9.3 Bit Vectors for Representing Sets 323 0 1 2 3 4 5 6 7 8 9 10 11 12 15 0000010100 11 101 13 14 0 Figure 9.1 The bit array for the set of primes in the range 0 to 15. The bit at position i is set to 1 if and only if i is prime. This approach to compression is similar in spirit to Ziv-Lempel coding, which is a class of coding algorithms commonly used in file compression utilities. Ziv- Lempel coding replaces repeated occurrences of strings with a pointer to the lo- cation in the file of the first occurrence of the string. The codes are stored in a self-organizing list in order to speed up the time required to search for a string that has previously been seen. 9.3 Bit Vectors for Representing Sets Determining whether a value is a member of a particular set is a special case of searching for keys in a sequence of records. Thus, any of the search methods discussed in this book can be used to check for set membership. However, we can also take advantage of the restricted circumstances imposed by this problem to develop another representation. In the case where the set values fall within a limited range, we can represent the set using a bit array with a bit position allocated for each potential member. Those members actually in the set store a value of 1 in their corresponding bit; those members not in the set store a value of 0 in their corresponding bit. For example, consider the set of primes between 0 and 15. Figure 9.1 shows the corresponding bit array. To determine if a particular value is prime, we simply check the corre- sponding bit. This representation scheme is called a bit vector or a bitmap. The mark array used in several of the graph algorithms of Chapter 11 is an example of such a set representation. If the set fits within a single computer word, then set union, intersection, and difference can be performed by logical bit-wise operations. The union of sets A and B is the bit-wise OR function (whose symbol is | in C++). The intersection of sets A and B is the bit-wise AND function (whose symbol is & in C++). For example, if we would like to compute the set of numbers between 0 and 15 that are both prime and odd numbers, we need only compute the expression 0011010100010100 & 0101010101010101. The set difference A B can be implemented in C++ using the expression A&˜B (˜ is the symbol for bit-wise negation). For larger sets that do not fit into a single computer word, the equivalent operations can be performed in turn on the series of words making up the entire bit vector. 324 Chap. 9 Searching This method of computing sets from bit vectors is sometimes applied to doc- ument retrieval. Consider the problem of picking from a collection of documents those few which contain selected keywords. For each keyword, the document re- trieval system stores a bit vector with one bit for each document. If the user wants to know which documents contain a certain three keywords, the corresponding three bit vectors are AND’ed together. Those bit positions resulting in a value of 1 cor- respond to the desired documents. Alternatively, a bit vector can be stored for each document to indicate those keywords appearing in the document. Such an organiza- tion is called a signature file. The signatures can be manipulated to find documents with desired combinations of keywords. 9.4 Hashing This section presents a completely different approach to searching arrays: by direct access based on key value. The process of finding a record using some computa- tion to map its key value to a position in the array is called hashing. Most hash- ing schemes place records in the array in whatever order satisfies the needs of the address calculation, thus the records are not ordered by value or frequency. The function that maps key values to positions is called a hash function and will be denoted by h. The array that holds the records is called the hash table and will be denoted by HT. A position in the hash table is also known as a slot. The number of slots in hash table HT will be denoted by the variable M, with slots numbered from 0 to M 1. The goal for a hashing system is to arrange things such that, for any key value K and some hash function h, i = h(K) is a slot in the table such that 0  h(K) void hashdict:: hashInsert(const Key& k, const E& e) { int home; // Home position for e int pos = home = h(k); // Init probe sequence for (int i=1; EMPTYKEY != (HT[pos]).key(); i++) { pos = (home + p(k, i)) % M; // probe Assert(k != (HT[pos]).key(), "Duplicates not allowed"); } KVpair temp(k, e); HT[pos] = temp; } Figure 9.6 Insertion method for a dictionary implemented by a hash table. potentially hold the record. The first slot in the sequence will be the home position for the key. If the home position is occupied, then the collision resolution policy goes to the next slot in the sequence. If this is occupied as well, then another slot must be found, and so on. This sequence of slots is known as the probe sequence, and it is generated by some probe function that we will call p. The insert function is shown in Figure 9.6. Method hashInsert first checks to see if the home slot for the key is empty. If the home slot is occupied, then we use the probe function, p(k, i) to locate a free slot in the table. Function p has two parameters, the key k and a count i for where in the probe sequence we wish to be. That is, to get the first position in the probe sequence after the home slot for key K, we call p(K, 1). For the next slot in the probe sequence, call p(K, 2). Note that the probe function returns an offset from the original home position, rather than a slot in the hash table. Thus, the for loop in hashInsert is computing positions in the table at each iteration by adding the value returned from the probe function to the home position. The ith call to p returns the ith offset to be used. Searching in a hash table follows the same probe sequence that was followed when inserting records. In this way, a record not in its home position can be recov- ered. A C++ implementation for the search procedure is shown in Figure 9.7. The insert and search routines assume that at least one slot on the probe se- quence of every key will be empty. Otherwise, they will continue in an infinite loop on unsuccessful searches. Thus, the dictionary should keep a count of the number of records stored, and refuse to insert into a table that has only one free slot. The discussion on bucket hashing presented a simple method of collision reso- lution. If the home position for the record is occupied, then move down the bucket until a free slot is found. This is an example of a technique for collision resolution known as linear probing. The probe function for simple linear probing is p(K, i)=i. Sec. 9.4 Hashing 335 // Search for the record with Key K template E hashdict:: hashSearch(const Key& k) const { int home; // Home position for k int pos = home = h(k); // Initial position is home slot for (int i = 1; (k != (HT[pos]).key()) && (EMPTYKEY != (HT[pos]).key()); i++) pos = (home + p(k, i)) % M; // Next on probe sequence if (k == (HT[pos]).key()) // Found it return (HT[pos]).value(); else return NULL; // k not in hash table } Figure 9.7 Search method for a dictionary implemented by a hash table. That is, the ith offset on the probe sequence is just i, meaning that the ith step is simply to move down i slots in the table. Once the bottom of the table is reached, the probe sequence wraps around to the beginning of the table. Linear probing has the virtue that all slots in the table will be candidates for inserting a new record before the probe sequence returns to the home position. While linear probing is probably the first idea that comes to mind when consid- ering collision resolution policies, it is not the only one possible. Probe function p allows us many options for how to do collision resolution. In fact, linear probing is one of the worst collision resolution methods. The main problem is illustrated by Figure 9.8. Here, we see a hash table of ten slots used to store four-digit numbers, with hash function h(K)=K mod 10. In Figure 9.8(a), five numbers have been placed in the table, leaving five slots remaining. The ideal behavior for a collision resolution mechanism is that each empty slot in the table will have equal probability of receiving the next record inserted (assum- ing that every slot in the table has equal probability of being hashed to initially). In this example, assume that the hash function gives each slot (roughly) equal proba- bility of being the home position for the next key. However, consider what happens to the next record if its key has its home position at slot 0. Linear probing will send the record to slot 2. The same will happen to records whose home position is at slot 1. A record with home position at slot 2 will remain in slot 2. Thus, the probability is 3/10 that the next record inserted will end up in slot 2. In a similar manner, records hashing to slots 7 or 8 will end up in slot 9. However, only records hashing to slot 3 will be stored in slot 3, yielding one chance in ten of this happen- ing. Likewise, there is only one chance in ten that the next record will be stored in slot 4, one chance in ten for slot 5, and one chance in ten for slot 6. Thus, the resulting probabilities are not equal. To make matters worse, if the next record ends up in slot 9 (which already has a higher than normal chance of happening), then the following record will end up 336 Chap. 9 Searching 0 1 2 4 3 5 6 7 9 0 1 2 3 4 5 6 7 8 9 8 9050 1001 9877 9050 1001 9877 2037 1059 2037 (a) (b) Figure 9.8 Example of problems with linear probing. (a) Four values are inserted in the order 1001, 9050, 9877, and 2037 using hash function h(K)=K mod 10. (b) The value 1059 is added to the hash table. in slot 2 with probability 6/10. This is illustrated by Figure 9.8(b). This tendency of linear probing to cluster items together is known as primary clustering. Small clusters tend to merge into big clusters, making the problem worse. The objection to primary clustering is that it leads to long probe sequences. Improved Collision Resolution Methods How can we avoid primary clustering? One possible improvement might be to use linear probing, but to skip slots by a constant c other than 1. This would make the probe function p(K, i)=ci, and so the ith slot in the probe sequence will be (h(K)+ic)modM. In this way, records with adjacent home positions will not follow the same probe sequence. For example, if we were to skip by twos, then our offsets from the home slot would be 2, then 4, then 6, and so on. One quality of a good probe sequence is that it will cycle through all slots in the hash table before returning to the home position. Clearly linear probing (which “skips” slots by one each time) does this. Unfortunately, not all values for c will make this happen. For example, if c =2and the table contains an even number of slots, then any key whose home position is in an even slot will have a probe sequence that cycles through only the even slots. Likewise, the probe sequence for a key whose home position is in an odd slot will cycle through the odd slots. Thus, this combination of table size and linear probing constant effectively divides Sec. 9.4 Hashing 337 the records into two sets stored in two disjoint sections of the hash table. So long as both sections of the table contain the same number of records, this is not really important. However, just from chance it is likely that one section will become fuller than the other, leading to more collisions and poorer performance for those records. The other section would have fewer records, and thus better performance. But the overall system performance will be degraded, as the additional cost to the side that is more full outweighs the improved performance of the less-full side. Constant c must be relatively prime to M to generate a linear probing sequence that visits all slots in the table (that is, c and M must share no factors). For a hash table of size M = 10, if c is any one of 1, 3, 7, or 9, then the probe sequence will visit all slots for any key. When M = 11, any value for c between 1 and 10 generates a probe sequence that visits all slots for every key. Consider the situation where c =2and we wish to insert a record with key k1 such that h(k1)=3. The probe sequence for k1 is 3, 5, 7, 9, and so on. If another key k2 has home position at slot 5, then its probe sequence will be 5, 7, 9, and so on. The probe sequences of k1 and k2 are linked together in a manner that contributes to clustering. In other words, linear probing with a value of c>1 does not solve the problem of primary clustering. We would like to find a probe function that does not link keys together in this way. We would prefer that the probe sequence for k1 after the first step on the sequence should not be identical to the probe sequence of k2. Instead, their probe sequences should diverge. The ideal probe function would select the next position on the probe sequence at random from among the unvisited slots; that is, the probe sequence should be a random permutation of the hash table positions. Unfortunately, we cannot actually select the next position in the probe sequence at random, because then we would not be able to duplicate this same probe sequence when searching for the key. However, we can do something similar called pseudo-random probing. In pseudo-random probing, the ith slot in the probe sequence is (h(K)+ri)modM where ri is the ith value in a random permutation of the numbers from 1 to M 1. All insertion and search operations use the same random permutation. The probe function is p(K, i)=Perm[i 1], where Perm is an array of length M 1 containing a random permutation of the values from 1 to M 1. Example 9.9 Consider a table of size M = 101, with Perm[1] = 5, Perm[2] = 2, and Perm[3] = 32. Assume that we have two keys k1 and k2 where h(k1) = 30 and h(k2) = 35. The probe sequence for k1 is 30, then 35, then 32, then 62. The probe sequence for k2 is 35, then 40, then 37, then 67. Thus, while k2 will probe to k1’s home position as its second choice, the two keys’ probe sequences diverge immediately thereafter. 338 Chap. 9 Searching Another probe function that eliminates primary clustering is called quadratic probing. Here the probe function is some quadratic function p(K, i)=c1i2 + c2i + c3 for some choice of constants c1, c2, and c3. The simplest variation is p(K, i)=i2 (i.e., c1 =1, c2 =0, and c3 =0. Then the ith value in the probe sequence would be (h(K)+i2)modM. Under quadratic probing, two keys with different home positions will have diverging probe sequences. Example 9.10 Given a hash table of size M = 101, assume for keys k1 and k2 that h(k1) = 30 and h(k2) = 29. The probe sequence for k1 is 30, then 31, then 34, then 39. The probe sequence for k2 is 29, then 30, then 33, then 38. Thus, while k2 will probe to k1’s home position as its second choice, the two keys’ probe sequences diverge immediately thereafter. Unfortunately, quadratic probing has the disadvantage that typically not all hash table slots will be on the probe sequence. Using p(K, i)=i2 gives particularly in- consistent results. For many hash table sizes, this probe function will cycle through a relatively small number of slots. If all slots on that cycle happen to be full, then the record cannot be inserted at all! For example, if our hash table has three slots, then records that hash to slot 0 can probe only to slots 0 and 1 (that is, the probe sequence will never visit slot 2 in the table). Thus, if slots 0 and 1 are full, then the record cannot be inserted even though the table is not full. A more realistic example is a table with 105 slots. The probe sequence starting from any given slot will only visit 23 other slots in the table. If all 24 of these slots should happen to be full, even if other slots in the table are empty, then the record cannot be inserted because the probe sequence will continually hit only those same 24 slots. Fortunately, it is possible to get good results from quadratic probing at low cost. The right combination of probe function and table size will visit many slots in the table. In particular, if the hash table size is a prime number and the probe function is p(K, i)=i2, then at least half the slots in the table will be visited. Thus, if the table is less than half full, we can be certain that a free slot will be found. Alternatively, if the hash table size is a power of two and the probe function is p(K, i)=(i2 + i)/2, then every slot in the table will be visited by the probe function. Both pseudo-random probing and quadratic probing eliminate primary cluster- ing, which is the problem of keys sharing substantial segments of a probe sequence. If two keys hash to the same home position, however, then they will always follow the same probe sequence for every collision resolution method that we have seen so far. The probe sequences generated by pseudo-random and quadratic probing (for example) are entirely a function of the home position, not the original key value. Sec. 9.4 Hashing 339 This is because function p ignores its input parameter K for these collision resolu- tion methods. If the hash function generates a cluster at a particular home position, then the cluster remains under pseudo-random and quadratic probing. This problem is called secondary clustering. To avoid secondary clustering, we need to have the probe sequence make use of the original key value in its decision-making process. A simple technique for doing this is to return to linear probing by a constant step size for the probe function, but to have that constant be determined by a second hash function, h2. Thus, the probe sequence would be of the form p(K, i)=i ⇤ h2(K). This method is called double hashing. Example 9.11 Assume a hash table has size M = 101, and that there are three keys k1, k2, and k3 with h(k1) = 30, h(k2) = 28, h(k3) = 30, h2(k1)=2, h2(k2)=5, and h2(k3)=5. Then, the probe sequence for k1 will be 30, 32, 34, 36, and so on. The probe sequence for k2 will be 28, 33, 38, 43, and so on. The probe sequence for k3 will be 30, 35, 40, 45, and so on. Thus, none of the keys share substantial portions of the same probe sequence. Of course, if a fourth key k4 has h(k4) = 28 and h2(k4)=2, then it will follow the same probe sequence as k1. Pseudo- random or quadratic probing can be combined with double hashing to solve this problem. A good implementation of double hashing should ensure that all of the probe sequence constants are relatively prime to the table size M. This can be achieved easily. One way is to select M to be a prime number, and have h2 return a value in the range 1  h2(K)  M 1. Another way is to set M =2m for some value m and have h2 return an odd value between 1 and 2m. Figure 9.9 shows an implementation of the dictionary ADT by means of a hash table. The simplest hash function is used, with collision resolution by linear prob- ing, as the basis for the structure of a hash table implementation. A suggested project at the end of this chapter asks you to improve the implementation with other hash functions and collision resolution policies. 9.4.4 Analysis of Closed Hashing How efficient is hashing? We can measure hashing performance in terms of the number of record accesses required when performing an operation. The primary operations of concern are insertion, deletion, and search. It is useful to distinguish between successful and unsuccessful searches. Before a record can be deleted, it must be found. Thus, the number of accesses required to delete a record is equiv- alent to the number required to successfully search for it. To insert a record, an empty slot along the record’s probe sequence must be found. This is equivalent to 340 Chap. 9 Searching // Dictionary implemented with a hash table template class hashdict : public Dictionary { private: KVpair* HT; // The hash table int M; // Size of HT int currcnt; // The current number of elements in HT Key EMPTYKEY; // User-supplied key value for an empty slot int p(Key K, int i) const // Probe using linear probing { return i; } int h(int x) const { return x % M; } // Poor hash function int h(char* x) const { // Hash function for character keys int i, sum; for (sum=0, i=0; x[i] != ’\0’; i++) sum += (int) x[i]; return sum % M; } void hashInsert(const Key&, const E&); E hashSearch(const Key&) const; public: hashdict(int sz, Key k){ // "k" defines an empty slot M = sz; EMPTYKEY = k; currcnt = 0; HT = new KVpair[sz]; // Make HT of size sz for (int i=0; i class TTNode { // 2-3 tree node structure public: E lval; // The node’s left record Key lkey; // Left record’s key E rval; // The node’s right record Key rkey; // Right record’s key TTNode* left; // Pointer to left child TTNode* center; // Pointer to middle child TTNode* right; // Pointer to right child TTNode() { center = left = right = NULL; lkey = rkey = EMPTYKEY; } TTNode(Key lk, E lv, Key rk, E rv, TTNode* p1, TTNode* p2, TTNode* p3) { lkey = lk; rkey = rk; lval = lv; rval = rv; left = p1; center = p2; right = p3; } ˜TTNode() { } bool isLeaf() { return left == NULL; } TTNode* add(TTNode* it); }; Figure 10.10 The 2-3 tree node implementation. techniques of Section 5.3.1 can be applied here to implement separate internal and leaf node types. From the defining rules for 2-3 trees we can derive relationships between the number of nodes in the tree and the depth of the tree. A 2-3 tree of height k has at least 2k1 leaves, because if every internal node has two children it degenerates to the shape of a complete binary tree. A 2-3 tree of height k has at most 3k1 leaves, because each internal node can have at most three children. Searching for a value in a 2-3 tree is similar to searching in a BST. Search begins at the root. If the root does not contain the search key K, then the search progresses to the only subtree that can possibly contain K. The value(s) stored in the root node determine which is the correct subtree. For example, if searching for the value 30 in the tree of Figure 10.9, we begin with the root node. Because 30 is between 18 and 33, it can only be in the middle subtree. Searching the middle child of the root node yields the desired record. If searching for 15, then the first step is again to search the root node. Because 15 is less than 18, the first (left) branch is taken. At the next level, we take the second branch to the leaf node containing 15. If the search key were 16, then upon encountering the leaf containing 15 we would find that the search key is not in the tree. Figure 10.11 is an implementation for the 2-3 tree search method. 362 Chap. 10 Indexing // Find the record that matches a given key value template E TTTree:: findhelp(TTNode* root, Key k) const { if (root == NULL) return NULL; // value not found if (k == root->lkey) return root->lval; if (k == root->rkey) return root->rval; if (k < root->lkey) // Go left return findhelp(root->left, k); else if (root->rkey == EMPTYKEY) // 2 child node return findhelp(root->center, k); // Go center else if (k < root->rkey) return findhelp(root->center, k); // Go center else return findhelp(root->right, k); // Go right } Figure 10.11 Implementation for the 2-3 tree search method. 12 10 20 21 33 23 30 24 31 50 18 45 48 47 5215 15 14 Figure 10.12 Simple insert into the 2-3 tree of Figure 10.9. The value 14 is inserted into the tree at the leaf node containing 15. Because there is room in the node for a second key, it is simply added to the left position with 15 moved to the right position. Insertion into a 2-3 tree is similar to insertion into a BST to the extent that the new record is placed in the appropriate leaf node. Unlike BST insertion, a new child is not created to hold the record being inserted, that is, the 2-3 tree does not grow downward. The first step is to find the leaf node that would contain the record if it were in the tree. If this leaf node contains only one value, then the new record can be added to that node with no further modification to the tree, as illustrated in Figure 10.12. In this example, a record with key value 14 is inserted. Searching from the root, we come to the leaf node that stores 15. We add 14 as the left value (pushing the record with key 15 to the rightmost position). If we insert the new record into a leaf node L that already contains two records, then more space must be created. Consider the two records of node L and the record to be inserted without further concern for which two were already in L and which is the new record. The first step is to split L into two nodes. Thus, a new node — call it L0 — must be created from free store. L receives the record with the least of the three key values. L0 receives the greatest of the three. The record Sec. 10.4 2-3 Trees 363 33 15 23 30 48 52 45 47 50 5510 12 18 20 21 24 31 Figure 10.13 A simple node-splitting insert for a 2-3 tree. The value 55 is added to the 2-3 tree of Figure 10.9. This makes the node containing values 50 and 52 split, promoting value 52 to the parent node. with the middle of the three key value is passed up to the parent node along with a pointer to L0. This is called a promotion. The promoted key is then inserted into the parent. If the parent currently contains only one record (and thus has only two children), then the promoted record and the pointer to L0 are simply added to the parent node. If the parent is full, then the split-and-promote process is repeated. Figure 10.13 illustrates a simple promotion. Figure 10.14 illustrates what happens when promotions require the root to split, adding a new level to the tree. In either case, all leaf nodes continue to have equal depth. Figures 10.15 and 10.16 present an implementation for the insertion process. Note that inserthelp of Figure 10.15 takes three parameters. The first is a pointer to the root of the current subtree, named rt. The second is the key for the record to be inserted, and the third is the record itself. The return value for inserthelp is a pointer to a 2-3 tree node. If rt is unchanged, then a pointer to rt is returned. If rt is changed (due to the insertion causing the node to split), then a pointer to the new subtree root is returned, with the key value and record value in the leftmost fields, and a pointer to the (single) subtree in the center pointer field. This revised node will then be added to the parent, as illustrated in Figure 10.14. When deleting a record from the 2-3 tree, there are three cases to consider. The simplest occurs when the record is to be removed from a leaf node containing two records. In this case, the record is simply removed, and no other nodes are affected. The second case occurs when the only record in a leaf node is to be removed. The third case occurs when a record is to be removed from an internal node. In both the second and the third cases, the deleted record is replaced with another that can take its place while maintaining the correct order, similar to removing a node from a BST. If the tree is sparse enough, there is no such record available that will allow all nodes to still maintain at least one record. In this situation, sibling nodes are merged together. The delete operation for the 2-3 tree is excessively complex and will not be described further. Instead, a complete discussion of deletion will be postponed until the next section, where it can be generalized for a particular variant of the B-tree. 364 Chap. 10 Indexing 23 20 (a) (b) (c) 3020 24 31 21 24 3121 1919 12 10 19 24 30 31 33 45 47 50 52 23 18 20 21 48 15 3023 3318 Figure 10.14 Example of inserting a record that causes the 2-3 tree root to split. (a) The value 19 is added to the 2-3 tree of Figure 10.9. This causes the node containing 20 and 21 to split, promoting 20. (b) This in turn causes the internal node containing 23 and 30 to split, promoting 23. (c) Finally, the root node splits, promoting 23 to become the left record in the new root. The result is that the tree becomes one level higher. The 2-3 tree insert and delete routines do not add new nodes at the bottom of the tree. Instead they cause leaf nodes to split or merge, possibly causing a ripple effect moving up the tree to the root. If necessary the root will split, causing a new root node to be created and making the tree one level deeper. On deletion, if the last two children of the root merge, then the root node is removed and the tree will lose a level. In either case, all leaf nodes are always at the same level. When all leaf nodes are at the same level, we say that a tree is height balanced. Because the 2-3 tree is height balanced, and every internal node has at least two children, we know that the maximum depth of the tree is log n. Thus, all 2-3 tree insert, find, and delete operations require ⇥(log n) time. 10.5 B-Trees This section presents the B-tree. B-trees are usually attributed to R. Bayer and E. McCreight who described the B-tree in a 1972 paper. By 1979, B-trees had re- Sec. 10.5 B-Trees 365 template TTNode* TTTree:: inserthelp(TTNode* rt, const Key k, const E e) { TTNode* retval; if (rt == NULL) // Empty tree: create a leaf node for root return new TTNode(k, e, EMPTYKEY, NULL, NULL, NULL, NULL); if (rt->isLeaf()) // At leaf node: insert here return rt->add(new TTNode(k, e, EMPTYKEY, NULL, NULL, NULL, NULL)); // Add to internal node if (k < rt->lkey) { retval = inserthelp(rt->left, k, e); if (retval == rt->left) return rt; else return rt->add(retval); } else if((rt->rkey == EMPTYKEY) || (k < rt->rkey)) { retval = inserthelp(rt->center, k, e); if (retval == rt->center) return rt; else return rt->add(retval); } else { // Insert right retval = inserthelp(rt->right, k, e); if (retval == rt->right) return rt; else return rt->add(retval); } } Figure 10.15 The 2-3 tree insert routine. placed virtually all large-file access methods other than hashing. B-trees, or some variant of B-trees, are the standard file organization for applications requiring inser- tion, deletion, and key range searches. They are used to implement most modern file systems. B-trees address effectively all of the major problems encountered when implementing disk-based search trees: 1. B-trees are always height balanced, with all leaf nodes at the same level. 2. Update and search operations affect only a few disk blocks. The fewer the number of disk blocks affected, the less disk I/O is required. 3. B-trees keep related records (that is, records with similar key values) on the same disk block, which helps to minimize disk I/O on searches due to locality of reference. 4. B-trees guarantee that every node in the tree will be full at least to a certain minimum percentage. This improves space efficiency while reducing the typical number of disk fetches necessary during a search or update operation. A B-tree of order m is defined to have the following shape properties: • The root is either a leaf or has at least two children. • Each internal node, except for the root, has between dm/2e and m children. 366 Chap. 10 Indexing // Add a new key/value pair to the node. There might be a // subtree associated with the record being added. This // information comes in the form of a 2-3 tree node with // one key and a (possibly NULL) subtree through the // center pointer field. template TTNode* TTNode::add(TTNode* it) { if (rkey == EMPTYKEY) { // Only one key, add here if (lkey < it->lkey) { rkey = it->lkey; rval = it->lval; right = center; center = it->center; } else { rkey = lkey; rval = lval; right = center; lkey = it->lkey; lval = it->lval; center = it->center; } return this; } else if (lkey >= it->lkey) { // Add left center = new TTNode(rkey, rval, EMPTYKEY, NULL, center, right, NULL); rkey = EMPTYKEY; rval = NULL; right = NULL; it->left = left; left = it; return this; } else if (rkey < it->lkey) { // Add center it->center = new TTNode(rkey, rval, EMPTYKEY, NULL, it->center, right, NULL); it->left = this; rkey = EMPTYKEY; rval = NULL; right = NULL; return it; } else { // Add right TTNode* N1 = new TTNode(rkey, rval, EMPTYKEY, NULL, this, it, NULL); it->left = right; right = NULL; rkey = EMPTYKEY; rval = NULL; return N1; } } Figure 10.16 The 2-3 tree node add method. Sec. 10.5 B-Trees 367 20 12 18 21 23 30 31 38 4710 15 24 33 45 48 50 52 60 Figure 10.17 A B-tree of order four. • All leaves are at the same level in the tree, so the tree is always height bal- anced. The B-tree is a generalization of the 2-3 tree. Put another way, a 2-3 tree is a B-tree of order three. Normally, the size of a node in the B-tree is chosen to fill a disk block. A B-tree node implementation typically allows 100 or more children. Thus, a B-tree node is equivalent to a disk block, and a “pointer” value stored in the tree is actually the number of the block containing the child node (usually interpreted as an offset from the beginning of the corresponding disk file). In a typical application, the B-tree’s access to the disk file will be managed using a buffer pool and a block-replacement scheme such as LRU (see Section 8.3). Figure 10.17 shows a B-tree of order four. Each node contains up to three keys, and internal nodes have up to four children. Search in a B-tree is a generalization of search in a 2-3 tree. It is an alternating two-step process, beginning with the root node of the B-tree. 1. Perform a binary search on the records in the current node. If a record with the search key is found, then return that record. If the current node is a leaf node and the key is not found, then report an unsuccessful search. 2. Otherwise, follow the proper branch and repeat the process. For example, consider a search for the record with key value 47 in the tree of Figure 10.17. The root node is examined and the second (right) branch taken. After examining the node at level 1, the third branch is taken to the next level to arrive at the leaf node containing a record with key value 47. B-tree insertion is a generalization of 2-3 tree insertion. The first step is to find the leaf node that should contain the key to be inserted, space permitting. If there is room in this node, then insert the key. If there is not, then split the node into two and promote the middle key to the parent. If the parent becomes full, then it is split in turn, and its middle key promoted. Note that this insertion process is guaranteed to keep all nodes at least half full. For example, when we attempt to insert into a full internal node of a B-tree of order four, there will now be five children that must be dealt with. The node is split into 368 Chap. 10 Indexing two nodes containing two keys each, thus retaining the B-tree property. The middle of the five children is promoted to its parent. 10.5.1 B+-Trees The previous section mentioned that B-trees are universally used to implement large-scale disk-based systems. Actually, the B-tree as described in the previ- ous section is almost never implemented, nor is the 2-3 tree as described in Sec- tion 10.4. What is most commonly implemented is a variant of the B-tree, called the B+-tree. When greater efficiency is required, a more complicated variant known as the B⇤-tree is used. When data are static, a linear index provides an extremely efficient way to search. The problem is how to handle those pesky inserts and deletes. We could try to keep the core idea of storing a sorted array-based list, but make it more flexible by breaking the list into manageable chunks that are more easily updated. How might we do that? First, we need to decide how big the chunks should be. Since the data are on disk, it seems reasonable to store a chunk that is the size of a disk block, or a small multiple of the disk block size. If the next record to be inserted belongs to a chunk that hasn’t filled its block then we can just insert it there. The fact that this might cause other records in that chunk to move a little bit in the array is not important, since this does not cause any extra disk accesses so long as we move data within that chunk. But what if the chunk fills up the entire block that contains it? We could just split it in half. What if we want to delete a record? We could just take the deleted record out of the chunk, but we might not want a lot of near-empty chunks. So we could put adjacent chunks together if they have only a small amount of data between them. Or we could shuffle data between adjacent chunks that together contain more data. The big problem would be how to find the desired chunk when processing a record with a given key. Perhaps some sort of tree-like structure could be used to locate the appropriate chunk. These ideas are exactly what motivate the B+-tree. The B+-tree is essentially a mechanism for managing a sorted array-based list, where the list is broken into chunks. The most significant difference between the B+-tree and the BST or the stan- dard B-tree is that the B+-tree stores records only at the leaf nodes. Internal nodes store key values, but these are used solely as placeholders to guide the search. This means that internal nodes are significantly different in structure from leaf nodes. Internal nodes store keys to guide the search, associating each key with a pointer to a child B+-tree node. Leaf nodes store actual records, or else keys and pointers to actual records in a separate disk file if the B+-tree is being used purely as an index. Depending on the size of a record as compared to the size of a key, a leaf node in a B+-tree of order m might have enough room to store more or less than m records. The requirement is simply that the leaf nodes store enough records to remain at least half full. The leaf nodes of a B+-tree are normally linked together Sec. 10.5 B-Trees 369 23 30 31 33 45 47 48 48 50 5210 12 15 18 19 20 21 22 33 18 23 Figure 10.18 Example of a B+-tree of order four. Internal nodes must store between two and four children. For this example, the record size is assumed to be such that leaf nodes store between three and five records. to form a doubly linked list. Thus, the entire collection of records can be traversed in sorted order by visiting all the leaf nodes on the linked list. Here is a C++-like pseudocode representation for the B+-tree node class. Leaf node and internal node subclasses would implement this base class. // Abstract class definition for B+-trees template class BPNode { public: BPNode* lftptr; BPNode* rghtptr; // Links to siblings virtual ˜BPNode() {} // Base destructor virtual bool isLeaf() const =0; // True if node is a leaf virtual bool isFull() const =0; // True if node is full virtual int numrecs() const =0; // Current num of records virtual Key* keys() const=0; // Return array of keys }; An important implementation detail to note is that while Figure 10.17 shows internal nodes containing three keys and four pointers, class BPNode is slightly different in that it stores key/pointer pairs. Figure 10.17 shows the B+-tree as it is traditionally drawn. To simplify implementation in practice, nodes really do associate a key with each pointer. Each internal node should be assumed to hold in the leftmost position an additional key that is less than or equal to any possible key value in the node’s leftmost subtree. B+-tree implementations typically store an additional dummy record in the leftmost leaf node whose key value is less than any legal key value. B+-trees are exceptionally good for range queries. Once the first record in the range has been found, the rest of the records with keys in the range can be accessed by sequential processing of the remaining records in the first node, and then continuing down the linked list of leaf nodes as far as necessary. Figure 10.18 illustrates the B+-tree. Search in a B+-tree is nearly identical to search in a regular B-tree, except that the search must always continue to the proper leaf node. Even if the search-key value is found in an internal node, this is only a placeholder and does not provide 370 Chap. 10 Indexing template E BPTree::findhelp(BPNode* rt, const Key k) const { int currec = binaryle(rt->keys(), rt->numrecs(), k); if (rt->isLeaf()) if ((((BPLeaf*)rt)->keys())[currec] == k) return ((BPLeaf*)rt)->recs(currec); else return NULL; else return findhelp(((BPInternal*)rt)-> pointers(currec), k); } Figure 10.19 Implementation for the B+-tree search method. access to the actual record. To find a record with key value 33 in the B+-tree of Figure 10.18, search begins at the root. The value 33 stored in the root merely serves as a placeholder, indicating that keys with values greater than or equal to 33 are found in the second subtree. From the second child of the root, the first branch is taken to reach the leaf node containing the actual record (or a pointer to the actual record) with key value 33. Figure 10.19 shows a pseudocode sketch of the B+-tree search algorithm. B+-tree insertion is similar to B-tree insertion. First, the leaf L that should contain the record is found. If L is not full, then the new record is added, and no other B+-tree nodes are affected. If L is already full, split it in two (dividing the records evenly among the two nodes) and promote a copy of the least-valued key in the newly formed right node. As with the 2-3 tree, promotion might cause the parent to split in turn, perhaps eventually leading to splitting the root and causing the B+-tree to gain a new level. B+-tree insertion keeps all leaf nodes at equal depth. Figure 10.20 illustrates the insertion process through several examples. Fig- ure 10.21 shows a C++-like pseudocode sketch of the B+-tree insert algorithm. To delete record R from the B+-tree, first locate the leaf L that contains R. If L is more than half full, then we need only remove R, leaving L still at least half full. This is demonstrated by Figure 10.22. If deleting a record reduces the number of records in the node below the min- imum threshold (called an underflow), then we must do something to keep the node sufficiently full. The first choice is to look at the node’s adjacent siblings to determine if they have a spare record that can be used to fill the gap. If so, then enough records are transferred from the sibling so that both nodes have about the same number of records. This is done so as to delay as long as possible the next time when a delete causes this node to underflow again. This process might require that the parent node has its placeholder key value revised to reflect the true first key value in each node. Figure 10.23 illustrates the process. Sec. 10.5 B-Trees 371 33 (b)(a) 1012 233348 10 23 33 5012 483318 (c) 33 23 4818 (d) 48 1012 18 20 2123 31 33 45 47 48 5015 52 12 18 20 21 23 30 31 33 45 4710 15 48 50 52 Figure 10.20 Examples of B+-tree insertion. (a) A B+-tree containing five records. (b) The result of inserting a record with key value 50 into the tree of (a). The leaf node splits, causing creation of the first internal node. (c) The B+-tree of (b) after further insertions. (d) The result of inserting a record with key value 30 into the tree of (c). The second leaf node splits, which causes the internal node to split in turn, creating a new root. template BPNode* BPTree::inserthelp(BPNode* rt, const Key& k, const E& e) { if (rt->isLeaf()) // At leaf node: insert here return ((BPLeaf*)rt)->add(k, e); // Add to internal node int currec = binaryle(rt->keys(), rt->numrecs(), k); BPNode* temp = inserthelp( ((BPInternal*)root)->pointers(currec), k, e); if (temp != ((BPInternal*)rt)->pointers(currec)) return ((BPInternal*)rt)-> add(k, (BPInternal*)temp); else return rt; } Figure 10.21 A C++-like pseudocode sketch of the B+-tree insert algorithm. 372 Chap. 10 Indexing 33 23 4818 101215 23 30 3119 2021 22 4733 45 48 50 52 Figure 10.22 Simple deletion from a B+-tree. The record with key value 18 is removed from the tree of Figure 10.18. Note that even though 18 is also a place- holder used to direct search in the parent node, that value need not be removed from internal nodes even if no record in the tree has key value 18. Thus, the leftmost node at level one in this example retains the key with value 18 after the record with key value 18 has been removed from the second leaf node. 33 19 4823 101518 19 20 21 22 33 45 4723 30 31 48 50 52 Figure 10.23 Deletion from the B+-tree of Figure 10.18 via borrowing from a sibling. The key with value 12 is deleted from the leftmost leaf, causing the record with key value 18 to shift to the leftmost leaf to take its place. Note that the parent must be updated to properly indicate the key range within the subtrees. In this example, the parent node has its leftmost key value changed to 19. If neither sibling can lend a record to the under-full node (call it N), then N must give its records to a sibling and be removed from the tree. There is certainly room to do this, because the sibling is at most half full (remember that it had no records to contribute to the current node), and N has become less than half full because it is under-flowing. This merge process combines two subtrees of the par- ent, which might cause it to underflow in turn. If the last two children of the root merge together, then the tree loses a level. Figure 10.24 illustrates the node-merge deletion process. Figure 10.25 shows C++-like pseudocode for the B+-tree delete algorithm. The B+-tree requires that all nodes be at least half full (except for the root). Thus, the storage utilization must be at least 50%. This is satisfactory for many implementations, but note that keeping nodes fuller will result both in less space required (because there is less empty space in the disk file) and in more efficient processing (fewer blocks on average will be read into memory because the amount of information in each block is greater). Because B-trees have become so popular, many algorithm designers have tried to improve B-tree performance. One method Sec. 10.5 B-Trees 373 48 (a) 45 4748 50 52 23 3318 (b) 18 19 20 21 23 30 31101215 22 4850524547 Figure 10.24 Deleting the record with key value 33 from the B+-tree of Fig- ure 10.18 via collapsing siblings. (a) The two leftmost leaf nodes merge together to form a single leaf. Unfortunately, the parent node now has only one child. (b) Because the left subtree has a spare leaf node, that node is passed to the right subtree. The placeholder values of the root and the right internal node are updated to reflect the changes. Value 23 moves to the root, and old root value 33 moves to the rightmost internal node. /** Delete a record with the given key value, and return true if the root underflows */ template bool BPTree::removehelp(BPNode* rt, const Key& k) { int currec = binaryle(rt->keys(), rt->numrecs(), k); if (rt->isLeaf()) if (((BPLeaf*)rt)->keys()[currec] == k) return ((BPLeaf*)rt)->del(currec); else return false; else // Process internal node if (removehelp(((BPInternal*)rt)-> pointers(currec), k)) // Child will merge if necessary return ((BPInternal*)rt)->underflow(currec); else return false; } Figure 10.25 C++-like pseudocode for the B+-tree delete algorithm. 374 Chap. 10 Indexing for doing so is to use the B+-tree variant known as the B⇤-tree. The B⇤-tree is identical to the B+-tree, except for the rules used to split and merge nodes. Instead of splitting a node in half when it overflows, the B⇤-tree gives some records to its neighboring sibling, if possible. If the sibling is also full, then these two nodes split into three. Similarly, when a node underflows, it is combined with its two siblings, and the total reduced to two nodes. Thus, the nodes are always at least two thirds full.2 10.5.2 B-Tree Analysis The asymptotic cost of search, insertion, and deletion of records from B-trees, B+-trees, and B⇤-trees is ⇥(log n) where n is the total number of records in the tree. However, the base of the log is the (average) branching factor of the tree. Typical database applications use extremely high branching factors, perhaps 100 or more. Thus, in practice the B-tree and its variants are extremely shallow. As an illustration, consider a B+-tree of order 100 and leaf nodes that contain up to 100 records. A B+-tree with height one (that is, just a single leaf node) can have at most 100 records. A B+-tree with height two (a root internal node whose children are leaves) must have at least 100 records (2 leaves with 50 records each). It has at most 10,000 records (100 leaves with 100 records each). A B+-tree with height three must have at least 5000 records (two second-level nodes with 50 chil- dren containing 50 records each) and at most one million records (100 second-level nodes with 100 full children each). A B+-tree with height four must have at least 250,000 records and at most 100 million records. Thus, it would require an ex- tremely large database to generate a B+-tree of more than height four. The B+-tree split and insert rules guarantee that every node (except perhaps the root) is at least half full. So they are on average about 3/4 full. But the internal nodes are purely overhead, since the keys stored there are used only by the tree to direct search, rather than store actual data. Does this overhead amount to a signifi- cant use of space? No, because once again the high fan-out rate of the tree structure means that the vast majority of nodes are leaf nodes. Recall (from Section 6.4) that a full K-ary tree has approximately 1/K of its nodes as internal nodes. This means that while half of a full binary tree’s nodes are internal nodes, in a B+-tree of order 100 probably only about 1/75 of its nodes are internal nodes. This means that the overhead associated with internal nodes is very low. We can reduce the number of disk fetches required for the B-tree even more by using the following methods. First, the upper levels of the tree can be stored in 2This concept can be extended further if higher space utilization is required. However, the update routines become much more complicated. I once worked on a project where we implemented 3-for-4 node split and merge routines. This gave better performance than the 2-for-3 node split and merge routines of the B⇤-tree. However, the spitting and merging routines were so complicated that even their author could no longer understand them once they were completed! Sec. 10.6 Further Reading 375 main memory at all times. Because the tree branches so quickly, the top two levels (levels 0 and 1) require relatively little space. If the B-tree is only height four, then at most two disk fetches (internal nodes at level two and leaves at level three) are required to reach the pointer to any given record. A buffer pool could be used to manage nodes of the B-tree. Several nodes of the tree would typically be in main memory at one time. The most straightforward approach is to use a standard method such as LRU to do node replacement. How- ever, sometimes it might be desirable to “lock” certain nodes such as the root into the buffer pool. In general, if the buffer pool is even of modest size (say at least twice the depth of the tree), no special techniques for node replacement will be required because the upper-level nodes will naturally be accessed frequently. 10.6 Further Reading For an expanded discussion of the issues touched on in this chapter, see a gen- eral file processing text such as File Structures: A Conceptual Toolkit by Folk and Zoellick [FZ98]. In particular, Folk and Zoellick provide a good discussion of the relationship between primary and secondary indices. The most thorough dis- cussion on various implementations for the B-tree is the survey article by Comer [Com79]. Also see [Sal88] for further details on implementing B-trees. See Shaf- fer and Brown [SB93] for a discussion of buffer pool management strategies for B+-tree-like data structures. 10.7 Exercises 10.1 Assume that a computer system has disk blocks of 1024 bytes, and that you are storing records that have 4-byte keys and 4-byte data fields. The records are sorted and packed sequentially into the disk file. (a) Assume that a linear index uses 4 bytes to store the key and 4 bytes to store the block ID for the associated records. What is the greatest number of records that can be stored in the file if a linear index of size 256KB is used? (b) What is the greatest number of records that can be stored in the file if the linear index is also stored on disk (and thus its size is limited only by the second-level index) when using a second-level index of 1024 bytes (i.e., 256 key values) as illustrated by Figure 10.2? Each element of the second-level index references the smallest key value for a disk block of the linear index. 10.2 Assume that a computer system has disk blocks of 4096 bytes, and that you are storing records that have 4-byte keys and 64-byte data fields. The records are sorted and packed sequentially into the disk file. 376 Chap. 10 Indexing (a) Assume that a linear index uses 4 bytes to store the key and 4 bytes to store the block ID for the associated records. What is the greatest number of records that can be stored in the file if a linear index of size 2MB is used? (b) What is the greatest number of records that can be stored in the file if the linear index is also stored on disk (and thus its size is limited only by the second-level index) when using a second-level index of 4096 bytes (i.e., 1024 key values) as illustrated by Figure 10.2? Each element of the second-level index references the smallest key value for a disk block of the linear index. 10.3 Modify the function binary of Section 3.5 so as to support variable-length records with fixed-length keys indexed by a simple linear index as illustrated by Figure 10.1. 10.4 Assume that a database stores records consisting of a 2-byte integer key and a variable-length data field consisting of a string. Show the linear index (as illustrated by Figure 10.1) for the following collection of records: 397 Hello world! 82 XYZ 1038 This string is rather long 1037 This is shorter 42 ABC 2222 Hello new world! 10.5 Each of the following series of records consists of a four-digit primary key (with no duplicates) and a four-character secondary key (with many dupli- cates). 3456 DEER 2398 DEER 2926 DUCK 9737 DEER 7739 GOAT 9279 DUCK 1111 FROG 8133 DEER 7183 DUCK 7186 FROG (a) Show the inverted list (as illustrated by Figure 10.4) for this collection of records. (b) Show the improved inverted list (as illustrated by Figure 10.5) for this collection of records. Sec. 10.8 Projects 377 10.6 Under what conditions will ISAM be more efficient than a B+-tree imple- mentation? 10.7 Prove that the number of leaf nodes in a 2-3 tree with height k is between 2k1 and 3k1. 10.8 Show the result of inserting the values 55 and 46 into the 2-3 tree of Fig- ure 10.9. 10.9 You are given a series of records whose keys are letters. The records arrive in the following order: C, S, D, T, A, M, P, I, B, W, N, G, U, R, K, E, H, O, L, J. Show the 2-3 tree that results from inserting these records. 10.10 You are given a series of records whose keys are letters. The records are inserted in the following order: C, S, D, T, A, M, P, I, B, W, N, G, U, R, K, E, H, O, L, J. Show the tree that results from inserting these records when the 2-3 tree is modified to be a 2-3+ tree, that is, the internal nodes act only as placeholders. Assume that the leaf nodes are capable of holding up to two records. 10.11 Show the result of inserting the value 55 into the B-tree of Figure 10.17. 10.12 Show the result of inserting the values 1, 2, 3, 4, 5, and 6 (in that order) into the B+-tree of Figure 10.18. 10.13 Show the result of deleting the values 18, 19, and 20 (in that order) from the B+-tree of Figure 10.24b. 10.14 You are given a series of records whose keys are letters. The records are inserted in the following order: C, S, D, T, A, M, P, I, B, W, N, G, U, R, K, E, H, O, L, J. Show the B+-tree of order four that results from inserting these records. Assume that the leaf nodes are capable of storing up to three records. 10.15 Assume that you have a B+-tree whose internal nodes can store up to 100 children and whose leaf nodes can store up to 15 records. What are the minimum and maximum number of records that can be stored by the B+-tree with heights 1, 2, 3, 4, and 5? 10.16 Assume that you have a B+-tree whose internal nodes can store up to 50 children and whose leaf nodes can store up to 50 records. What are the minimum and maximum number of records that can be stored by the B+-tree with heights 1, 2, 3, 4, and 5? 10.8 Projects 10.1 Implement a two-level linear index for variable-length records as illustrated by Figures 10.1 and 10.2. Assume that disk blocks are 1024 bytes in length. Records in the database file should typically range between 20 and 200 bytes, including a 4-byte key value. Each record of the index file should store a key value and the byte offset in the database file for the first byte of the 378 Chap. 10 Indexing corresponding record. The top-level index (stored in memory) should be a simple array storing the lowest key value on the corresponding block in the index file. 10.2 Implement the 2-3+ tree, that is, a 2-3 tree where the internal nodes act only as placeholders. Your 2-3+ tree should implement the dictionary interface of Section 4.4. 10.3 Implement the dictionary ADT of Section 4.4 for a large file stored on disk by means of the B+-tree of Section 10.5. Assume that disk blocks are 1024 bytes, and thus both leaf nodes and internal nodes are also 1024 bytes. Records should store a 4-byte (int) key value and a 60-byte data field. Inter- nal nodes should store key value/pointer pairs where the “pointer” is actually the block number on disk for the child node. Both internal nodes and leaf nodes will need room to store various information such as a count of the records stored on that node, and a pointer to the next node on that level. Thus, leaf nodes will store 15 records, and internal nodes will have room to store about 120 to 125 children depending on how you implement them. Use a buffer pool (Section 8.3) to manage access to the nodes stored on disk. PART IV Advanced Data Structures 379 11 Graphs Graphs provide the ultimate in data structure flexibility. Graphs can model both real-world systems and abstract problems, so they are used in hundreds of applica- tions. Here is a small sampling of the range of problems that graphs are routinely applied to. 1. Modeling connectivity in computer and communications networks. 2. Representing a map as a set of locations with distances between locations; used to compute shortest routes between locations. 3. Modeling flow capacities in transportation networks. 4. Finding a path from a starting condition to a goal condition; for example, in artificial intelligence problem solving. 5. Modeling computer algorithms, showing transitions from one program state to another. 6. Finding an acceptable order for finishing subtasks in a complex activity, such as constructing large buildings. 7. Modeling relationships such as family trees, business or military organiza- tions, and scientific taxonomies. We begin in Section 11.1 with some basic graph terminology and then define two fundamental representations for graphs, the adjacency matrix and adjacency list. Section 11.2 presents a graph ADT and simple implementations based on the adjacency matrix and adjacency list. Section 11.3 presents the two most commonly used graph traversal algorithms, called depth-first and breadth-first search, with application to topological sorting. Section 11.4 presents algorithms for solving some problems related to finding shortest routes in a graph. Finally, Section 11.5 presents algorithms for finding the minimum-cost spanning tree, useful for deter- mining lowest-cost connectivity in a network. Besides being useful and interesting in their own right, these algorithms illustrate the use of some data structures pre- sented in earlier chapters. 381 382 Chap. 11 Graphs (b) (c) 0 3 4 1 2 7 1 2 3 4 (a) 1 Figure 11.1 Examples of graphs and terminology. (a) A graph. (b) A directed graph (digraph). (c) A labeled (directed) graph with weights associated with the edges. In this example, there is a simple path from Vertex 0 to Vertex 3 containing Vertices 0, 1, and 3. Vertices 0, 1, 3, 2, 4, and 1 also form a path, but not a simple path because Vertex 1 appears twice. Vertices 1, 3, 2, 4, and 1 form a simple cycle. 11.1 Terminology and Representations A graph G =(V, E) consists of a set of vertices V and a set of edges E, such that each edge in E is a connection between a pair of vertices in V.1 The number of vertices is written |V|, and the number of edges is written |E|. |E| can range from zero to a maximum of |V| 2 |V|. A graph with relatively few edges is called sparse, while a graph with many edges is called dense. A graph containing all possible edges is said to be complete. A graph with edges directed from one vertex to another (as in Figure 11.1(b)) is called a directed graph or digraph. A graph whose edges are not directed is called an undirected graph (as illustrated by Figure 11.1(a)). A graph with labels associated with its vertices (as in Figure 11.1(c)) is called a labeled graph.Two vertices are adjacent if they are joined by an edge. Such vertices are also called neighbors. An edge connecting Vertices U and V is written (U, V). Such an edge is said to be incident on Vertices U and V. Associated with each edge may be a cost or weight. Graphs whose edges have weights (as in Figure 11.1(c)) are said to be weighted. A sequence of vertices v1, v2, ..., vn forms a path of length n 1 if there exist edges from vi to vi+1 for 1  ifirst(v); w < G->n(); w = G->next(v,w)) This for loop gets the first neighbor of v, then works through the remaining neigh- bors of v until a value equal to G->n() is returned, signaling that all neighbors of v have been visited. For example, first(1) in Figure 11.4 would return 0. next(1, 0) would return 3. next(0, 3) would return 4. next(1, 4) would return 5, which is not a vertex in the graph. Sec. 11.2 Graph Implementations 387 // Graph abstract class. This ADT assumes that the number // of vertices is fixed when the graph is created. class Graph { private: void operator =(const Graph&) {} // Protect assignment Graph(const Graph&) {} // Protect copy constructor public: Graph() {} // Default constructor virtual ˜Graph() {} // Base destructor // Initialize a graph of n vertices virtual void Init(int n) =0; // Return: the number of vertices and edges virtual int n() =0; virtual int e() =0; // Return v’s first neighbor virtual int first(int v) =0; // Return v’s next neighbor virtual int next(int v, int w) =0; // Set the weight for an edge // i, j: The vertices // wgt: Edge weight virtual void setEdge(int v1, int v2, int wght) =0; // Delete an edge // i, j: The vertices virtual void delEdge(int v1, int v2) =0; // Determine if an edge is in the graph // i, j: The vertices // Return: true if edge i,j has non-zero weight virtual bool isEdge(int i, int j) =0; // Return an edge’s weight // i, j: The vertices // Return: The weight of edge i,j, or zero virtual int weight(int v1, int v2) =0; // Get and Set the mark value for a vertex // v: The vertex // val: The value to set virtual int getMark(int v) =0; virtual void setMark(int v, int val) =0; }; Figure 11.5 A graph ADT. This ADT assumes that the number of vertices is fixed when the graph is created, but that edges can be added and removed. It also supports a mark array to aid graph traversal algorithms. 388 Chap. 11 Graphs It is reasonably straightforward to implement our graph and edge ADTs using either the adjacency list or adjacency matrix. The sample implementations pre- sented here do not address the issue of how the graph is actually created. The user of these implementations must add functionality for this purpose, perhaps reading the graph description from a file. The graph can be built up by using the setEdge function provided by the ADT. Figure 11.6 shows an implementation for the adjacency matrix. Array Mark stores the information manipulated by the setMark and getMark functions. The edge matrix is implemented as an integer array of size n ⇥ n for a graph of n ver- tices. Position (i, j) in the matrix stores the weight for edge (i, j) if it exists. A weight of zero for edge (i, j) is used to indicate that no edge connects Vertices i and j. Given a vertex V, function first locates the position in matrix of the first edge (if any) of V by beginning with edge (V, 0) and scanning through row V until an edge is found. If no edge is incident on V, then first returns n. Function next locates the edge following edge (i, j) (if any) by continuing down the row of Vertex i starting at position j +1, looking for an edge. If no such edge exists, next returns n. Functions setEdge and delEdge adjust the appropriate value in the array. Function weight returns the value stored in the appropriate position in the array. Figure 11.7 presents an implementation of the adjacency list representation for graphs. Its main data structure is an array of linked lists, one linked list for each vertex. These linked lists store objects of type Edge, which merely stores the index for the vertex pointed to by the edge, along with the weight of the edge. Because the Edge class is assumed to be private to the Graphl class, its data members have been made public for convenience. // Edge class for Adjacency List graph representation class Edge { int vert, wt; public: Edge() { vert = -1; wt = -1; } Edge(int v, int w) { vert = v; wt = w; } int vertex() { return vert; } int weight() { return wt; } }; Implementation for Graphl member functions is straightforward in principle, with the key functions being setEdge, delEdge, and weight. They simply start at the beginning of the adjacency list and move along it until the desired vertex has been found. Note that isEdge checks to see if j is already the current neighbor in i’s adjacency list, since this will often be true when processing the neighbors of each vertex in turn. Sec. 11.2 Graph Implementations 389 // Implementation for the adjacency matrix representation class Graphm : public Graph { private: int numVertex, numEdge; // Store number of vertices, edges int **matrix; // Pointer to adjacency matrix int *mark; // Pointer to mark array public: Graphm(int numVert) // Constructor { Init(numVert); } ˜Graphm() { // Destructor delete [] mark; // Return dynamically allocated memory for (int i=0; i0, "Illegal weight value"); if (matrix[v1][v2] == 0) numEdge++; matrix[v1][v2] = wt; } void delEdge(int v1, int v2) { // Delete edge (v1, v2) if (matrix[v1][v2] != 0) numEdge--; matrix[v1][v2] = 0; } bool isEdge(int i, int j) // Is (i, j) an edge? { return matrix[i][j] != 0; } int weight(int v1, int v2) { return matrix[v1][v2]; } int getMark(int v) { return mark[v]; } void setMark(int v, int val) { mark[v] = val; } }; Figure 11.6 (continued) 11.3 Graph Traversals Often it is useful to visit the vertices of a graph in some specific order based on the graph’s topology. This is known as a graph traversal and is similar in concept to a tree traversal. Recall that tree traversals visit every node exactly once, in some specified order such as preorder, inorder, or postorder. Multiple tree traversals exist because various applications require the nodes to be visited in a particular order. For example, to print a BST’s nodes in ascending order requires an inorder traver- sal as opposed to some other traversal. Standard graph traversal orders also exist. Each is appropriate for solving certain problems. For example, many problems in artificial intelligence programming are modeled using graphs. The problem domain may consist of a large collection of states, with connections between various pairs of states. Solving the problem may require getting from a specified start state to a specified goal state by moving between states only through the connections. Typi- cally, the start and goal states are not directly connected. To solve this problem, the vertices of the graph must be searched in some organized manner. Graph traversal algorithms typically begin with a start vertex and attempt to visit the remaining vertices from there. Graph traversals must deal with a number of troublesome cases. First, it may not be possible to reach all vertices from the start vertex. This occurs when the graph is not connected. Second, the graph may contain cycles, and we must make sure that cycles do not cause the algorithm to go into an infinite loop. Sec. 11.3 Graph Traversals 391 class Graphl : public Graph { private: List** vertex; // List headers int numVertex, numEdge; // Number of vertices, edges int *mark; // Pointer to mark array public: Graphl(int numVert) { Init(numVert); } ˜Graphl() { // Destructor delete [] mark; // Return dynamically allocated memory for (int i=0; i**) new List*[numVertex]; for (i=0; i(); } int n() { return numVertex; } // Number of vertices int e() { return numEdge; } // Number of edges int first(int v) { // Return first neighbor of "v" if (vertex[v]->length() == 0) return numVertex; // No neighbor vertex[v]->moveToStart(); Edge it = vertex[v]->getValue(); return it.vertex(); } // Get v’s next neighbor after w int next(int v, int w) { Edge it; if (isEdge(v, w)) { if ((vertex[v]->currPos()+1) < vertex[v]->length()) { vertex[v]->next(); it = vertex[v]->getValue(); return it.vertex(); } } return n(); // No neighbor } Figure 11.7 An implementation for the adjacency list. 392 Chap. 11 Graphs // Set edge (i, j) to "weight" void setEdge(int i, int j, int weight) { Assert(weight>0, "May not set weight to 0"); Edge currEdge(j, weight); if (isEdge(i, j)) { // Edge already exists in graph vertex[i]->remove(); vertex[i]->insert(currEdge); } else { // Keep neighbors sorted by vertex index numEdge++; for (vertex[i]->moveToStart(); vertex[i]->currPos() < vertex[i]->length(); vertex[i]->next()) { Edge temp = vertex[i]->getValue(); if (temp.vertex() > j) break; } vertex[i]->insert(currEdge); } } void delEdge(int i, int j) { // Delete edge (i, j) if (isEdge(i,j)) { vertex[i]->remove(); numEdge--; } } bool isEdge(int i, int j) { // Is (i,j) an edge? Edge it; for (vertex[i]->moveToStart(); vertex[i]->currPos() < vertex[i]->length(); vertex[i]->next()) { // Check whole list Edge temp = vertex[i]->getValue(); if (temp.vertex() == j) return true; } return false; } int weight(int i, int j) { // Return weight of (i, j) Edge curr; if (isEdge(i, j)) { curr = vertex[i]->getValue(); return curr.weight(); } else return 0; } int getMark(int v) { return mark[v]; } void setMark(int v, int val) { mark[v] = val; } }; Figure 11.7 (continued) Sec. 11.3 Graph Traversals 393 Graph traversal algorithms can solve both of these problems by maintaining a mark bit for each vertex on the graph. At the beginning of the algorithm, the mark bit for all vertices is cleared. The mark bit for a vertex is set when the vertex is first visited during the traversal. If a marked vertex is encountered during traversal, it is not visited a second time. This keeps the program from going into an infinite loop when it encounters a cycle. Once the traversal algorithm completes, we can check to see if all vertices have been processed by checking the mark bit array. If not all vertices are marked, we can continue the traversal from another unmarked vertex. Note that this process works regardless of whether the graph is directed or undirected. To ensure visiting all vertices, graphTraverse could be called as follows on a graph G: void graphTraverse(Graph* G) { int v; for (v=0; vn(); v++) G->setMark(v, UNVISITED); // Initialize mark bits for (v=0; vn(); v++) if (G->getMark(v) == UNVISITED) doTraverse(G, v); } Function “doTraverse” might be implemented by using one of the graph traver- sals described in this section. 11.3.1 Depth-First Search The first method of organized graph traversal is called depth-first search (DFS). Whenever a vertex V is visited during the search, DFS will recursively visit all of V’s unvisited neighbors. Equivalently, DFS will add all edges leading out of v to a stack. The next vertex to be visited is determined by popping the stack and following that edge. The effect is to follow one branch through the graph to its conclusion, then it will back up and follow another branch, and so on. The DFS process can be used to define a depth-first search tree. This tree is composed of the edges that were followed to any new (unvisited) vertex during the traversal, and leaves out the edges that lead to already visited vertices. DFS can be applied to directed or undirected graphs. Here is an implementation for the DFS algorithm: void DFS(Graph* G, int v) { // Depth first search PreVisit(G, v); // Take appropriate action G->setMark(v, VISITED); for (int w=G->first(v); wn(); w = G->next(v,w)) if (G->getMark(w) == UNVISITED) DFS(G, w); PostVisit(G, v); // Take appropriate action } This implementation contains calls to functions PreVisit and PostVisit. These functions specify what activity should take place during the search. Just 394 Chap. 11 Graphs (a) (b) AB D F AB C D F E C E Figure 11.8 (a) A graph. (b) The depth-first search tree for the graph when starting at Vertex A. as a preorder tree traversal requires action before the subtrees are visited, some graph traversals require that a vertex be processed before ones further along in the DFS. Alternatively, some applications require activity after the remaining vertices are processed; hence the call to function PostVisit. This would be a natural opportunity to make use of the visitor design pattern described in Section 1.3.2. Figure 11.8 shows a graph and its corresponding depth-first search tree. Fig- ure 11.9 illustrates the DFS process for the graph of Figure 11.8(a). DFS processes each edge once in a directed graph. In an undirected graph, DFS processes each edge from both directions. Each vertex must be visited, but only once, so the total cost is ⇥(|V| + |E|). 11.3.2 Breadth-First Search Our second graph traversal algorithm is known as a breadth-first search (BFS). BFS examines all vertices connected to the start vertex before visiting vertices fur- ther away. BFS is implemented similarly to DFS, except that a queue replaces the recursion stack. Note that if the graph is a tree and the start vertex is at the root, BFS is equivalent to visiting vertices level by level from top to bottom. Fig- ure 11.10 provides an implementation for the BFS algorithm. Figure 11.11 shows a graph and the corresponding breadth-first search tree. Figure 11.12 illustrates the BFS process for the graph of Figure 11.11(a). 11.3.3 Topological Sort Assume that we need to schedule a series of tasks, such as classes or construction jobs, where we cannot start one task until after its prerequisites are completed. We wish to organize the tasks into a linear order that allows us to complete them one Sec. 11.3 Graph Traversals 395 Call DFS on A Mark B Process (B, C) Process (B, F) Print (B, F) and call DFS on F Process (F, E) Print (F, E) and call DFS on E Done with B Pop B Mark A Process (A, C) Print (A, C) and call DFS on C Mark F Process (F, B) Process (F, C) Process (F, D) Print (F, D) and call DFS on D Mark E Process (E, A) Process (E, F) Pop E Continue with C Process (C, E) Process (C, F) Pop C Mark C Process (C, A) Process (C, B) Print (C, B) and call DFS on C Mark D Done with F Pop F Continue with A Process (A, E) Pop A DFS complete Pop D Process (D, C) Process (D, F) E F B C A A F B C A C A F B C A C A D F B C A A B C A B C A F B C A Figure 11.9 A detailed illustration of the DFS process for the graph of Fig- ure 11.8(a) starting at Vertex A. The steps leading to each change in the recursion stack are described. 396 Chap. 11 Graphs void BFS(Graph* G, int start, Queue* Q) { int v, w; Q->enqueue(start); // Initialize Q G->setMark(start, VISITED); while (Q->length() != 0) { // Process all vertices on Q v = Q->dequeue(); PreVisit(G, v); // Take appropriate action for (w=G->first(v); wn(); w = G->next(v,w)) if (G->getMark(w) == UNVISITED) { G->setMark(w, VISITED); Q->enqueue(w); } } } Figure 11.10 Implementation for the breadth-first graph traversal algorithm (a) (b) B C A C B DD F EE A F Figure 11.11 (a) A graph. (b) The breadth-first search tree for the graph when starting at Vertex A. at a time without violating any prerequisites. We can model the problem using a DAG. The graph is directed because one task is a prerequisite of another — the vertices have a directed relationship. It is acyclic because a cycle would indicate a conflicting series of prerequisites that could not be completed without violating at least one prerequisite. The process of laying out the vertices of a DAG in a linear order to meet the prerequisite rules is called a topological sort. Figure 11.14 illustrates the problem. An acceptable topological sort for this example is J1, J2, J3, J4, J5, J6, J7. A topological sort may be found by performing a DFS on the graph. When a vertex is visited, no action is taken (i.e., function PreVisit does nothing). When the recursion pops back to that vertex, function PostVisit prints the vertex. This yields a topological sort in reverse order. It does not matter where the sort starts, as long as all vertices are visited in the end. Figure 11.13 shows an implementation for the DFS-based algorithm. Sec. 11.3 Graph Traversals 397 Initial call to BFS on A. Mark A and put on the queue. Dequeue A. Process (A, C). Mark and enqueue C. Print (A, C). Process (A, E). Mark and enqueue E. Print(A, E). Dequeue C. Process (C, A). Ignore. Process (C, B). Mark and enqueue B. Print (C, B). Process (C, D). Mark and enqueue D. Print (C, D). Process (C, F). Mark and enqueue F. Print (C, F). Dequeue E. Process (E, A). Ignore. Process (E, F). Ignore. Dequeue B. Process (B, C). Ignore. Process (B, F). Ignore. Dequeue D. Process (D, C). Ignore. Process (D, F). Ignore. Dequeue F. Process (F, B). Ignore. Process (F, C). Ignore. Process (F, D). Ignore. BFS is complete. A EBDF DF CE BDF F Figure 11.12 A detailed illustration of the BFS process for the graph of Fig- ure 11.11(a) starting at Vertex A. The steps leading to each change in the queue are described. 398 Chap. 11 Graphs void topsort(Graph* G) { // Topological sort: recursive int i; for (i=0; in(); i++) // Initialize Mark array G->setMark(i, UNVISITED); for (i=0; in(); i++) // Process all vertices if (G->getMark(i) == UNVISITED) tophelp(G, i); // Call recursive helper function } void tophelp(Graph* G, int v) { // Process vertex v G->setMark(v, VISITED); for (int w=G->first(v); wn(); w = G->next(v,w)) if (G->getMark(w) == UNVISITED) tophelp(G, w); printout(v); // PostVisit for Vertex v } Figure 11.13 Implementation for the recursive topological sort. J1 J2 J3 J4 J5 J7 J6 Figure 11.14 An example graph for topological sort. Seven tasks have depen- dencies as shown by the directed graph. Using this algorithm starting at J1 and visiting adjacent neighbors in alphabetic order, vertices of the graph in Figure 11.14 are printed out in the order J7, J5, J4, J6, J2, J3, J1. Reversing this yields the topological sort J1, J3, J2, J6, J4, J5, J7. We can implement topological sort using a queue instead of recursion, as fol- lows. First visit all edges, counting the number of edges that lead to each vertex (i.e., count the number of prerequisites for each vertex). All vertices with no pre- requisites are placed on the queue. We then begin processing the queue. When Vertex V is taken off of the queue, it is printed, and all neighbors of V (that is, all vertices that have V as a prerequisite) have their counts decremented by one. Place on the queue any neighbor whose count becomes zero. If the queue becomes empty without printing all of the vertices, then the graph contains a cycle (i.e., there is no possible ordering for the tasks that does not violate some prerequisite). The printed order for the vertices of the graph in Figure 11.14 using the queue version of topo- logical sort is J1, J2, J3, J6, J4, J5, J7. Figure 11.15 shows an implementation for the algorithm. Sec. 11.4 Shortest-Paths Problems 399 // Topological sort: Queue void topsort(Graph* G, Queue* Q) { int Count[G->n()]; int v, w; for (v=0; vn(); v++) Count[v] = 0; // Initialize for (v=0; vn(); v++) // Process every edge for (w=G->first(v); wn(); w = G->next(v,w)) Count[w]++; // Add to v2’s prereq count for (v=0; vn(); v++) // Initialize queue if (Count[v] == 0) // Vertex has no prerequisites Q->enqueue(v); while (Q->length() != 0) { // Process the vertices v = Q->dequeue(); printout(v); // PreVisit for "v" for (w=G->first(v); wn(); w = G->next(v,w)) { Count[w]--; // One less prerequisite if (Count[w] == 0) // This vertex is now free Q->enqueue(w); } } } Figure 11.15 A queue-based topological sort algorithm. 11.4 Shortest-Paths Problems On a road map, a road connecting two towns is typically labeled with its distance. We can model a road network as a directed graph whose edges are labeled with real numbers. These numbers represent the distance (or other cost metric, such as travel time) between two vertices. These labels may be called weights, costs, or distances, depending on the application. Given such a graph, a typical problem is to find the total length of the shortest path between two specified vertices. This is not a trivial problem, because the shortest path may not be along the edge (if any) connecting two vertices, but rather may be along a path involving one or more intermediate vertices. For example, in Figure 11.16, the cost of the path from A to B to D is 15. The cost of the edge directly from A to D is 20. The cost of the path from A to C to B to D is 10. Thus, the shortest path from A to D is 10 (not along the edge connecting A to D). We use the notation d(A, D) = 10 to indicate that the shortest distance from A to D is 10. In Figure 11.16, there is no path from E to B, so we set d(E, B) = 1. We define w(A, D) = 20 to be the weight of edge (A, D), that is, the weight of the direct connection from A to D. Because there is no edge from E to B, w(E, B) = 1. Note that w(D, A) = 1 because the graph of Figure 11.16 is directed. We assume that all weights are positive. 400 Chap. 11 Graphs 5 20 2 10 D B A 3 11 EC 15 Figure 11.16 Example graph for shortest-path definitions. 11.4.1 Single-Source Shortest Paths This section presents an algorithm to solve the single-source shortest-paths prob- lem. Given Vertex S in Graph G, find a shortest path from S to every other vertex in G. We might want only the shortest path between two vertices, S and T. How- ever in the worst case, while finding the shortest path from S to T, we might find the shortest paths from S to every other vertex as well. So there is no better alg- orithm (in the worst case) for finding the shortest path to a single vertex than to find shortest paths to all vertices. The algorithm described here will only compute the distance to every such vertex, rather than recording the actual path. Recording the path requires modifications to the algorithm that are left as an exercise. Computer networks provide an application for the single-source shortest-paths problem. The goal is to find the cheapest way for one computer to broadcast a message to all other computers on the network. The network can be modeled by a graph with edge weights indicating time or cost to send a message to a neighboring computer. For unweighted graphs (or whenever all edges have the same cost), the single- source shortest paths can be found using a simple breadth-first search. When weights are added, BFS will not give the correct answer. One approach to solving this problem when the edges have differing weights might be to process the vertices in a fixed order. Label the vertices v0 to vn1, with S = v0. When processing Vertex v1, we take the edge connecting v0 and v1. When processing v2, we consider the shortest distance from v0 to v2 and compare that to the shortest distance from v0 to v1 to v2. When processing Vertex vi, we consider the shortest path for Vertices v0 through vi1 that have already been processed. Unfortunately, the true shortest path to vi might go through Vertex vj for j>i. Such a path will not be considered by this algorithm. However, the problem would not occur if we process the vertices in order of distance from S. Assume that we have processed in order of distance from S to the first i 1 vertices that are closest to S; call this set of vertices S. We are now about to process the ith closest vertex; Sec. 11.4 Shortest-Paths Problems 401 // Compute shortest path distances from "s". // Return these distances in "D". void Dijkstra(Graph* G, int* D, int s) { int i, v, w; for (i=0; in(); i++) { // Process the vertices v = minVertex(G, D); if (D[v] == INFINITY) return; // Unreachable vertices G->setMark(v, VISITED); for (w=G->first(v); wn(); w = G->next(v,w)) if (D[w] > (D[v] + G->weight(v, w))) D[w] = D[v] + G->weight(v, w); } } Figure 11.17 An implementation for Dijkstra’s algorithm. call it X. A shortest path from S to X must have its next-to-last vertex in S. Thus, d(S, X)=min U2S (d(S, U)+w(U, X)). In other words, the shortest path from S to X is the minimum over all paths that go from S to U, then have an edge from U to X, where U is some vertex in S. This solution is usually referred to as Dijkstra’s algorithm. It works by main- taining a distance estimate D(X) for all vertices X in V. The elements of D are ini- tialized to the value INFINITE. Vertices are processed in order of distance from S. Whenever a vertex V is processed, D(X) is updated for every neighbor X of V. Figure 11.17 shows an implementation for Dijkstra’s algorithm. At the end, array D will contain the shortest distance values. There are two reasonable solutions to the key issue of finding the unvisited vertex with minimum distance value during each pass through the main for loop. The first method is simply to scan through the list of |V| vertices searching for the minimum value, as follows: int minVertex(Graph* G, int* D) { // Find min cost vertex int i, v = -1; // Initialize v to some unvisited vertex for (i=0; in(); i++) if (G->getMark(i) == UNVISITED) { v = i; break; } for (i++; in(); i++) // Now find smallest D value if ((G->getMark(i) == UNVISITED) && (D[i] < D[v])) v = i; return v; } Because this scan is done |V| times, and because each edge requires a constant- time update to D, the total cost for this approach is ⇥(|V| 2 + |E|)=⇥(|V| 2), because |E| is in O(|V| 2). The second method is to store unprocessed vertices in a min-heap ordered by distance values. The next-closest vertex can be found in the heap in ⇥(log |V|) 402 Chap. 11 Graphs time. Every time we modify D(X), we could reorder X in the heap by deleting and reinserting it. This is an example of a priority queue with priority update, as described in Section 5.5. To implement true priority updating, we would need to store with each vertex its array index within the heap. A simpler approach is to add the new (smaller) distance value for a given vertex as a new record in the heap. The smallest value for a given vertex currently in the heap will be found first, and greater distance values found later will be ignored because the vertex will already be marked as VISITED. The only disadvantage to repeatedly inserting distance values is that it will raise the number of elements in the heap from ⇥(|V|) to ⇥(|E|) in the worst case. The time complexity is ⇥((|V| + |E|) log |E|), because for each edge we must reorder the heap. Because the objects stored on the heap need to know both their vertex number and their distance, we create a simple class for the purpose called DijkElem, as follows. DijkElem is quite similar to the Edge class used by the adjacency list representation. class DijkElem { public: int vertex, distance; DijkElem() { vertex = -1; distance = -1; } DijkElem(int v, int d) { vertex = v; distance = d; } }; Figure 11.18 shows an implementation for Dijkstra’s algorithm using the prior- ity queue. Using MinVertex to scan the vertex list for the minimum value is more ef- ficient when the graph is dense, that is, when |E| approaches |V| 2. Using a prior- ity queue is more efficient when the graph is sparse because its cost is ⇥((|V| + |E|) log |E|). However, when the graph is dense, this cost can become as great as ⇥(|V| 2 log |E|)=⇥(|V | 2 log |V |). Figure 11.19 illustrates Dijkstra’s algorithm. The start vertex is A. All vertices except A have an initial value of 1. After processing Vertex A, its neighbors have their D estimates updated to be the direct distance from A. After processing C (the closest vertex to A), Vertices B and E are updated to reflect the shortest path through C. The remaining vertices are processed in order B, D, and E. 11.5 Minimum-Cost Spanning Trees The minimum-cost spanning tree (MST) problem takes as input a connected, undirected graph G, where each edge has a distance or weight measure attached. The MST is the graph containing the vertices of G along with the subset of G’s edges that (1) has minimum total cost as measured by summing the values for all of the edges in the subset, and (2) keeps the vertices connected. Applications where a solution to this problem is useful include soldering the shortest set of wires needed Sec. 11.5 Minimum-Cost Spanning Trees 403 // Dijkstra’s shortest paths algorithm with priority queue void Dijkstra(Graph* G, int* D, int s) { int i, v, w; // v is current vertex DijkElem temp; DijkElem E[G->e()]; // Heap array with lots of space temp.distance = 0; temp.vertex = s; E[0] = temp; // Initialize heap array heap H(E, 1, G->e()); // Create heap for (i=0; in(); i++) { // Now, get distances do { if (H.size() == 0) return; // Nothing to remove temp = H.removefirst(); v = temp.vertex; } while (G->getMark(v) == VISITED); G->setMark(v, VISITED); if (D[v] == INFINITY) return; // Unreachable vertices for (w=G->first(v); wn(); w = G->next(v,w)) if (D[w] > (D[v] + G->weight(v, w))) { // Update D D[w] = D[v] + G->weight(v, w); temp.distance = D[w]; temp.vertex = w; H.insert(temp); // Insert new distance in heap } } } Figure 11.18 An implementation for Dijkstra’s algorithm using a priority queue. A B C D E Initial 0 1 1 1 1Process A 0 10 3 20 1Process C 0 5 3 20 18 Process B 0 5 3 10 18 Process D 0 5 3 10 18 Process E 0 5 3 10 18 Figure 11.19 A listing for the progress of Dijkstra’s algorithm operating on the graph of Figure 11.16. The start vertex is A. to connect a set of terminals on a circuit board, and connecting a set of cities by telephone lines in such a way as to require the least amount of cable. The MST contains no cycles. If a proposed MST did have a cycle, a cheaper MST could be had by removing any one of the edges in the cycle. Thus, the MST is a free tree with |V|1 edges. The name “minimum-cost spanning tree” comes from the fact that the required set of edges forms a tree, it spans the vertices (i.e., it connects them together), and it has minimum cost. Figure 11.20 shows the MST for an example graph. 404 Chap. 11 Graphs A 9 7 5 B C 1 2 6 D 2 1E F Figure 11.20 A graph and its MST. All edges appear in the original graph. Those edges drawn with heavy lines indicate the subset making up the MST. Note that edge (C, F) could be replaced with edge (D, F) to form a different MST with equal cost. 11.5.1 Prim’s Algorithm The first of our two algorithms for finding MSTs is commonly referred to as Prim’s algorithm. Prim’s algorithm is very simple. Start with any Vertex N in the graph, setting the MST to be N initially. Pick the least-cost edge connected to N. This edge connects N to another vertex; call this M. Add Vertex M and Edge (N, M) to the MST. Next, pick the least-cost edge coming from either N or M to any other vertex in the graph. Add this edge and the new vertex it reaches to the MST. This process continues, at each step expanding the MST by selecting the least-cost edge from a vertex currently in the MST to a vertex not currently in the MST. Prim’s algorithm is quite similar to Dijkstra’s algorithm for finding the single- source shortest paths. The primary difference is that we are seeking not the next closest vertex to the start vertex, but rather the next closest vertex to any vertex currently in the MST. Thus we replace the lines if (D[w] > (D[v] + G->weight(v, w))) D[w] = D[v] + G->weight(v, w); in Djikstra’s algorithm with the lines if (D[w] > G->weight(v, w)) D[w] = G->weight(v, w); in Prim’s algorithm. Figure 11.21 shows an implementation for Prim’s algorithm that searches the distance matrix for the next closest vertex. For each vertex I, when I is processed by Prim’s algorithm, an edge going to I is added to the MST that we are building. Array V[I] stores the previously visited vertex that is closest to Vertex I. This information lets us know which edge goes into the MST when Vertex I is processed. Sec. 11.5 Minimum-Cost Spanning Trees 405 void Prim(Graph* G, int* D, int s) { // Prim’s MST algorithm int V[G->n()]; // Store closest vertex int i, w; for (i=0; in(); i++) { // Process the vertices int v = minVertex(G, D); G->setMark(v, VISITED); if (v != s) AddEdgetoMST(V[v], v); // Add edge to MST if (D[v] == INFINITY) return; // Unreachable vertices for (w=G->first(v); wn(); w = G->next(v,w)) if (D[w] > G->weight(v,w)) { D[w] = G->weight(v,w); // Update distance V[w] = v; // Where it came from } } } Figure 11.21 An implementation for Prim’s algorithm. The implementation of Figure 11.21 also contains calls to AddEdgetoMST to indicate which edges are actually added to the MST. Alternatively, we can implement Prim’s algorithm using a priority queue to find the next closest vertex, as shown in Figure 11.22. As with the priority queue version of Dijkstra’s algorithm, the heap’s Elem type stores a DijkElem object. Prim’s algorithm is an example of a greedy algorithm. At each step in the for loop, we select the least-cost edge that connects some marked vertex to some unmarked vertex. The algorithm does not otherwise check that the MST really should include this least-cost edge. This leads to an important question: Does Prim’s algorithm work correctly? Clearly it generates a spanning tree (because each pass through the for loop adds one edge and one unmarked vertex to the spanning tree until all vertices have been added), but does this tree have minimum cost? Theorem 11.1 Prim’s algorithm produces a minimum-cost spanning tree. Proof: We will use a proof by contradiction. Let G =(V, E) be a graph for which Prim’s algorithm does not generate an MST. Define an ordering on the vertices according to the order in which they were added by Prim’s algorithm to the MST: v0, v1,...,vn1. Let edge ei connect (vx, vi) for some xn()]; // V[I] stores I’s closest neighbor DijkElem temp; DijkElem E[G->e()]; // Heap array with lots of space temp.distance = 0; temp.vertex = s; E[0] = temp; // Initialize heap array heap H(E, 1, G->e()); // Create heap for (i=0; in(); i++) { // Now build MST do { if(H.size() == 0) return; // Nothing to remove temp = H.removefirst(); v = temp.vertex; } while (G->getMark(v) == VISITED); G->setMark(v, VISITED); if (v != s) AddEdgetoMST(V[v], v); // Add edge to MST if (D[v] == INFINITY) return; // Ureachable vertex for (w=G->first(v); wn(); w = G->next(v,w)) if (D[w] > G->weight(v, w)) { // Update D D[w] = G->weight(v, w); V[w] = v; // Update who it came from temp.distance = D[w]; temp.vertex = w; H.insert(temp); // Insert new distance in heap } } } Figure 11.22 An implementation of Prim’s algorithm using a priority queue. be of lower cost than edge ej, because Prim’s algorithm did not generate an MST. This situation is illustrated in Figure 11.23. However, Prim’s algorithm would have selected the least-cost edge available. It would have selected e0, not ej. Thus, it is a contradiction that Prim’s algorithm would have selected the wrong edge, and thus, Prim’s algorithm must be correct. 2 Example 11.3 For the graph of Figure 11.20, assume that we begin by marking Vertex A. From A, the least-cost edge leads to Vertex C. Vertex C and edge (A, C) are added to the MST. At this point, our candidate edges connecting the MST (Vertices A and C) with the rest of the graph are (A, E), (C, B), (C, D), and (C, F). From these choices, the least-cost edge from the MST is (C, D). So we add Vertex D to the MST. For the next iteration, our edge choices are (A, E), (C, B), (C, F), and (D, F). Because edges (C, F) and (D, F) happen to have equal cost, it is an arbitrary decision as to which gets selected. Say we pick (C, F). The next step marks Vertex E and adds edge (F, E) to the MST. Following in this manner, Vertex B (through edge (C, B)) is marked. At this point, the algorithm terminates. Sec. 11.5 Minimum-Cost Spanning Trees 407 j i u p i u j Marked Unmarked ’’correct’’ edge e’ Prim’s edge v vv v e Vertices v , i < j Vertices v , i >= j Figure 11.23 Prim’s MST algorithm proof. The left oval contains that portion of the graph where Prim’s MST and the “true” MST T agree. The right oval contains the rest of the graph. The two portions of the graph are connected by (at least) edges ej (selected by Prim’s algorithm to be in the MST) and e0 (the “correct” edge to be placed in the MST). Note that the path from vw to vj cannot include any marked vertex vi, i  j, because to do so would form a cycle. 11.5.2 Kruskal’s Algorithm Our next MST algorithm is commonly referred to as Kruskal’s algorithm. Kruskal’s algorithm is also a simple, greedy algorithm. First partition the set of vertices into |V| equivalence classes (see Section 6.2), each consisting of one vertex. Then pro- cess the edges in order of weight. An edge is added to the MST, and two equiva- lence classes combined, if the edge connects two vertices in different equivalence classes. This process is repeated until only one equivalence class remains. Example 11.4 Figure 11.24 shows the first three steps of Kruskal’s Alg- orithm for the graph of Figure 11.20. Edge (C, D) has the least cost, and because C and D are currently in separate MSTs, they are combined. We next select edge (E, F) to process, and combine these vertices into a single MST. The third edge we process is (C, F), which causes the MST contain- ing Vertices C and D to merge with MST containing Vertices E and F. The next edge to process is (D, F). But because Vertices D and F are currently in the same MST, this edge is rejected. The algorithm will continue on to accept edges (B, C) and (A, C) into the MST. The edges can be processed in order of weight by using a min-heap. This is generally faster than sorting the edges first, because in practice we need only visit a small fraction of the edges before completing the MST. This is an example of finding only a few smallest elements in a list, as discussed in Section 7.6. 408 Chap. 11 Graphs Initial Step 1 A B C 1 D EF Step 2 Process edge (E, F) 1 1 Step 3 Process edge (C, F) B 1 2 E 1 F Process edge (C, D) A AB DEFC C D B C D EA F Figure 11.24 Illustration of the first three steps of Kruskal’s MST algorithm as applied to the graph of Figure 11.20. The only tricky part to this algorithm is determining if two vertices belong to the same equivalence class. Fortunately, the ideal algorithm is available for the purpose — the UNION/FIND algorithm based on the parent pointer representation for trees described in Section 6.2. Figure 11.25 shows an implementation for the algorithm. Class KruskalElem is used to store the edges on the min-heap. Kruskal’s algorithm is dominated by the time required to process the edges. The differ and UNION functions are nearly constant in time if path compression and weighted union is used. Thus, the total cost of the algorithm is ⇥(|E| log |E|) in the worst case, when nearly all edges must be processed before all the edges of the spanning tree are found and the algorithm can stop. More often the edges of the spanning tree are the shorter ones,and only about |V| edges must be processed. If so, the cost is often close to ⇥(|V| log |E|) in the average case. Sec. 11.6 Further Reading 409 class KruskElem { // An element for the heap public: int from, to, distance; // The edge being stored KruskElem() { from = -1; to = -1; distance = -1; } KruskElem(int f, int t, int d) { from = f; to = t; distance = d; } }; void Kruskel(Graph* G) { // Kruskal’s MST algorithm ParPtrTree A(G->n()); // Equivalence class array KruskElem E[G->e()]; // Array of edges for min-heap int i; int edgecnt = 0; for (i=0; in(); i++) // Put the edges on the array for (int w=G->first(i); wn(); w = G->next(i,w)) { E[edgecnt].distance = G->weight(i, w); E[edgecnt].from = i; E[edgecnt++].to = w; } // Heapify the edges heap H(E, edgecnt, edgecnt); int numMST = G->n(); // Initially n equiv classes for (i=0; numMST>1; i++) { // Combine equiv classes KruskElem temp; temp = H.removefirst(); // Get next cheapest edge int v = temp.from; int u =; if (A.differ(v, u)) { // If in different equiv classes A.UNION(v, u); // Combine equiv classes AddEdgetoMST(temp.from,; // Add edge to MST numMST--; // One less MST } } } Figure 11.25 An implementation for Kruskal’s algorithm. 11.6 Further Reading Many interesting properties of graphs can be investigated by playing with the pro- grams in the Stanford Graphbase. This is a collection of benchmark databases and graph processing programs. The Stanford Graphbase is documented in [Knu94]. 11.7 Exercises 11.1 Prove by induction that a graph with n vertices has at most n(n1)/2 edges. 11.2 Prove the following implications regarding free trees. (a) IF an undirected graph is connected and has no simple cycles, THEN the graph has |V|1 edges. (b) IF an undirected graph has |V|1 edges and no cycles, THEN the graph is connected. 410 Chap. 11 Graphs 11.3 (a) Draw the adjacency matrix representation for the graph of Figure 11.26. (b) Draw the adjacency list representation for the same graph. (c) If a pointer requires four bytes, a vertex label requires two bytes, and an edge weight requires two bytes, which representation requires more space for this graph? (d) If a pointer requires four bytes, a vertex label requires one byte, and an edge weight requires two bytes, which representation requires more space for this graph? 11.4 Show the DFS tree for the graph of Figure 11.26, starting at Vertex 1. 11.5 Write a pseudocode algorithm to create a DFS tree for an undirected, con- nected graph starting at a specified vertex V. 11.6 Show the BFS tree for the graph of Figure 11.26, starting at Vertex 1. 11.7 Write a pseudocode algorithm to create a BFS tree for an undirected, con- nected graph starting at a specified vertex V. 11.8 The BFS topological sort algorithm can report the existence of a cycle if one is encountered. Modify this algorithm to print the vertices possibly appearing in cycles (if there are any cycles). 11.9 Explain why, in the worst case, Dijkstra’s algorithm is (asymptotically) as efficient as any algorithm for finding the shortest path from some vertex I to another vertex J. 11.10 Show the shortest paths generated by running Dijkstra’s shortest-paths alg- orithm on the graph of Figure 11.26, beginning at Vertex 4. Show the D values as each vertex is processed, as in Figure 11.19. 11.11 Modify the algorithm for single-source shortest paths to actually store and return the shortest paths rather than just compute the distances. 11.12 The root of a DAG is a vertex R such that every vertex of the DAG can be reached by a directed path from R. Write an algorithm that takes a directed graph as input and determines the root (if there is one) for the graph. The running time of your algorithm should be ⇥(|V| + |E|). 11.13 Write an algorithm to find the longest path in a DAG, where the length of the path is measured by the number of edges that it contains. What is the asymptotic complexity of your algorithm? 11.14 Write an algorithm to determine whether a directed graph of |V| vertices contains a cycle. Your algorithm should run in ⇥(|V| + |E|) time. 11.15 Write an algorithm to determine whether an undirected graph of |V| vertices contains a cycle. Your algorithm should run in ⇥(|V|) time. 11.16 The single-destination shortest-paths problem for a directed graph is to find the shortest path from every vertex to a specified vertex V. Write an algorithm to solve the single-destination shortest-paths problem. 11.17 List the order in which the edges of the graph in Figure 11.26 are visited when running Prim’s MST algorithm starting at Vertex 3. Show the final MST. Sec. 11.8 Projects 411 2 5 420 103 6 11 33 15 5 10 2 1 Figure 11.26 Example graph for Chapter 11 exercises. 11.18 List the order in which the edges of the graph in Figure 11.26 are visited when running Kruskal’s MST algorithm. Each time an edge is added to the MST, show the result on the equivalence array, (e.g., show the array as in Figure 6.7). 11.19 Write an algorithm to find a maximum cost spanning tree, that is, the span- ning tree with highest possible cost. 11.20 When can Prim’s and Kruskal’s algorithms yield different MSTs? 11.21 Prove that, if the costs for the edges of Graph G are distinct, then only one MST exists for G. 11.22 Does either Prim’s or Kruskal’s algorithm work if there are negative edge weights? 11.23 Consider the collection of edges selected by Dijkstra’s algorithm as the short- est paths to the graph’s vertices from the start vertex. Do these edges form a spanning tree (not necessarily of minimum cost)? Do these edges form an MST? Explain why or why not. 11.24 Prove that a tree is a bipartite graph. 11.25 Prove that any tree (i.e., a connected, undirected graph with no cycles) can be two-colored. (A graph can be two colored if every vertex can be assigned one of two colors such that no adjacent vertices have the same color.) 11.26 Write an algorithm that determines if an arbitrary undirected graph is a bipar- tite graph. If the graph is bipartite, then your algorithm should also identify the vertices as to which of the two partitions each belongs to. 11.8 Projects 11.1 Design a format for storing graphs in files. Then implement two functions: one to read a graph from a file and the other to write a graph to a file. Test your functions by implementing a complete MST program that reads an undi- rected graph in from a file, constructs the MST, and then writes to a second file the graph representing the MST. 412 Chap. 11 Graphs 11.2 An undirected graph need not explicitly store two separate directed edges to represent a single undirected edge. An alternative would be to store only a single undirected edge (I, J) to connect Vertices I and J. However, what if the user asks for edge (J, I)? We can solve this problem by consistently storing the edge such that the lesser of I and J always comes first. Thus, if we have an edge connecting Vertices 5 and 3, requests for edge (5, 3) and (3, 5) both map to (3, 5) because 3 < 5. Looking at the adjacency matrix, we notice that only the lower triangle of the array is used. Thus we could cut the space required by the adjacency matrix from |V| 2 positions to |V|(|V|1)/2 positions. Read Section 12.2 on triangular matrices. The re-implement the adjacency matrix representation of Figure 11.6 to implement undirected graphs using a triangular array. 11.3 While the underlying implementation (whether adjacency matrix or adja- cency list) is hidden behind the graph ADT, these two implementations can have an impact on the efficiency of the resulting program. For Dijkstra’s shortest paths algorithm, two different implementations were given in Sec- tion 11.4.1 that provide different ways for determining the next closest vertex at each iteration of the algorithm. The relative costs of these two variants depend on who sparse or dense the graph is. They might also depend on whether the graph is implemented using an adjacency list or adjacency ma- trix. Design and implement a study to compare the effects on performance for three variables: (i) the two graph representations (adjacency list and adja- cency matrix); (ii) the two implementations for Djikstra’s shortest paths alg- orithm (searching the table of vertex distances or using a priority queue to track the distances), and (iii) sparse versus dense graphs. Be sure to test your implementations on a variety of graphs that are sufficiently large to generate meaningful times. 11.4 The example implementations for DFS and BFS show calls to functions PreVisit and PostVisit. Re-implement the BFS and DFS functions to make use of the visitor design pattern to handle the pre/post visit function- ality. 11.5 Write a program to label the connected components for an undirected graph. In other words, all vertices of the first component are given the first com- ponent’s label, all vertices of the second component are given the second component’s label, and so on. Your algorithm should work by defining any two vertices connected by an edge to be members of the same equivalence class. Once all of the edges have been processed, all vertices in a given equiv- alence class will be connected. Use the UNION/FIND implementation from Section 6.2 to implement equivalence classes. 12 Lists and Arrays Revisited Simple lists and arrays are the right tools for the many applications. Other situa- tions require support for operations that cannot be implemented efficiently by the standard list representations of Chapter 4. This chapter presents a range of topics, whose unifying thread is that the data structures included are all list- or array-like. These structures overcome some of the problems of simple linked list and con- tiguous array representations. This chapter also seeks to reinforce the concept of logical representation versus physical implementation, as some of the “list” imple- mentations have quite different organizations internally. Section 12.1 describes a series of representations for multilists, which are lists that may contain sublists. Section 12.2 discusses representations for implementing sparse matrices, large matrices where most of the elements have zero values. Sec- tion 12.3 discusses memory management techniques, which are essentially a way of allocating variable-length sections from a large array. 12.1 Multilists Recall from Chapter 4 that a list is a finite, ordered sequence of items of the form hx0,x1,...,xn1i where n 0. We can represent the empty list by NULL or hi. In Chapter 4 we assumed that all list elements had the same data type. In this section, we extend the definition of lists to allow elements to be arbitrary in nature. In general, list elements are one of two types. 1. An atom, which is a data record of some type such as a number, symbol, or string. 2. Another list, which is called a sublist. A list containing sublists will be written as hx1, hy1, ha1, a2i, y3i, hz1, z2i, x4i. 413 414 Chap. 12 Lists and Arrays Revisited x1 y1 a1 a2 y3 z1 z2 x4 Figure 12.1 Example of a multilist represented by a tree. L2 L1 ab cde L3 Figure 12.2 Example of a reentrant multilist. The shape of the structure is a DAG (all edges point downward). In this example, the list has four elements. The second element is the sublist hy1, ha1, a2i, y3i and the third is the sublist hz1, z2i. The sublist hy1, ha1, a2i, y3iitself contains a sublist. If a list L has one or more sublists, we call L a multi- list. Lists with no sublists are often referred to as linear lists or chains. Note that this definition for multilist fits well with our definition of sets from Definition 2.1, where a set’s members can be either primitive elements or sets. We can restrict the sublists of a multilist in various ways, depending on whether the multilist should have the form of a tree, a DAG, or a generic graph. A pure list is a list structure whose graph corresponds to a tree, such as in Figure 12.1. In other words, there is exactly one path from the root to any node, which is equivalent to saying that no object may appear more than once in the list. In the pure list, each pair of angle brackets corresponds to an internal node of the tree. The members of the list correspond to the children for the node. Atoms on the list correspond to leaf nodes. A reentrant list is a list structure whose graph corresponds to a DAG. Nodes might be accessible from the root by more than one path, which is equivalent to saying that objects (including sublists) may appear multiple times in the list as long as no cycles are formed. All edges point downward, from the node representing a list or sublist to its elements. Figure 12.2 illustrates a reentrant list. To write out this list in bracket notation, we can duplicate nodes as necessary. Thus, the bracket notation for the list of Figure 12.2 could be written hhha, bii, hha, bi, ci, hc, d, ei, heii. For convenience, we will adopt a convention of allowing sublists and atoms to be labeled, such as “L1:”. Whenever a label is repeated, the element corresponding to Sec. 12.1 Multilists 415 L1 L2 L4 bd a c L3 Figure 12.3 Example of a cyclic list. The shape of the structure is a directed graph. that label will be substituted when we write out the list. Thus, the bracket notation for the list of Figure 12.2 could be written hhL1:ha, bii, hL1, L2:ci, hL2, d, L3:ei, hL3ii. A cyclic list is a list structure whose graph corresponds to any directed graph, possibly containing cycles. Figure 12.3 illustrates such a list. Labels are required to write this in bracket notation. Here is the bracket notation for the list of Figure 12.3. hL1:hL2:ha, L1ii, hL2, L3:bi, hL3, c, di, L4:hL4ii. Multilists can be implemented in a number of ways. Most of these should be familiar from implementations suggested earlier in the book for list, tree, and graph data structures. One simple approach is to use a simple array to represent the list. This works well for chains with fixed-length elements, equivalent to the simple array-based list of Chapter 4. We can view nested sublists as variable-length elements. To use this approach, we require some indication of the beginning and end of each sublist. In essence, we are using a sequential tree implementation as discussed in Section 6.5. This should be no surprise, because the pure list is equivalent to a general tree structure. Unfortunately, as with any sequential representation, access to the nth sublist must be done sequentially from the beginning of the list. Because pure lists are equivalent to trees, we can also use linked allocation methods to support direct access to the list of children. Simple linear lists are represented by linked lists. Pure lists can be represented as linked lists with an additional tag field to indicate whether the node is an atom or a sublist. If it is a sublist, the data field points to the first element on the sublist. This is illustrated by Figure 12.4. Another approach is to represent all list elements with link nodes storing two pointer fields, except for atoms. Atoms just contain data. This is the system used by the programming language LISP. Figure 12.5 illustrates this representation. Either the pointer contains a tag bit to identify what it points to, or the object being pointed to stores a tag bit to identify itself. Tags distinguish atoms from list nodes. This 416 Chap. 12 Lists and Arrays Revisited root y1 − + y3 + a2 + z1 x4 z2 + a1 −x1 + + + + − Figure 12.4 Linked representation for the pure list of Figure 12.1. The first field in each link node stores a tag bit. If the tag bit stores “+,” then the data field stores an atom. If the tag bit stores “,” then the data field stores a pointer to a sublist. root BCD A Figure 12.5 LISP-like linked representation for the cyclic multilist of Fig- ure 12.3. Each link node stores two pointers. A pointer either points to an atom, or to another link node. Link nodes are represented by two boxes, and atoms by circles. implementation can easily support reentrant and cyclic lists, because non-atoms can point to any other node. 12.2 Matrix Representations Sometimes we need to represent a large, two-dimensional matrix where many of the elements have a value of zero. One example is the lower triangular matrix that results from solving systems of simultaneous equations. A lower triangular matrix stores zero values at all positions [r, c] such that rc, as shown in Figure 12.6(b). For an n ⇥ n upper triangular matrix, the equation to convert from matrix coordinates to list positions would be matrix[r, c]=list[rn (r2 + r)/2+c]. A more difficult situation arises when the vast majority of values stored in an n ⇥ m matrix are zero, but there is no restriction on which positions are zero and which are non-zero. This is known as a sparse matrix. One approach to representing a sparse matrix is to concatenate (or otherwise combine) the row and column coordinates into a single value and use this as a key in a hash table. Thus, if we want to know the value of a particular position in the matrix, we search the hash table for the appropriate key. If a value for this position is not found, it is assumed to be zero. This is an ideal approach when all queries to the matrix are in terms of access by specified position. However, if we wish to find the first non-zero element in a given row, or the next non-zero element below the current one in a given column, then the hash table requires us to check sequentially through all possible positions in some row or column. 418 Chap. 12 Lists and Arrays Revisited Another approach is to implement the matrix as an orthogonal list. Consider the following sparse matrix: 10 23 0 0 0 0 19 45 5 0 93 0 0 0 0000000 0000000 40 0 0 0 0 0 0 0000000 0000000 0 32 0 12 0 0 7 The corresponding orthogonal array is shown in Figure 12.7. Here we have a list of row headers, each of which contains a pointer to a list of matrix records. A second list of column headers also contains pointers to matrix records. Each non-zero matrix element stores pointers to its non-zero neighbors in the row, both following and preceding it. Each non-zero element also stores pointers to its non- zero neighbors following and preceding it in the column. Thus, each non-zero element stores its own value, its position within the matrix, and four pointers. Non- zero elements are found by traversing a row or column list. Note that the first non-zero element in a given row could be in any column; likewise, the neighboring non-zero element in any row or column list could be at any (higher) row or column in the array. Thus, each non-zero element must also store its row and column position explicitly. To find if a particular position in the matrix contains a non-zero element, we traverse the appropriate row or column list. For example, when looking for the element at Row 7 and Column 1, we can traverse the list either for Row 7 or for Column 1. When traversing a row or column list, if we come to an element with the correct position, then its value is non-zero. If we encounter an element with a higher position, then the element we are looking for is not in the sparse matrix. In this case, the element’s value is zero. For example, when traversing the list for Row 7 in the matrix of Figure 12.7, we first reach the element at Row 7 and Column 1. If this is what we are looking for, then the search can stop. If we are looking for the element at Row 7 and Column 2, then the search proceeds along the Row 7 list to next reach the element at Column 3. At this point we know that no element at Row 7 and Column 2 is stored in the sparse matrix. Insertion and deletion can be performed by working in a similar way to insert or delete elements within the appropriate row and column lists. Each non-zero element stored in the sparse matrix representation takes much more space than an element stored in a simple n ⇥ n matrix. When is the sparse matrix more space efficient than the standard representation? To calculate this, we need to determine how much space the standard matrix requires, and how much Sec. 12.2 Matrix Representations 419 0,0 0,1 0,6 0,3 1,1 1,3 0,4 7,1 7,3 7,6 0 1 4 7 0136Cols Rows 10 1923 45 5 93 40 32 12 7 Figure 12.7 The orthogonal list sparse matrix representation. the sparse matrix requires. The size of the sparse matrix depends on the number of non-zero elements (we will refer to this value as NNZ), while the size of the standard matrix representation does not vary. We need to know the (relative) sizes of a pointer and a data value. For simplicity, our calculation will ignore the space taken up by the row and column header (which is not much affected by the number of elements in the sparse array). As an example, assume that a data value, a row or column index, and a pointer each require four bytes. An n ⇥ m matrix requires 4nm bytes. The sparse matrix requires 28 bytes per non-zero element (four pointers, two array indices, and one data value). If we set X to be the percentage of non-zero elements, we can solve for the value of X below which the sparse matrix representation is more space efficient. Using the equation 28X =4mn and solving for X, we find that the sparse matrix using this implementation is more space efficient when X<1/7, that is, when less than about 14% of the elements 420 Chap. 12 Lists and Arrays Revisited are non-zero. Different values for the relative sizes of data values, pointers, or matrix indices can lead to a different break-even point for the two implementations. The time required to process a sparse matrix should ideally depend on NNZ. When searching for an element, the cost is the number of elements preceding the desired element on its row or column list. The cost for operations such as adding two matrices should be ⇥(n + m) in the worst case when the one matrix stores n non-zero elements and the other stores m non-zero elements. Another representation for sparse matrices is sometimes called the Yale rep- resentation. Matlab uses a similar representation, with a primary difference being that the Matlab representation uses column-major order.1 The Matlab representa- tion stores the sparse matrix using three lists. The first is simply all of the non-zero element values, in column-major order. The second list stores the start position within the first list for each column. The third list stores the row positions for each of the corresponding non-zero values. In the Yale representation, the matrix of Figure 12.7 would appear as: Values: 10 45 40 23 5 32 93 12 19 7 Column starts: 0 3 5 5 7 7 7 7 Row positions: 0 1 4 0 1 7 1 7 0 7 If the matrix has c columns, then the total space required will be proportional to c +2NNZ. This is good in terms of space. It allows fairly quick access to any column, and allows for easy processing of the non-zero values along a column. However, it does not do a good job of providing access to the values along a row, and is terrible when values need to be added or removed from the representation. Fortunately, when doing computations such as adding or multiplying two sparse matrices, the processing of the input matrices and construction of the output matrix can be done reasonably efficiently. 12.3 Memory Management Most data structures are designed to store and access objects of uniform size. A typical example would be an integer stored in a list or a queue. Some applications require the ability to store variable-length records, such as a string of arbitrary length. One solution is to store in the list or queue fixed-length pointers to the variable-length strings. This is fine for data structures stored in main memory. But if the collection of strings is meant to be stored on disk, then we might need to worry about where exactly these strings are stored. And even when stored in main memory, something has to figure out where there are available bytes to hold the string. We could easily store variable-size records in a queue or stack, where 1Scientific packages tend to prefer column-oriented representations for matrices since this the dominant access need for the operations to be performed. Sec. 12.3 Memory Management 421 // Memory Manager abstract class class MemManager { public: virtual ˜MemManager() {} // Base destructor // Store a record and return a handle to it virtual MemHandle insert(void* info, int length) =0; // Get back a copy of a stored record virtual int get(void* info, MemHandle h) =0; // Release the space associated with a record virtual void release(MemHandle h) =0; }; Figure 12.8 A simple ADT for a memory manager. the restricted order of insertions and deletions makes this easy to deal with. But in a language like C++ or Java, programmers can allocate and deallocate space in complex ways through use of new. Where does this space come from? This section discusses memory management techniques for the general problem of handling space requests of variable size. The basic model for memory management is that we have a (large) block of contiguous memory locations, which we will call the memory pool. Periodically, memory requests are issued for some amount of space in the pool. The memory manager has the job of finding a contiguous block of locations of at least the re- quested size from somewhere within the memory pool. Honoring such a request is called a memory allocation. The memory manager will typically return some piece of information that the requester can hold on to so that later it can recover the record that was just stored by the memory manager. This piece of information is called a handle. At some point, space that has been requested might no longer be needed, and this space can be returned to the memory manager so that it can be reused. This is called a memory deallocation. The memory manager should then be able to reuse this space to satisfy later memory requests. We can define an ADT for the memory manager as shown in Figure 12.8. The user of the MemManager ADT provides a pointer (in parameter info) to space that holds some record or message to be stored or retrieved. This is similar to the C++ basic file read/write methods presented in Section 8.4. The fundamental idea is that the client gives messages to the memory manager for safe keeping. The memory manager returns a “receipt” for the message in the form of a MemHandle object. Of course to be practical, a MemHandle must be much smaller than the typical message to be stored. The client holds the MemHandle object until it wishes to get the message back. Method insert lets the client tell the memory manager the length and con- tents of the message to be stored. This ADT assumes that the memory manager will 422 Chap. 12 Lists and Arrays Revisited Figure 12.9 Dynamic storage allocation model. Memory is made up of a series of variable-size blocks, some allocated and some free. In this example, shaded areas represent memory currently allocated and unshaded areas represent unused memory available for future allocation. remember the length of the message associated with a given handle (perhaps in the handle itself), thus method get does not include a length parameter but instead returns the length of the message actually stored. Method release allows the client to tell the memory manager to release the space that stores a given message. When all inserts and releases follow a simple pattern, such as last requested, first released (stack order), or first requested, first released (queue order), memory management is fairly easy. We are concerned here with the general case where blocks of any size might be requested and released in any order. This is known as dynamic storage allocation. One example of dynamic storage allocation is managing free store for a compiler’s runtime environment, such as the system-level new and delete operations in C++. Another example is managing main memory in a multitasking operating system. Here, a program might require a certain amount of space, and the memory manager must keep track of which programs are using which parts of the main memory. Yet another example is the file manager for a disk drive. When a disk file is created, expanded, or deleted, the file manager must allocate or deallocate disk space. A block of memory or disk space managed in this way is sometimes referred to as a heap. The term “heap” is being used here in a different way than the heap data structure discussed in Section 5.5. Here “heap” refers to the memory controlled by a dynamic memory management scheme. In the rest of this section, we first study techniques for dynamic memory man- agement. We then tackle the issue of what to do when no single block of memory in the memory pool is large enough to honor a given request. 12.3.1 Dynamic Storage Allocation For the purpose of dynamic storage allocation, we view memory as a single array which, after a series of memory requests and releases tends to become broken into a series of variable-size blocks, where some of the blocks are free and some are reserved or already allocated to store messages. The memory manager typically uses a linked list to keep track of the free blocks, called the freelist, which is used for servicing future memory requests. Figure 12.9 illustrates the situation that can arise after a series of memory allocations and deallocations. Sec. 12.3 Memory Management 423 Small block: External fragmentation Unused space in allocated block: Internal fragmentation Figure 12.10 An illustration of internal and external fragmentation. The small white block labeled ”External fragmentation” is too small to satisfy typical mem- ory requests. The small grey block labeled ”Internal fragmentation” was allocated as part of the grey block to its left, but it does not actually store information. When a memory request is received by the memory manager, some block on the freelist must be found that is large enough to service the request. If no such block is found, then the memory manager must resort to a failure policy such as discussed in Section 12.3.2. If there is a request for m words, and no block exists of exactly size m, then a larger block must be used instead. One possibility in this case is that the entire block is given away to the memory allocation request. This might be desirable when the size of the block is only slightly larger than the request. This is because saving a tiny block that is too small to be useful for a future memory request might not be worthwhile. Alternatively, for a free block of size k, with k>m, up to k m space may be retained by the memory manager to form a new free block, while the rest is used to service the request. Memory managers can suffer from two types of fragmentation, which refers to unused space that is too small to be useful. External fragmentation occurs when a series of memory requests and releases results in small free blocks. Internal fragmentation occurs when more than m words are allocated to a request for m words, wasting free storage. This is equivalent to the internal fragmentation that occurs when files are allocated in multiples of the cluster size. The difference between internal and external fragmentation is illustrated by Figure 12.10. Some memory management schemes sacrifice space to internal fragmentation to make memory management easier (and perhaps reduce external fragmentation). For example, external fragmentation does not happen in file management systems that allocate file space in clusters. Another example of sacrificing space to inter- nal fragmentation so as to simplify memory management is the buddy method described later in this section. The process of searching the memory pool for a block large enough to service the request, possibly reserving the remaining space as a free block, is referred to as a sequential fit method. 424 Chap. 12 Lists and Arrays Revisited Figure 12.11 A doubly linked list of free blocks as seen by the memory manager. Shaded areas represent allocated memory. Unshaded areas are part of the freelist. Sequential Fit Methods Sequential-fit methods attempt to find a “good” block to service a storage request. The three sequential-fit methods described here assume that the free blocks are organized into a doubly linked list, as illustrated by Figure 12.11. There are two basic approaches to implementing the freelist. The simpler ap- proach is to store the freelist separately from the memory pool. In other words, a simple linked-list implementation such as described in Chapter 4 can be used, where each node of the linked list contains a pointer to a single free block in the memory pool. This is fine if there is space available for the linked list itself, sepa- rate from the memory pool. The second approach to storing the freelist is more complicated but saves space. Because the free space is free, it can be used by the memory manager to help it do its job. That is, the memory manager can temporarily “borrow” space within the free blocks to maintain its doubly linked list. To do so, each unallocated block must be large enough to hold these pointers. In addition, it is usually worthwhile to let the memory manager add a few bytes of space to each reserved block for its own purposes. In other words, a request for m bytes of space might result in slightly more than m bytes being allocated by the memory manager, with the extra bytes used by the memory manager itself rather than the requester. We will assume that all memory blocks are organized as shown in Figure 12.12, with space for tags and linked list pointers. Here, free and reserved blocks are distinguished by a tag bit at both the beginning and the end of the block, for reasons that will be explained. In addition, both free and reserved blocks have a size indicator immediately after the tag bit at the beginning of the block to indicate how large the block is. Free blocks have a second size indicator immediately preceding the tag bit at the end of the block. Finally, free blocks have left and right pointers to their neighbors in the free block list. The information fields associated with each block permit the memory manager to allocate and deallocate blocks as needed. When a request comes in for m words of storage, the memory manager searches the linked list of free blocks until it finds a “suitable” block for allocation. How it determines which block is suitable will be discussed below. If the block contains exactly m words (plus space for the tag and size fields), then it is removed from the freelist. If the block (of size k) is large Sec. 12.3 Memory Management 425 + Tag Llink Size Tag (a) k Size (b) TagSize Rlink + − k − Tag k Figure 12.12 Blocks as seen by the memory manager. Each block includes additional information such as freelist link pointers, start and end tags, and a size field. (a) The layout for a free block. The beginning of the block contains the tag bit field, the block size field, and two pointers for the freelist. The end of the block contains a second tag field and a second block size field. (b) A reserved block of k bytes. The memory manager adds to these k bytes an additional tag bit field and block size field at the beginning of the block, and a second tag field at the end of the block. enough, then the remaining k m words are reserved as a block on the freelist, in the current location. When a block F is freed, it must be merged into the freelist. If we do not care about merging adjacent free blocks, then this is a simple insertion into the doubly linked list of free blocks. However, we would like to merge adjacent blocks, because this allows the memory manager to serve requests of the largest possible size. Merging is easily done due to the tag and size fields stored at the ends of each block, as illustrated by Figure 12.13. Here, the memory manager first checks the unit of memory immediately preceding block F to see if the preceding block (call it P) is also free. If it is, then the memory unit before P’s tag bit stores the size of P, thus indicating the position for the beginning of the block in memory. P can then simply have its size extended to include block F. If block P is not free, then we just add block F to the freelist. Finally, we also check the bit following the end of block F. If this bit indicates that the following block (call it S) is free, then S is removed from the freelist and the size of F is extended appropriately. We now consider how a “suitable” free block is selected to service a memory request. To illustrate the process, assume that we have a memory pool with 200 units of storage. After some series of allocation requests and releases, we have reached a point where there are four free blocks on the freelist of sizes 25, 35, 32, and 45 (in that order). Assume that a request is made for 30 units of storage. For our examples, we ignore the overhead imposed for the tag, link, and size fields discussed above. 426 Chap. 12 Lists and Arrays Revisited + P SF k k− − Figure 12.13 Adding block F to the freelist. The word immediately preceding the start of F in the memory pool stores the tag bit of the preceding block P. If P is free, merge F into P. We find the end of F by using F’s size field. The word following the end of F is the tag field for block S. If S is free, merge it into F. The simplest method for selecting a block would be to move down the free block list until a block of size at least 30 is found. Any remaining space in this block is left on the freelist. If we begin at the beginning of the list and work down to the first free block at least as large as 30, we select the block of size 35. 30 units of storage will be allocated, leaving a free block with 5 units of space. Because this approach selects the first block with enough space, it is called first fit. A simple variation that will improve performance is, instead of always beginning at the head of the freelist, remember the last position reached in the previous search and start from there. When the end of the freelist is reached, search begins again at the head of the freelist. This modification reduces the number of unnecessary searches through small blocks that were passed over by previous requests. There is a potential disadvantage to first fit: It might “waste” larger blocks by breaking them up, and so they will not be available for large requests later. A strategy that avoids using large blocks unnecessarily is called best fit. Best fit looks at the entire list and picks the smallest block that is at least as large as the request (i.e., the “best” or closest fit to the request). Continuing with the preceding example, the best fit for a request of 30 units is the block of size 32, leaving a remainder of size 2. Best fit has the disadvantage that it requires that the entire list be searched. Another problem is that the remaining portion of the best-fit block is likely to be small, and thus useless for future requests. In other words, best fit tends to maximize problems of external fragmentation while it minimizes the chance of not being able to service an occasional large request. A strategy contrary to best fit might make sense because it tends to minimize the effects of external fragmentation. This is called worst fit, which always allocates the largest block on the list hoping that the remainder of the block will be useful for servicing a future request. In our example, the worst fit is the block of size 45, leaving a remainder of size 15. If there are a few unusually large requests, this approach will have less chance of servicing them. If requests generally tend Sec. 12.3 Memory Management 427 to be of the same size, then this might be an effective strategy. Like best fit, worst fit requires searching the entire freelist at each memory request to find the largest block. Alternatively, the freelist can be ordered from largest to smallest free block, possibly by using a priority queue implementation. Which strategy is best? It depends on the expected types of memory requests. If the requests are of widely ranging size, best fit might work well. If the requests tend to be of similar size, with rare large and small requests, first or worst fit might work well. Unfortunately, there are always request patterns that one of the three sequential fit methods will service, but which the other two will not be able to service. For example, if the series of requests 600, 650, 900, 500, 100 is made to a freelist containing blocks 500, 700, 650, 900 (in that order), the requests can all be serviced by first fit, but not by best fit. Alternatively, the series of requests 600, 500, 700, 900 can be serviced by best fit but not by first fit on this same freelist. Buddy Methods Sequential-fit methods rely on a linked list of free blocks, which must be searched for a suitable block at each memory request. Thus, the time to find a suitable free block would be ⇥(n) in the worst case for a freelist containing n blocks. Merging adjacent free blocks is somewhat complicated. Finally, we must either use addi- tional space for the linked list, or use space within the memory pool to support the memory manager operations. In the second option, both free and reserved blocks require tag and size fields. Fields in free blocks do not cost any space (because they are stored in memory that is not otherwise being used), but fields in reserved blocks create additional overhead. The buddy system solves most of these problems. Searching for a block of the proper size is efficient, merging adjacent free blocks is simple, and no tag or other information fields need be stored within reserved blocks. The buddy system assumes that memory is of size 2N for some integer N. Both free and reserved blocks will always be of size 2k for k  N. At any given time, there might be both free and reserved blocks of various sizes. The buddy system keeps a separate list for free blocks of each size. There can be at most N such lists, because there can only be N distinct block sizes. When a request comes in for m words, we first determine the smallest value of k such that 2k m. A block of size 2k is selected from the free list for that block size if one exists. The buddy system does not worry about internal fragmentation: The entire block of size 2k is allocated. If no block of size 2k exists, the next larger block is located. This block is split in half (repeatedly if necessary) until the desired block of size 2k is created. Any other blocks generated as a by-product of this splitting process are placed on the appropriate freelists. 428 Chap. 12 Lists and Arrays Revisited (a) (b) 0000 1000 0000 0100 1000 1100 Buddies Buddies Buddies Figure 12.14 Example of the buddy system. (a) Blocks of size 8. (b) Blocks of size 4. The disadvantage of the buddy system is that it allows internal fragmentation. For example, a request for 257 words will require a block of size 512. The primary advantages of the buddy system are (1) there is less external fragmentation; (2) search for a block of the right size is cheaper than, say, best fit because we need only find the first available block on the block list for blocks of size 2k; and (3) merging adjacent free blocks is easy. The reason why this method is called the buddy system is because of the way that merging takes place. The buddy for any block of size 2k is another block of the same size, and with the same address (i.e., the byte position in memory, read as a binary value) except that the kth bit is reversed. For example, the block of size 8 with beginning address 0000 in Figure 12.14(a) has buddy with address 1000. Likewise, in Figure 12.14(b), the block of size 4 with address 0000 has buddy 0100. If free blocks are sorted by address value, the buddy can be found by searching the correct block-size list. Merging simply requires that the address for the combined buddies be moved to the freelist for the next larger block size. Other Memory Allocation Methods In addition to sequential-fit and buddy methods, there are many ad hoc approaches to memory management. If the application is sufficiently complex, it might be desirable to break available memory into several memory zones, each with a differ- ent memory management scheme. For example, some zones might have a simple memory access pattern of first-in, first-out. This zone can therefore be managed ef- ficiently by using a simple stack. Another zone might allocate only records of fixed size, and so can be managed with a simple freelist as described in Section 4.1.2. Other zones might need one of the general-purpose memory allocation methods discussed in this section. The advantage of zones is that some portions of memory Sec. 12.3 Memory Management 429 can be managed more efficiently. The disadvantage is that one zone might fill up while other zones have excess free memory if the zone sizes are chosen poorly. Another approach to memory management is to impose a standard size on all memory requests. We have seen an example of this concept already in disk file management, where all files are allocated in multiples of the cluster size. This approach leads to internal fragmentation, but managing files composed of clusters is easier than managing arbitrarily sized files. The cluster scheme also allows us to relax the restriction that the memory request be serviced by a contiguous block of memory. Most disk file managers and operating system main memory managers work on a cluster or page system. Block management is usually done with a buffer pool to allocate available blocks in main memory efficiently. 12.3.2 Failure Policies and Garbage Collection At some point when processing a series of requests, a memory manager could en- counter a request for memory that it cannot satisfy. In some situations, there might be nothing that can be done: There simply might not be enough free memory to service the request, and the application may require that the request be serviced im- mediately. In this case, the memory manager has no option but to return an error, which could in turn lead to a failure of the application program. However, in many cases there are alternatives to simply returning an error. The possible options are referred to collectively as failure policies. In some cases, there might be sufficient free memory to satisfy the request, but it is scattered among small blocks. This can happen when using a sequential- fit memory allocation method, where external fragmentation has led to a series of small blocks that collectively could service the request. In this case, it might be possible to compact memory by moving the reserved blocks around so that the free space is collected into a single block. A problem with this approach is that the application must somehow be able to deal with the fact that its data have now been moved to different locations. If the application program relies on the absolute positions of the data in any way, this would be disastrous. One approach for dealing with this problem involves the handles returned by the memory manager. A handle works as a second level of indirection to a memory location. The memory allocation routine does not return a pointer to the block of storage, but rather a pointer to a the handle that in turn gives access to the storage. The handle never moves its position, but the position of the block might be moved and the value of the handle updated. Of course, this requires that the memory manager keep track of the handles and how they associate with the stored messages. Figure 12.15 illustrates the concept. Another failure policy that might work in some applications is to defer the memory request until sufficient memory becomes available. For example, a multi- tasking operating system could adopt the strategy of not allowing a process to run until there is sufficient memory available. While such a delay might be annoying 430 Chap. 12 Lists and Arrays Revisited Memory BlockHandle Figure 12.15 Using handles for dynamic memory management. The memory manager returns the address of the handle in response to a memory request. The handle stores the address of the actual memory block. In this way, the memory block might be moved (with its address updated in the handle) without disrupting the application program. to the user, it is better than halting the entire system. The assumption here is that other processes will eventually terminate, freeing memory. Another option might be to allocate more memory to the memory manager. In a zoned memory allocation system where the memory manager is part of a larger system, this might be a viable option. In a C++ program that implements its own memory manager, it might be possible to get more memory from the system-level new operator, such as is done by the freelist of Section 4.1.2. The last failure policy that we will consider is garbage collection. Consider the following series of statements. int* p = new int[5]; int* q = new int[10]; p = q; While in Java this would be no problem (due to automatic garbage collection), in languages such as C++, this would be considered bad form because the original space allocated to p is lost as a result of the third assignment. This space cannot be used again by the program. Such lost memory is referred to as garbage, also known as a memory leak. When no program variable points to a block of space, no future access to that space is possible. Of course, if another variable had first been assigned to point to p’s space, then reassigning p would not create garbage. Some programming languages take a different view towards garbage. In par- ticular, the LISP programming language uses the multilist representation of Fig- ure 12.5, and all storage is in the form either of internal nodes with two pointers or atoms. Figure 12.16 shows a typical collection of LISP structures, headed by variables named A, B, and C, along with a freelist. In LISP, list objects are constantly being put together in various ways as tem- porary variables, and then all reference to them is lost when the object is no longer needed. Thus, garbage is normal in LISP, and in fact cannot be avoided during routine program behavior. When LISP runs out of memory, it resorts to a garbage collection process to recover the space tied up in garbage. Garbage collection con- sists of examining the managed memory pool to determine which parts are still Sec. 12.3 Memory Management 431 a c de f g h A B C Freelist Figure 12.16 Example of LISP list variables, including the system freelist. being used and which parts are garbage. In particular, a list is kept of all program variables, and any memory locations not reachable from one of these variables are considered to be garbage. When the garbage collector executes, all unused memory locations are placed in free store for future access. This approach has the advantage that it allows for easy collection of garbage. It has the disadvantage, from a user’s point of view, that every so often the system must halt while it performs garbage collection. For example, garbage collection is noticeable in the Emacs text edi- tor, which is normally implemented in LISP. Occasionally the user must wait for a moment while the memory management system performs garbage collection. The Java programming language also makes use of garbage collection. As in LISP, it is common practice in Java to allocate dynamic memory as needed, and to later drop all references to that memory. The garbage collector is responsible for reclaiming such unused space as necessary. This might require extra time when running the program, but it makes life considerably easier for the programmer. In contrast, many large applications written in C++ (even commonly used commercial software) contain memory leaks that will in time cause the program to fail. Several algorithms have been used for garbage collection. One is the reference count algorithm. Here, every dynamically allocated memory block includes space for a count field. Whenever a pointer is directed to a memory block, the reference count is increased. Whenever a pointer is directed away from a memory block, the reference count is decreased. If the count ever becomes zero, then the memory block is considered garbage and is immediately placed in free store. This approach has the advantage that it does not require an explicit garbage collection phase, be- cause information is put in free store immediately when it becomes garbage. 432 Chap. 12 Lists and Arrays Revisited g h Figure 12.17 Garbage cycle example. All memory elements in the cycle have non-zero reference counts because each element has one pointer to it, even though the entire cycle is garbage (i.e., no static variable in the program points to it). Reference counts are used by the UNIX file system. Files can have multiple names, called links. The file system keeps a count of the number of links to each file. Whenever a file is “deleted,” in actuality its link field is simply reduced by one. If there is another link to the file, then no space is recovered by the file system. When the number of links goes to zero, the file’s space becomes available for reuse. Reference counts have several major disadvantages. First, a reference count must be maintained for each memory object. This works well when the objects are large, such as a file. However, it will not work well in a system such as LISP where the memory objects typically consist of two pointers or a value (an atom). Another major problem occurs when garbage contains cycles. Consider Figure 12.17. Here each memory object is pointed to once, but the collection of objects is still garbage because no pointer points to the collection. Thus, reference counts only work when the memory objects are linked together without cycles, such as the UNIX file sys- tem where files can only be organized as a DAG. Another approach to garbage collection is the mark/sweep strategy. Here, each memory object needs only a single mark bit rather than a reference counter field. When free store is exhausted, a separate garbage collection phase takes place as follows. 1. Clear all mark bits. 2. Perform depth-first search (DFS) following pointers beginning with each variable on the system’s list of static variables. Each memory element en- countered during the DFS has its mark bit turned on. 3. A “sweep” is made through the memory pool, visiting all elements. Un- marked elements are considered garbage and placed in free store. The advantages of the mark/sweep approach are that it needs less space than is necessary for reference counts, and it works for cycles. However, there is a major disadvantage. This is a “hidden” space requirement needed to do the processing. DFS is a recursive algorithm: Either it must be implemented recursively, in which case the compiler’s runtime system maintains a stack, or else the memory manager can maintain its own stack. What happens if all memory is contained in a single linked list? Then the depth of the recursion (or the size of the stack) is the number of memory cells! Unfortunately, the space for the DFS stack must be available at the worst conceivable time, that is, when free memory has been exhausted. Sec. 12.4 Further Reading 433 (a) a b 4 prev ec curr (b) 4 6 6 2 2 3 5 1 3 1 5 a b ce Figure 12.18 Example of the Deutsch-Schorr-Waite garbage collection alg- orithm. (a) The initial multilist structure. (b) The multilist structure of (a) at the instant when link node 5 is being processed by the garbage collection alg- orithm. A chain of pointers stretching from variable prev to the head node of the structure has been (temporarily) created by the garbage collection algorithm. Fortunately, a clever technique allows DFS to be performed without requiring additional space for a stack. Instead, the structure being traversed is used to hold the stack. At each step deeper into the traversal, instead of storing a pointer on the stack, we “borrow” the pointer being followed. This pointer is set to point back to the node we just came from in the previous step, as illustrated by Figure 12.18. Each borrowed pointer stores an additional bit to tell us whether we came down the left branch or the right branch of the link node being pointed to. At any given instant we have passed down only one path from the root, and we can follow the trail of pointers back up. As we return (equivalent to popping the recursion stack), we set the pointer back to its original position so as to return the structure to its original condition. This is known as the Deutsch-Schorr-Waite garbage collection algorithm. 12.4 Further Reading For information on LISP, see The Little LISPer by Friedman and Felleisen [FF89]. Another good LISP reference is Common LISP: The Language by Guy L. Steele [Ste90]. For information on Emacs, which is both an excellent text editor and a programming environment, see the GNU Emacs Manual by Richard Stallman 434 Chap. 12 Lists and Arrays Revisited (c)(b)(a) a b d e c aL1 L1 L2 L4 a bdc L3 Figure 12.19 Some example multilists. [Sta11b]. You can get more information about Java’s garbage collection system from The Java Programming Language by Ken Arnold and James Gosling [AG06]. For more details on sparse matrix representations, the Yale representation is de- scribed by Eisenstat, Schultz and Sherman [ESS81]. The MATLAB sparse matrix representation is described by Gilbert, Moler, and Schreiber [GMS91]. An introductory text on operating systems covers many topics relating to mem- ory management issues, including layout of files on disk and caching of information in main memory. All of the topics covered here on memory management, buffer pools, and paging are relevant to operating system implementation. For example, see Operating Systems by William Stallings[Sta11a]. 12.5 Exercises 12.1 For each of the following bracket notation descriptions, draw the equivalent multilist in graphical form such as shown in Figure 12.2. (a) ha, b, hc, d, ei, hf,hgi,hii(b) ha, b, hc, d, L1:ei,L1i(c) hL1:a, L1, hL2:bi,L2, hL1ii12.2 (a) Show the bracket notation for the list of Figure 12.19(a). (b) Show the bracket notation for the list of Figure 12.19(b). (c) Show the bracket notation for the list of Figure 12.19(c). 12.3 Given the linked representation of a pure list such as hx1, hy1,y2, hz1,z2i,y4i, hw1,w2i,x4i, write an in-place reversal algorithm to reverse the sublists at all levels in- cluding the topmost level. For this example, the result would be a linked representation corresponding to hx4, hw2,w1i, hy4, hz2,z1i,y2,y1i,x1i. 12.4 What fraction of the values in a matrix must be zero for the sparse matrix representation of Section 12.2 to be more space efficient than the standard two-dimensional matrix representation when data values require eight bytes, array indices require two bytes, and pointers require four bytes? Sec. 12.6 Projects 435 12.5 Write a function to add an element at a given position to the sparse matrix representation of Section 12.2. 12.6 Write a function to delete an element from a given position in the sparse matrix representation of Section 12.2. 12.7 Write a function to transpose a sparse matrix as represented in Section 12.2. 12.8 Write a function to add two sparse matrices as represented in Section 12.2. 12.9 Write memory manager allocation and deallocation routines for the situation where all requests and releases follow a last-requested, first-released (stack) order. 12.10 Write memory manager allocation and deallocation routines for the situation where all requests and releases follow a last-requested, last-released (queue) order. 12.11 Show the result of allocating the following blocks from a memory pool of size 1000 using first fit for each series of block requests. State if a given request cannot be satisfied. (a) Take 300 (call this block A), take 500, release A, take 200, take 300. (b) Take 200 (call this block A), take 500, release A, take 200, take 300. (c) Take 500 (call this block A), take 300, release A, take 300, take 200. 12.12 Show the result of allocating the following blocks from a memory pool of size 1000 using best fit for each series of block requests. State if a given request cannot be satisfied. (a) Take 300 (call this block A), take 500, release A, take 200, take 300. (b) Take 200 (call this block A), take 500, release A, take 200, take 300. (c) Take 500 (call this block A), take 300, release A, take 300, take 200. 12.13 Show the result of allocating the following blocks from a memory pool of size 1000 using worst fit for each series of block requests. State if a given request cannot be satisfied. (a) Take 300 (call this block A), take 500, release A, take 200, take 300. (b) Take 200 (call this block A), take 500, release A, take 200, take 300. (c) Take 500 (call this block A), take 300, release A, take 300, take 200. 12.14 Assume that the memory pool contains three blocks of free storage. Their sizes are 1300, 2000, and 1000. Give examples of storage requests for which (a) first-fit allocation will work, but not best fit or worst fit. (b) best-fit allocation will work, but not first fit or worst fit. (c) worst-fit allocation will work, but not first fit or best fit. 12.6 Projects 12.1 Implement the orthogonal list sparse matrix representation of Section 12.2. Your implementation should support the following operations on the matrix: 436 Chap. 12 Lists and Arrays Revisited • insert an element at a given position, • delete an element from a given position, • return the value of the element at a given position, • take the transpose of a matrix, • add two matrices, and • multiply two matrices. 12.2 Implement the Yale model for sparse matrices described at the end of Sec- tion 12.2. Your implementation should support the following operations on the matrix: • insert an element at a given position, • delete an element from a given position, • return the value of the element at a given position, • take the transpose of a matrix, • add two matrices, and • multiply two matrices. 12.3 Implement the MemManager ADT shown at the beginning of Section 12.3. Use a separate linked list to implement the freelist. Your implementation should work for any of the three sequential-fit methods: first fit, best fit, and worst fit. Test your system empirically to determine under what conditions each method performs well. 12.4 Implement the MemManager ADT shown at the beginning of Section 12.3. Do not use separate memory for the free list, but instead embed the free list into the memory pool as shown in Figure 12.12. Your implementation should work for any of the three sequential-fit methods: first fit, best fit, and worst fit. Test your system empirically to determine under what conditions each method performs well. 12.5 Implement the MemManager ADT shown at the beginning of Section 12.3 using the buddy method of Section 12.3.1. Your system should support requests for blocks of a specified size and release of previously requested blocks. 12.6 Implement the Deutsch-Schorr-Waite garbage collection algorithm that is il- lustrated by Figure 12.18. 13 Advanced Tree Structures This chapter introduces several tree structures designed for use in specialized ap- plications. The trie of Section 13.1 is commonly used to store and retrieve strings. It also serves to illustrate the concept of a key space decomposition. The AVL tree and splay tree of Section 13.2 are variants on the BST. They are examples of self-balancing search trees and have guaranteed good performance regardless of the insertion order for records. An introduction to several spatial data structures used to organize point data by xy-coordinates is presented in Section 13.3. Descriptions of the fundamental operations are given for each data structure. One purpose for this chapter is to provide opportunities for class programming projects, so detailed implementations are left to the reader. 13.1 Tries Recall that the shape of a BST is determined by the order in which its data records are inserted. One permutation of the records might yield a balanced tree while another might yield an unbalanced tree, with the extreme case becoming the shape of a linked list. The reason is that the value of the key stored in the root node splits the key range into two parts: those key values less than the root’s key value, and those key values greater than the root’s key value. Depending on the relationship between the root node’s key value and the distribution of the key values for the other records in the the tree, the resulting BST might be balanced or unbalanced. Thus, the BST is an example of a data structure whose organization is based on an object space decomposition, so called because the decomposition of the key range is driven by the objects (i.e., the key values of the data records) stored in the tree. The alternative to object space decomposition is to predefine the splitting posi- tion within the key range for each node in the tree. In other words, the root could be predefined to split the key range into two equal halves, regardless of the particular values or order of insertion for the data records. Those records with keys in the lower half of the key range will be stored in the left subtree, while those records 437 438 Chap. 13 Advanced Tree Structures with keys in the upper half of the key range will be stored in the right subtree. While such a decomposition rule will not necessarily result in a balanced tree (the tree will be unbalanced if the records are not well distributed within the key range), at least the shape of the tree will not depend on the order of key insertion. Further- more, the depth of the tree will be limited by the resolution of the key range; that is, the depth of the tree can never be greater than the number of bits required to store a key value. For example, if the keys are integers in the range 0 to 1023, then the resolution for the key is ten bits. Thus, two keys can be identical only until the tenth bit. In the worst case, two keys will follow the same path in the tree only until the tenth branch. As a result, the tree will never be more than ten levels deep. In contrast, a BST containing n records could be as much as n levels deep. Splitting based on predetermined subdivisions of the key range is called key space decomposition. In computer graphics, the technique is known as image space decomposition, and this term is sometimes used to describe the process for data structures as well. A data structure based on key space decomposition is called a trie. Folklore has it that “trie” comes from “retrieval.” Unfortunately, that would imply that the word is pronounced “tree,” which would lead to confusion with reg- ular use of the word “tree.” “Trie” is actually pronounced as “try.” Like the B+-tree, a trie stores data records only in leaf nodes. Internal nodes serve as placeholders to direct the search process. but since the split points are pre- determined, internal nodes need not store “traffic-directing” key values. Figure 13.1 illustrates the trie concept. Upper and lower bounds must be imposed on the key values so that we can compute the middle of the key range. Because the largest value inserted in this example is 120, a range from 0 to 127 is assumed, as 128 is the smallest power of two greater than 120. The binary value of the key determines whether to select the left or right branch at any given point during the search. The most significant bit determines the branch direction at the root. Figure 13.1 shows a binary trie, so called because in this example the trie structure is based on the value of the key interpreted as a binary number, which results in a binary tree. The Huffman coding tree of Section 5.6 is another example of a binary trie. All data values in the Huffman tree are at the leaves, and each branch splits the range of possible letter codes in half. The Huffman codes are actually reconstructed from the letter positions within the trie. These are examples of binary tries, but tries can be built with any branching factor. Normally the branching factor is determined by the alphabet used. For binary numbers, the alphabet is {0, 1} and a binary trie results. Other alphabets lead to other branching factors. One application for tries is to store a dictionary of words. Such a trie will be referred to as an alphabet trie. For simplicity, our examples will ignore case in letters. We add a special character ($) to the 26 standard English letters. The $ character is used to represent the end of a string. Thus, the branching factor for Sec. 13.1 Tries 439 0 0 0 27 1 24 1 1 1 120 0 0 1 1 32 0 01 40 42 37 0 0 0 Figure 13.1 The binary trie for the collection of values 2, 7, 24, 31, 37, 40, 42, 120. All data values are stored in the leaf nodes. Edges are labeled with the value of the bit used to determine the branching direction of each node. The binary form of the key value determines the path to the record, assuming that each key is represented as a 7-bit value representing a number in the range 0 to 127. each node is (up to) 27. Once constructed, the alphabet trie is used to determine if a given word is in the dictionary. Consider searching for a word in the alphabet trie of Figure 13.2. The first letter of the search word determines which branch to take from the root, the second letter determines which branch to take at the next level, and so on. Only the letters that lead to a word are shown as branches. In Figure 13.2(b) the leaf nodes of the trie store a copy of the actual words, while in Figure 13.2(a) the word is built up from the letters associated with each branch. One way to implement a node of the alphabet trie is as an array of 27 pointers indexed by letter. Because most nodes have branches to only a small fraction of the possible letters in the alphabet, an alternate implementation is to use a linked list of pointers to the child nodes, as in Figure 6.9. The depth of a leaf node in the alphabet trie of Figure 13.2(b) has little to do with the number of nodes in the trie, or even with the length of the corresponding string. Rather, a node’s depth depends on the number of characters required to distinguish this node’s word from any other. For example, if the words “anteater” and “antelope” are both stored in the trie, it is not until the fifth letter that the two words can be distinguished. Thus, these words must be stored at least as deep as level five. In general, the limiting factor on the depth of nodes in the alphabet trie is the length of the words stored. Poor balance and clumping can result when certain prefixes are heavily used. For example, an alphabet trie storing the common words in the English language would have many words in the “th” branch of the tree, but none in the “zq” branch. Any multiway branching trie can be replaced with a binary trie by replacing the original trie’s alphabet with an equivalent binary code. Alternatively, we can use the techniques of Section 6.3.4 for converting a general tree to a binary tree without modifying the alphabet. 440 Chap. 13 Advanced Tree Structures e l u o (b) ant e l chicken d u deer duck gh lo horse goosegoldfishgoat antelope (a) n t $ a t e r $ o p e $ c h i c k n $ e r $ c k $ g o a $ l d f i s h $ r s e a d t o h $ e e s e $ a n t $ a c eo a anteater Figure 13.2 Two variations on the alphabet trie representation for a set of ten words. (a) Each node contains a set of links corresponding to single letters, and each letter in the set of words has a corresponding link. “$” is used to indicate the end of a word. Internal nodes direct the search and also spell out the word one letter per link. The word need not be stored explicitly. “$” is needed to recognize the existence of words that are prefixes to other words, such as ‘ant’ in this example. (b) Here the trie extends only far enough to discriminate between the words. Leaf nodes of the trie each store a complete word; internal nodes merely direct the search. Sec. 13.1 Tries 441 1xxxxxx 0 120 01xxxxx00xxxxx 2 3 0101xxx 4 24 4 5 010101x 2 7 32 37 40 42 000xxxx 0xxxxxx 1 Figure 13.3 The PAT trie for the collection of values 2, 7, 24, 32, 37, 40, 42, 120. Contrast this with the binary trie of Figure 13.1. In the PAT trie, all data values are stored in the leaf nodes, while internal nodes store the bit position used to determine the branching decision, assuming that each key is represented as a 7- bit value representing a number in the range 0 to 127. Some of the branches in this PAT trie have been labeled to indicate the binary representation for all values in that subtree. For example, all values in the left subtree of the node labeled 0 must have value 0xxxxxx (where x means that bit can be either a 0 or a 1). All nodes in the right subtree of the node labeled 3 must have value 0101xxx. However, we can skip branching on bit 2 for this subtree because all values currently stored have a value of 0 for that bit. The trie implementations illustrated by Figures 13.1 and 13.2 are potentially quite inefficient as certain key sets might lead to a large number of nodes with only a single child. A variant on trie implementation is known as PATRICIA, which stands for “Practical Algorithm To Retrieve Information Coded In Alphanumeric.” In the case of a binary alphabet, a PATRICIA trie (referred to hereafter as a PAT trie) is a full binary tree that stores data records in the leaf nodes. Internal nodes store only the position within the key’s bit pattern that is used to decide on the next branching point. In this way, internal nodes with single children (equivalently, bit positions within the key that do not distinguish any of the keys within the current subtree) are eliminated. A PAT trie corresponding to the values of Figure 13.1 is shown in Figure 13.3. Example 13.1 When searching for the value 7 (0000111 in binary) in the PAT trie of Figure 13.3, the root node indicates that bit position 0 (the leftmost bit) is checked first. Because the 0th bit for value 7 is 0, take the left branch. At level 1, branch depending on the value of bit 1, which again is 0. At level 2, branch depending on the value of bit 2, which again is 0. At level 3, the index stored in the node is 4. This means that bit 4 of the key is checked next. (The value of bit 3 is irrelevant, because all values stored in that subtree have the same value at bit position 3.) Thus, the single branch that extends from the equivalent node in Figure 13.1 is just skipped. For key value 7, bit 4 has value 1, so the rightmost branch is taken. Because 442 Chap. 13 Advanced Tree Structures this leads to a leaf node, the search key is compared against the key stored in that node. If they match, then the desired record has been found. Note that during the search process, only a single bit of the search key is com- pared at each internal node. This is significant, because the search key could be quite large. Search in the PAT trie requires only a single full-key comparison, which takes place once a leaf node has been reached. Example 13.2 Consider the situation where we need to store a library of DNA sequences. A DNA sequence is a series of letters, usually many thou- sands of characters long, with the string coming from an alphabet of only four letters that stand for the four amino acids making up a DNA strand. Similar DNA sequences might have long sections of their string that are identical. The PAT trie would avoid making multiple full key comparisons when searching for a specific sequence. 13.2 Balanced Trees We have noted several times that the BST has a high risk of becoming unbalanced, resulting in excessively expensive search and update operations. One solution to this problem is to adopt another search tree structure such as the 2-3 tree or the binary trie. An alternative is to modify the BST access functions in some way to guarantee that the tree performs well. This is an appealing concept, and it works well for heaps, whose access functions maintain the heap in the shape of a complete binary tree. Unfortunately, requiring that the BST always be in the shape of a complete binary tree requires excessive modification to the tree during update, as discussed in Section 10.3. If we are willing to weaken the balance requirements, we can come up with alternative update routines that perform well both in terms of cost for the update and in balance for the resulting tree structure. The AVL tree works in this way, using insertion and deletion routines altered from those of the BST to ensure that, for every node, the depths of the left and right subtrees differ by at most one. The AVL tree is described in Section 13.2.1. A different approach to improving the performance of the BST is to not require that the tree always be balanced, but rather to expend some effort toward making the BST more balanced every time it is accessed. This is a little like the idea of path compression used by the UNION/FIND algorithm presented in Section 6.2. One example of such a compromise is called the splay tree. The splay tree is described in Section 13.2.2. Sec. 13.2 Balanced Trees 443 7 2 32 42 40 120 37 42 24 7 2 32 42 40 120 37 42 24 5 Figure 13.4 Example of an insert operation that violates the AVL tree balance property. Prior to the insert operation, all nodes of the tree are balanced (i.e., the depths of the left and right subtrees for every node differ by at most one). After inserting the node with value 5, the nodes with values 7 and 24 are no longer balanced. 13.2.1 The AVL Tree The AVL tree (named for its inventors Adelson-Velskii and Landis) should be viewed as a BST with the following additional property: For every node, the heights of its left and right subtrees differ by at most 1. As long as the tree maintains this property, if the tree contains n nodes, then it has a depth of at most O(log n). As a result, search for any node will cost O(log n), and if the updates can be done in time proportional to the depth of the node inserted or deleted, then updates will also cost O(log n), even in the worst case. The key to making the AVL tree work is to alter the insert and delete routines so as to maintain the balance property. Of course, to be practical, we must be able to implement the revised update routines in ⇥(log n) time. Consider what happens when we insert a node with key value 5, as shown in Figure 13.4. The tree on the left meets the AVL tree balance requirements. After the insertion, two nodes no longer meet the requirements. Because the original tree met the balance requirement, nodes in the new tree can only be unbalanced by a difference of at most 2 in the subtrees. For the bottommost unbalanced node, call it S, there are 4 cases: 1. The extra node is in the left child of the left child of S. 2. The extra node is in the right child of the left child of S. 3. The extra node is in the left child