Mauerer ffirs.tex V2 - 08/26/2008 3:23am Page iii Professional Linux® Kernel Architecture Wolfgang Mauerer Wiley Publishing, Inc. Mauerer ffirs.tex V2 - 08/26/2008 3:23am Page ii Mauerer ffirs.tex V2 - 08/26/2008 3:23am Page i Professional Linux® Kernel Architecture Introduction ..................................................................xxvii Chapter 1: Introduction and Overview .......................................... 1 Chapter 2: Process Management and Scheduling .............................35 Chapter 3: Memory Management ............................................ 133 Chapter 4: Virtual Process Memory .......................................... 289 Chapter 5: Locking and Interprocess Communication ....................... 347 Chapter 6: Device Drivers .................................................... 391 Chapter 7: Modules .......................................................... 473 Chapter 8: The Virtual Filesystem............................................ 519 Chapter 9: The Extended Filesystem Family ................................. 583 Chapter 10: Filesystems without Persistent Storage ....................... 643 Chapter 11: Extended Attributes and Access Control Lists ................. 707 Chapter 12: Networks ........................................................ 733 Chapter 13: System Calls .................................................... 819 Chapter 14: Kernel Activities ................................................ 847 Chapter 15: Time management .............................................. 893 Chapter 16: Page and Buffer Cache .......................................... 949 Chapter 17: Data Synchronization ........................................... 989 Chapter 18: Page Reclaim and Swapping................................... 1023 Chapter 19: Auditing ........................................................ 1097 Appendix A: Architecture Specifics ......................................... 1117 Appendix B: Working with the Source Code ................................ 1141 Appendix C: Notes on C ..................................................... 1175 Appendix D: System Startup ................................................ 1223 Appendix E: The ELF Binary Format ......................................... 1241 Appendix F: The Kernel Development Process .............................. 1267 Bibliography ................................................................. 1289 Index ........................................................................ 1293 Mauerer ffirs.tex V2 - 08/26/2008 3:23am Page ii Mauerer ffirs.tex V2 - 08/26/2008 3:23am Page iii Professional Linux® Kernel Architecture Wolfgang Mauerer Wiley Publishing, Inc. Mauerer ffirs.tex V2 - 08/26/2008 3:23am Page iv Professional Linux® Kernel Architecture Published by Wiley Publishing, Inc. 10475 Crosspoint Boulevard Indianapolis, IN 46256 www.wiley.com Copyright © 2008 by Wolfgang Mauerer Published by Wiley Publishing, Inc., Indianapolis, Indiana Published simultaneously in Canada ISBN: 978-0-470-34343-2 Manufactured in the United States of America 10987654321 Library of Congress Cataloging-in-Publication Data: Mauerer, Wolfgang, 1978- Professional Linux kernel architecture / Wolfgang Mauerer. p. cm. Includes index. ISBN 978-0-470-34343-2 (pbk.) 1. Linux. 2. Computer architecture. 3. Application software. I. Title. QA76.9.A73M38 2008 005.4’32--dc22 2008028067 No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Legal Department, Wiley Publishing, Inc., 10475 Crosspoint Blvd., Indianapolis, IN 46256, (317) 572-3447, fax (317) 572-4355, or online at http://www.wiley.com/go/permissions. Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or Website is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or Website may provide or recommendations it may make. Further, readers should be aware that Internet Websites listed in this work may have changed or disappeared between when this work was written and when it is read. For general information on our other products and services please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002. Trademarks: Wiley, the Wiley logo, Wrox, the Wrox logo, Wrox Programmer to Programmer, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates, in the United States and other countries, and may not be used without written permission. All other trademarks are the property of their respective owners. Wiley Publishing, Inc., is not associated with any product or vendor mentioned in this book. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Mauerer fauth.tex V2 - 08/22/2008 4:52am Page v About the Author Wolfgang Mauerer is a quantum physicist whose professional interests are centered around quantum cryptography, quantum electrodynamics, and compilers for — you guessed it — quantum architectures. With the confirmed capacity of being the worst experimentalist in the known universe, he sticks to the theoretical side of his profession, which is especially reassuring considering his constant fear of acci- dentally destroying the universe. Outside his research work, he is fascinated by operating systems, and for more than a decade — starting with an article series about the kernel in 1997 — he has found great pleasure in documenting and explaining Linux kernel internals. He is also the author of a book about typesetting with LaTeX and has written numerous articles that have been translated into seven languages in total. When he’s not submerged in vast Hilbert spaces or large quantities of source code, he tries to take the opposite direction, namely, upward — be this with model planes, a paraglider, or on foot with an ice axe in his hands: Mountains especially have the power to outrival even the Linux kernel. Consequently, he considers planning and accomplishing a first-ascent expedition to the vast arctic glaciers of east Green- land to be the really unique achievement in his life. Being interested in everything that is fundamental, he is also the author of the first compiler for Plankalk¨ul, the world’s earliest high-level language devised in 1942–1946 by Konrad Zuse, the father of the computer. As an avid reader, he is proud that despite the two-digit number of computers present in his living room, the volume required for books still occupies a larger share. Mauerer fauth.tex V2 - 08/22/2008 4:52am Page vi Mauerer fcredit.tex V2 - 08/22/2008 4:53am Page vii Credits Executive Editor Carol Long Senior Development Editor Tom Dinse Production Editor Debra Banninger Copy Editors Cate Caffrey Kathryn Duggan Editorial Manager Mary Beth Wakefield Production Manager Tim Tate Vice President and Executive Group Publisher Richard Swadley Vice President and Executive Publisher Joseph B. Wikert Project Coordinator, Cover Lynsey Stanford Proofreader Publication Services, Inc. Indexer Jack Lewis Mauerer fcredit.tex V2 - 08/22/2008 4:53am Page viii Mauerer fack.tex V4 - 09/04/2008 3:36pm Page ix Acknowledgments First and foremost, I have to thank the thousands of programmers who have created the Linux kernel over the years — most of them commercially based, but some also just for their own private or academic joy. Without them, there would be no kernel, and I would have had nothing to write about. Please accept my apologies that I cannot list all several hundred names here, but in true UNIX style, you can easily generate the list by: for file in $ALL_FILES_COVERED_IN_THIS_BOOK; do git log --pretty="format:%an" $file; done | sort -u -k 2,2 It goes without saying that I admire your work very much — you are all the true heroes in this story! What you are reading right now is the result of an evolution over more than seven years: After two years of writing, the first edition was published in German by Carl Hanser Verlag in 2003. It then described kernel 2.6.0. The text was used as a basis for the low-level design documentation for the EAL4+ security evaluation of Red Hat Enterprise Linux 5, requiring to update it to kernel 2.6.18 (if the EAL acronym does not mean anything to you, then Wikipedia is once more your friend). Hewlett-Packard sponsored the translation into English and has, thankfully, granted the rights to publish the result. Updates to kernel 2.6.24 were then performed specifically for this book. Several people were involved in this evolution, and my appreciation goes to all of them: Leslie Mackay- Poulton, with support from David Jacobs, did a tremendous job at translating a huge pile of text into English. I’m also indebted to Sal La Pietra of atsec information security for pulling the strings to get the translation project rolling, and especially to Stephan M¨uller for close cooperation during the evaluation. My cordial thanks also go to all other HP and Red Hat people involved in this evaluation, and also to Claudio Kopper and Hans L¨ohr for our very enjoyable cooperation during this project. Many thanks also go to the people at Wiley — both visible and invisible to me — who helped to shape the book into its current form. The German edition was well received by readers and reviewers, but nevertheless comments about inaccuracies and suggestions for improvements were provided. I’m glad for all of them, and would also like to mention the instructors who answered the publisher’s survey for the original edition. Some of their suggestions were very valuable for improving the current publication. The same goes for the referees for this edition, especially to Dr. Xiaodong Zhang for providing numerous suggestions for Appendix F.4. Furthermore, I express my gratitude to Dr. Christine Silberhorn for granting me the opportunity to suspend my regular research work at the Max Planck Research Group for four weeks to work on this project. I hope you enjoyed the peace during this time when nobody was trying to install Linux on your MacBook! As with every book, I owe my deepest gratitude to my family for supporting me in every aspect of life — I more than appreciate this indispensable aid. Finally, I have to thank Hariet Fabritius for infinite Mauerer fack.tex V4 - 09/04/2008 3:36pm Page x Acknowledgments patience with an author whose work cycle not only perfectly matched the most alarming forms of sleep dyssomnias, but who was always right on the brink of confusing his native tongue with ‘‘C,’’ and whom she consequently had to rescue from numerous situations where he seemingly had lost his mind (see below...). Now that I have more free time again, I’m not only looking forward to our well-deserved holiday, but can finally embark upon the project of giving your laptop all joys of a proper operating system! (Writing these acknowledgments, I all of a sudden realize why people always hasten to lock away their laptops when they see me approaching....) x Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xi Contents Introduction xxvii Chapter 1: Introduction and Overview 1 Tasks of the Kernel 2 Implementation Strategies 3 Elements of the Kernel 3 Processes, Task Switching, and Scheduling 4 Unix Processes 4 Address Spaces and Privilege Levels 7 Page Tables 11 Allocation of Physical Memory 13 Timing 16 System Calls 17 Device Drivers, Block and Character Devices 17 Networks 18 Filesystems 18 Modules and Hotplugging 18 Caching 20 List Handling 20 Object Management and Reference Counting 22 Data Types 25 ... and Beyond the Infinite 27 Why the Kernel Is Special 28 Some Notes on Presentation 29 Summary 33 Chapter 2: Process Management and Scheduling 35 Process Priorities 36 Process Life Cycle 38 Preemptive Multitasking 40 Process Representation 41 Process Types 47 Namespaces 47 Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xii Contents Process Identification Numbers 54 Task Relationships 62 Process Management System Calls 63 Process Duplication 63 Kernel Threads 77 Starting New Programs 79 Exiting Processes 83 Implementation of the Scheduler 83 Overview 84 Data Structures 86 Dealing with Priorities 93 Core Scheduler 99 The Completely Fair Scheduling Class 106 Data Structures 106 CFS Operations 107 Queue Manipulation 112 Selecting the Next Task 113 Handling the Periodic Tick 114 Wake-up Preemption 115 Handling New Tasks 116 The Real-Time Scheduling Class 117 Properties 118 Data Structures 118 Scheduler Operations 119 Scheduler Enhancements 121 SMP Scheduling 121 Scheduling Domains and Control Groups 126 Kernel Preemption and Low Latency Efforts 127 Summary 132 Chapter 3: Memory Management 133 Overview 133 Organization in the (N)UMA Model 136 Overview 136 Data Structures 138 Page Tables 153 Data Structures 154 Creating and Manipulating Entries 161 Initialization of Memory Management 161 Data Structure Setup 162 Architecture-Specific Setup 169 Memory Management during the Boot Process 191 xii Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xiii Contents Management of Physical Memory 199 Structure of the Buddy System 199 Avoiding Fragmentation 201 Initializing the Zone and Node Data Structures 209 Allocator API 215 Reserving Pages 222 Freeing Pages 240 Allocation of Discontiguous Pages in the Kernel 244 Kernel Mappings 251 The Slab Allocator 256 Alternative Allocators 258 Memory Management in the Kernel 259 The Principle of Slab Allocation 261 Implementation 265 General Caches 283 Processor Cache and TLB Control 285 Summary 287 Chapter 4: Virtual Process Memory 289 Introduction 289 Virtual Process Address Space 290 Layout of the Process Address Space 290 Creating the Layout 294 Principle of Memory Mappings 297 Data Structures 298 Trees and Lists 299 Representation of Regions 300 The Priority Search Tree 302 Operations on Regions 306 Associating Virtual Addresses with a Region 306 Merging Regions 308 Inserting Regions 309 Creating Regions 310 Address Spaces 312 Memory Mappings 314 Creating Mappings 314 Removing Mappings 317 Nonlinear Mappings 319 Reverse Mapping 322 Data Structures 323 Creating a Reverse Mapping 324 Using Reverse Mapping 325 xiii Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xiv Contents Managing the Heap 327 Handling of Page Faults 330 Correction of Userspace Page Faults 336 Demand Allocation/Paging 337 Anonymous Pages 339 Copy on Write 340 Getting Nonlinear Mappings 341 Kernel Page Faults 341 Copying Data between Kernel and Userspace 344 Summary 345 Chapter 5: Locking and Interprocess Communication 347 Control Mechanisms 348 Race Conditions 348 Critical Sections 349 Kernel Locking Mechanisms 351 Atomic Operations on Integers 352 Spinlocks 354 Semaphores 355 The Read-Copy-Update Mechanism 357 Memory and Optimization Barriers 359 Reader/Writer Locks 361 The Big Kernel Lock 361 Mutexes 362 Approximate Per-CPU Counters 364 Lock Contention and Fine-Grained Locking 365 System V Interprocess Communication 366 System V Mechanisms 366 Semaphores 367 Message Queues 376 Shared Memory 380 Other IPC Mechanisms 381 Signals 381 Pipes and Sockets 389 Summary 390 Chapter 6: Device Drivers 391 I/O Architecture 391 Expansion Hardware 392 Access to Devices 397 Device Files 397 Character, Block, and Other Devices 397 xiv Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xv Contents Device Addressing Using Ioctls 400 Representation of Major and Minor Numbers 401 Registration 403 Association with the Filesystem 406 Device File Elements in Inodes 406 Standard File Operations 407 Standard Operations for Character Devices 407 Standard Operations for Block Devices 408 Character Device Operations 409 Representing Character Devices 409 Opening Device Files 409 Reading and Writing 412 Block Device Operations 412 Representation of Block Devices 413 Data Structures 415 Adding Disks and Partitions to the System 423 Opening Block Device Files 425 Request Structure 427 BIOs 430 Submitting Requests 432 I/O Scheduling 438 Implementation of Ioctls 441 Resource Reservation 442 Resource Management 442 I/O Memory 445 I/O Ports 446 Bus Systems 448 The Generic Driver Model 449 The PCI Bus 454 USB 463 Summary 471 Chapter 7: Modules 473 Overview 473 Using Modules 474 Adding and Removing 474 Dependencies 477 Querying Module Information 478 Automatic Loading 480 Inserting and Deleting Modules 483 Module Representation 483 Dependencies and References 488 xv Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xvi Contents Binary Structure of Modules 491 Inserting Modules 496 Removing Modules 505 Automation and Hotplugging 506 Automatic Loading with kmod 507 Hotplugging 508 Version Control 511 Checksum Methods 512 Version Control Functions 515 Summary 517 Chapter 8: The Virtual Filesystem 519 Filesystem Types 520 The Common File Model 521 Inodes 522 Links 522 Programming Interface 523 Files as a Universal Interface 524 Structure of the VFS 525 Structural Overview 525 Inodes 527 Process-Specific Information 532 File Operations 537 Directory Entry Cache 542 Working with VFS Objects 547 Filesystem Operations 548 File Operations 565 Standard Functions 572 Generic Read Routine 573 The fault Mechanism 576 Permission-Checking 578 Summary 581 Chapter 9: The Extended Filesystem Family 583 Introduction 583 Second Extended Filesystem 584 Physical Structure 585 Data Structures 592 xvi Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xvii Contents Creating a Filesystem 608 Filesystem Actions 610 Third Extended Filesystem 637 Concepts 638 Data Structures 639 Summary 642 Chapter 10: Filesystems without Persistent Storage 643 The proc Filesystem 644 Contents of /proc 644 Data Structures 652 Initialization 655 Mounting the Filesystem 657 Managing /proc Entries 660 Reading and Writing Information 664 Task-Related Information 666 System Control Mechanism 671 Simple Filesystems 680 Sequential Files 680 Writing Filesystems with Libfs 684 The Debug Filesystem 687 Pseudo Filesystems 689 Sysfs 689 Overview 690 Data Structures 690 Mounting the Filesystem 695 File and Directory Operations 697 Populating Sysfs 704 Summary 706 Chapter 11: Extended Attributes and Access Control Lists 707 Extended Attributes 707 Interface to the Virtual Filesystem 708 Implementation in Ext3 714 Implementation in Ext2 721 Access Control Lists 722 Generic Implementation 722 Implementation in Ext3 726 Implementation in Ext2 732 Summary 732 xvii Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xviii Contents Chapter 12: Networks 733 Linked Computers 734 ISO/OSI and TCP/IP Reference Model 734 Communication via Sockets 738 Creating a Socket 738 Using Sockets 740 Datagram Sockets 744 The Layer Model of Network Implementation 745 Networking Namespaces 747 Socket Buffers 749 Data Management Using Socket Buffers 750 Management Data of Socket Buffers 753 Network Access Layer 754 Representation of Network Devices 755 Receiving Packets 760 Sending Packets 768 Network Layer 768 IPv4 769 Receiving Packets 771 Local Delivery to the Transport Layer 772 Packet Forwarding 774 Sending Packets 775 Netfilter 778 IPv6 783 Transport Layer 785 UDP 785 TCP 787 Application Layer 799 Socket Data Structures 799 Sockets and Files 803 The socketcall System Call 804 Creating Sockets 805 Receiving Data 807 Sending Data 808 Networking from within the Kernel 808 Communication Functions 808 The Netlink Mechanism 810 Summary 817 xviii Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xix Contents Chapter 13: System Calls 819 Basics of System Programming 820 Tracing System Calls 820 Supported Standards 823 Restarting System Calls 824 Available System Calls 826 Implementation of System Calls 830 Structure of System Calls 830 Access to Userspace 837 System Call Tracing 838 Summary 846 Chapter 14: Kernel Activities 847 Interrupts 848 Interrupt Types 848 Hardware IRQs 849 Processing Interrupts 850 Data Structures 853 Interrupt Flow Handling 860 Initializing and Reserving IRQs 864 Servicing IRQs 866 Software Interrupts 875 Starting SoftIRQ Processing 877 The SoftIRQ Daemon 878 Tasklets 879 Generating Tasklets 880 Registering Tasklets 880 Executing Tasklets 881 Wait Queues and Completions 882 Wait Queues 882 Completions 887 Work Queues 889 Summary 891 Chapter 15: Time Management 893 Overview 893 Types of Timers 893 Configuration Options 896 xix Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xx Contents Implementation of Low-Resolution Timers 897 Timer Activation and Process Accounting 897 Working with Jiffies 900 Data Structures 900 Dynamic Timers 902 Generic Time Subsystem 907 Overview 908 Configuration Options 909 Time Representation 910 Objects for Time Management 911 High-Resolution Timers 920 Data Structures 921 Setting Timers 925 Implementation 926 Periodic Tick Emulation 931 Switching to High-Resolution Timers 932 Dynamic Ticks 933 Data Structures 934 Dynamic Ticks for Low-Resolution Systems 935 Dynamic Ticks for High-Resolution Systems 938 Stopping and Starting Periodic Ticks 939 Broadcast Mode 943 Implementing Timer-Related System Calls 944 Time Bases 944 The alarm and setitimer System Calls 945 Getting the Current Time 947 Managing Process Times 947 Summary 948 Chapter 16: Page and Buffer Cache 949 Structure of the Page Cache 950 Managing and Finding Cached Pages 951 Writing Back Modified Data 952 Structure of the Buffer Cache 954 Address Spaces 955 Data Structures 956 Page Trees 958 Operations on Address Spaces 961 Implementation of the Page Cache 966 Allocating Pages 966 Finding Pages 967 xx Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xxi Contents Waiting on Pages 968 Operations with Whole Pages 969 Page Cache Readahead 970 Implementation of the Buffer Cache 974 Data Structures 975 Operations 976 Interaction of Page and Buffer Cache 977 Independent Buffers 982 Summary 988 Chapter 17: Data Synchronization 989 Overview 989 The pdflush Mechanism 991 Starting a New Thread 993 Thread Initialization 994 Performing Actual Work 995 Periodic Flushing 996 Associated Data Structures 996 Page Status 996 Writeback Control 998 Adjustable Parameters 1000 Central Control 1000 Superblock Synchronization 1002 Inode Synchronization 1003 Walking the Superblocks 1003 Examining Superblock Inodes 1003 Writing Back Single Inodes 1006 Congestion 1009 Data Structures 1009 Thresholds 1010 Setting and Clearing the Congested State 1011 Waiting on Congested Queues 1012 Forced Writeback 1013 Laptop Mode 1015 System Calls for Synchronization Control 1016 Full Synchronization 1016 Synchronization of Inodes 1018 Synchronization of Individual Files 1019 Synchronization of Memory Mappings 1021 Summary 1022 xxi Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xxii Contents Chapter 18: Page Reclaim and Swapping 1023 Overview 1023 Swappable Pages 1024 Page Thrashing 1025 Page-Swapping Algorithms 1026 Page Reclaim and Swapping in the Linux Kernel 1027 Organization of the Swap Area 1028 Checking Memory Utilization 1029 Selecting Pages to Be Swapped Out 1029 Handling Page Faults 1029 Shrinking Kernel Caches 1030 Managing Swap Areas 1030 Data Structures 1030 Creating a Swap Area 1035 Activating a Swap Area 1036 The Swap Cache 1039 Identifying Swapped-Out Pages 1041 Structure of the Cache 1044 Adding New Pages 1045 Searching for a Page 1050 Writing Data Back 1051 Page Reclaim 1052 Overview 1053 Data Structures 1055 Determining Page Activity 1057 Shrinking Zones 1062 Isolating LRU Pages and Lumpy Reclaim 1065 Shrinking the List of Active Pages 1068 Reclaiming Inactive Pages 1072 The Swap Token 1079 Handling Swap-Page Faults 1082 Swapping Pages in 1083 Reading the Data 1084 Swap Readahead 1085 Initiating Memory Reclaim 1086 Periodic Reclaim with kswapd 1087 Swap-out in the Event of Acute Memory Shortage 1090 Shrinking Other Caches 1092 Data Structures 1092 Registering and Removing Shrinkers 1093 Shrinking Caches 1093 Summary 1095 xxii Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xxiii Contents Chapter 19: Auditing 1097 Overview 1097 Audit Rules 1099 Implementation 1100 Data Structures 1100 Initialization 1106 Processing Requests 1107 Logging Events 1108 System Call Auditing 1110 Summary 1116 Appendix A: Architecture Specifics 1117 Overview 1117 Data Types 1118 Alignment 1119 Memory Pages 1119 System Calls 1120 String Processing 1120 Thread Representation 1122 IA-32 1122 IA-64 1124 ARM 1126 Sparc64 1128 Alpha 1129 Mips 1131 PowerPC 1132 AMD64 1134 Bit Operations and Endianness 1135 Manipulation of Bit Chains 1135 Conversion between Byte Orders 1136 Page Tables 1137 Miscellaneous 1137 Checksum Calculation 1137 Context Switch 1137 Finding the Current Process 1138 Summary 1139 Appendix B: Working with the Source Code 1141 Organization of the Kernel Sources 1141 Configuration with Kconfig 1144 A Sample Configuration File 1144 xxiii Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xxiv Contents Language Elements of Kconfig 1147 Processing Configuration Information 1152 Compiling the Kernel with Kbuild 1154 Using the Kbuild System 1154 Structure of the Makefiles 1156 Useful Tools 1160 LXR 1161 Patch and Diff 1163 Git 1165 Debugging and Analyzing the Kernel 1169 GDB and DDD 1170 Local Kernel 1171 KGDB 1172 User-Mode Linux 1173 Summary 1174 Appendix C: Notes on C 1175 How the GNU C Compiler Works 1175 From Source Code to Machine Program 1176 Assembly and Linking 1180 Procedure Calls 1180 Optimization 1185 Inline Functions 1192 Attributes 1192 Inline Assembler 1194 __builtin Functions 1198 Pointer Arithmetic 1200 Standard Data Structures and Techniques of the Kernel 1200 Reference Counters 1200 Pointer Type Conversions 1201 Alignment Issues 1202 Bit Arithmetic 1203 Pre-Processor Tricks 1206 Miscellaneous 1207 Doubly Linked Lists 1209 Hash Lists 1214 Red-Black Trees 1214 Radix Trees 1216 Summary 1221 xxiv Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xxv Contents Appendix D: System Startup 1223 Architecture-Specific Setup on IA-32 Systems 1224 High-Level Initialization 1225 Subsystem Initialization 1225 Summary 1239 Appendix E: The ELF Binary Format 1241 Layout and Structure 1241 ELF Header 1243 Program Header Table 1244 Sections 1246 Symbol Table 1248 String Tables 1249 Data Structures in the Kernel 1250 Data Types 1250 Headers 1251 String Tables 1257 Symbol Tables 1257 Relocation Entries 1259 Dynamic Linking 1263 Summary 1265 Appendix F: The Kernel Development Process 1267 Introduction 1267 Kernel Trees and the Structure of Development 1268 The Command Chain 1269 The Development Cycle 1269 Online Resources 1272 The Structure of Patches 1273 Technical Issues 1273 Submission and Review 1277 Linux and Academia 1281 Some Examples 1282 Adopting Research 1284 Summary 1287 References 1289 Index 1293 xxv Mauerer ftoc.tex V4 - 09/03/2008 11:13pm Page xxvi Mauerer flast.tex V2 - 09/05/2008 12:08pm Page xxvii Introduction Unix is simple and coherent, but it takes a genius (or at any rate a programmer) to understand and appreciate the simplicity. — Dennis Ritchie Note from the authors: Yes, we have lost our minds. Be forewarned: You will lose yours too. — Benny Goodheart & James Cox Unix is distinguished by a simple, coherent, and elegant design — truly remarkable features that have enabled the system to influence the world for more than a quarter of a century. And especially thanks to the growing presence of Linux, the idea is still picking up momentum, with no end of the growth in sight. Unix and Linux carry a certain fascination, and the two quotations above hopefully capture the spirit of this attraction. Consider Dennis Ritchie’s quote: Is the coinventor of Unix at Bell Labs completely right in saying that only a genius can appreciate the simplicity of Unix? Luckily not, because he puts himself into perspective immediately by adding that programmers also qualify to value the essence of Unix. Understanding the meagerly documented, demanding, and complex sources of Unix as well as of Linux is not always an easy task. But once one has started to experience the rich insights that can be gained from the kernel sources, it is hard to escape the fascination of Linux. It seems fair to warn you that it’s easy to get addicted to the joy of the operating system kernel once starting to dive into it. This was already noted by Benny Goodheart and James Cox, whose preface to their book The Magic Garden Explained (second quotation above) explained the internals of Unix System V. And Linux is definitely also capable of helping you to lose your mind! This book acts as a guide and companion that takes you through the kernel sources and sharpens your awareness of the beauty, elegance, and — last but not least — esthetics of their concepts. There are, how- ever, some prerequisites to foster an understanding of the kernel. C should not just be a letter; neither should it be a foreign language. Operating systems are supposed to be more than just a ‘‘Start” button, and a small amount of algorithmics can also do no harm. Finally, it is preferable if computer architecture is not just about how to build the most fancy case. From an academic point of view, this comes closest to the lectures ‘‘Systems Programming,” ‘‘Algorithmics,” and ‘‘Fundamentals of Operating Systems.” The pre- vious edition of this book has been used to teach the fundamentals of Linux to advanced undergraduate students in several universities, and I hope that the current edition will serve the same purpose. Discussing all aforementioned topics in detail is outside the scope of this book, and when you consider the mass of paper you are holding in your hands right now (or maybe you are not holding it, for this very reason), you’ll surely agree that this would not be a good idea. When a topic not directly related to Mauerer flast.tex V2 - 09/05/2008 12:08pm Page xxviii Introduction the kernel, but required to understand what the kernel does, is encountered in this book, I will briefly introduce you to it. To gain a more thorough understanding, however, consult the books on computing fundamentals that I recommend. Naturally, there is a large selection of texts, but some books that I found particularly insightful and illuminating include C Programming Language, by Brian W. Kernighan and Denis M. Ritchie [KR88]; Modern Operating Systems, by Andrew S. Tanenbaum [Tan07] on the basics of operating systems in general; Operating Systems: Design and Implementation, by Andrew S. Tanenbaum and Albert S. Woodhull [TW06] on Unix (Minix) in particular; Advanced Programming in the Unix Environment, by W. Richard Stevens and Stephen A. Rago [SR05] on userspace programming; and the two volumes Computer Architecture and Computer Organization and Design, on the foundations of computer architecture by John L. Hennessy and David A. Patterson [HP06, PH07]. All have established themselves as classics in the literature. Additionally, Appendix C contains some information about extensions of the GNU C compiler that are used by the kernel, but do not necessarily find widespread use in general programming. When the first edition of this book was written, a schedule for kernel releases was more or less nonexis- tent. This has changed drastically during the development of kernel 2.6, and as I discuss in Appendix F, kernel developers have become pretty good at issuing new releases at periodic, predictable intervals. I have focused on kernel 2.6.24, but have also included some references to 2.6.25 and 2.6.26, which were released after this book was written but before all technical publishing steps had been completed. Since a number of comprehensive changes to the whole kernel have been merged into 2.6.24, picking this release as the target seems a good choice. While a detail here or there will have changed in more recent kernel versions as compared to the code discussed in this book, the big picture will remain the same for quite some time. In the discussion of the various components and subsystems of the kernel, I have tried to avoid over- loading the text with unimportant details. Likewise, I have tried not to lose track of the connection with source code. It is a very fortunate situation that, thanks to Linux, we are able to inspect the source of a real, working, production operating system, and it would be sad to neglect this essential aspect of the kernel. To keep the book’s volume below the space of a whole bookshelf, I have selected only the most crucial parts of the sources. Appendix F introduces some techniques that ease reading of and working with the real source, an indispensable step toward understanding the structure and implementation of the Linux kernel. One particularly interesting observation about Linux (and Unix in general) is that it is well suited to evoke emotions. Flame wars on the Internet and heated technical debates about operating systems may be one thing, but for which other operating system does there exist a handbook (The Unix-Haters Handbook, edited by Simson Garfinkel et al. [GWS94]) on how best to hate it? When I wrote the preface to the first edition, I noted that it is not a bad sign for the future that a certain international software company responds to Linux with a mixture of abstruse accusations and polemics. Five years later, the situation has improved, and the aforementioned vendor has more or less officially accepted the fact that Linux has become a serious competitor in the operating system world. And things are certainly going to improve even more during the next five years. . . . Naturally (and not astonishingly), I admit that I am definitely fascinated by Linux (and, sometimes, am also sure that I have lost my mind because of this), and if this book helps to carry this excitement to the reader, the long hours (and especially nights) spent writing it were worth every minute! Suggestions for improvements and constrictive critique can be passed to wm@linux-kernel.net,orvia www.wrox.com. Naturally, I’m also happy if you tell me that you liked the book! xxviii Mauerer flast.tex V2 - 09/05/2008 12:08pm Page xxix Introduction What This Book Covers This book discusses the concepts, structure, and implementation of the Linux kernel. In particular, the individual chapters cover the following topics: ❑ Chapter 1 provides an overview of the Linux kernel and describes the big picture that is investi- gated more closely in the following chapters. ❑ Chapter 2 talks about the basics of multitasking, scheduling, and process management, and investigates how these fundamental techniques and abstractions are implemented. ❑ Chapter 3 discusses how physical memory is managed. Both the interaction with hardware and the in-kernel distribution of RAM via the buddy system and the slab allocator are covered. ❑ Chapter 4 proceeds to describe how userland processes experience virtual memory, and the comprehensive data structures and actions required from the kernel to implement this view. ❑ Chapter 5 introduces the mechanisms required to ensure proper operation of the kernel on multiprocessor systems. Additionally, it covers the related question of how processes can com- municate with each other. ❑ Chapter 6 walks you through the means for writing device drivers that are required to add sup- port for new hardware to the kernel. ❑ Chapter 7 explains how modules allow for dynamically adding new functionality to the kernel. ❑ Chapter 8 discusses the virtual filesystem, a generic layer of the kernel that allows for supporting a wide range of different filesystems, both physical and virtual. ❑ Chapter 9 describes the extended filesystem family, that is, the Ext2 and Ext3 filesystems that are the standard workhorses of many Linux installations. ❑ Chapter 10 goes on to discuss procfs and sysfs, two filesystems that are not designed to store information, but to present meta-information about the kernel to userland. Additionally, a num- ber of means to ease writing filesystems are presented. ❑ Chapter 11 shows how extended attributes and access control lists that can help to improve sys- tem security are implemented. ❑ Chapter 12 discusses the networking implementation of the kernel, with a specific focus on IPv4, TCP, UDP, and netfilter. ❑ Chapter 13 introduces how systems calls that are the standard way to request a kernel action from userland are implemented. ❑ Chapter 14 analyzes how kernel activities are triggered with interrupts, and presents means of deferring work to a later point in time. ❑ Chapter 15 shows how the kernel handles all time-related requirements, both with low and high resolution. ❑ Chapter 16 talks about speeding up kernel operations with the help of the page and buffer caches. ❑ Chapter 17 discusses how cached data in memory are synchronized with their sources on persis- tent storage devices. ❑ Chapter 18 introduces how page reclaim and swapping work. xxix Mauerer flast.tex V2 - 09/05/2008 12:08pm Page xxx Introduction ❑ Chapter 19 gives an introduction to the audit implementation, which allows for observing in detail what the kernel is doing. ❑ Appendix A discusses peculiarities of various architectures supported by the kernel. ❑ Appendix B walks through various tools and means of working efficiently with the kernel sources. ❑ Appendix C provides some technical notes about the programming language C, and also discusses how the GNU C compiler is structured. ❑ Appendix D describes how the kernel is booted. ❑ Appendix E gives an introduction to the ELF binary format. ❑ Appendix F discusses numerous social aspects of kernel development and the Linux kernel community. xxx Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 1 Introduction and Overview Operating systems are not only regarded as a fascinating part of information technology, but are also the subject of controversial discussion among a wide public.1 Linux has played a major role in this development. Whereas just 10 years ago a strict distinction was made between relatively simple academic systems available in source code and commercial variants with varying perfor- mance capabilities whose sources were a well-guarded secret, nowadays anybody can download the sources of Linux (or of any other free systems) from the Internet in order to study them. Linux is now installed on millions of systems and is used by home users and professionals alike for a wide range of tasks. From miniature embedded systems in wristwatches to massively parallel mainframes, there are countless ways of exploiting Linux productively. And this makes the sources so interesting. A sound, well-established concept (Unix) melded with powerful innovations and a strong penchant for dealing with problems that do not arise in academic teaching systems — this is what makes Linux so fascinating. This book describes the central functions of the kernel, explains its underlying structures, and exam- ines its implementation. Because complex subjects are discussed, I assume that the reader already has some experience in operating systems and systems programming in C (it goes without saying that I assume some familiarity with using Linux systems). I touch briefly on several general concepts relevant to common operating system problems, but my prime focus is on the implementation of the Linux kernel. Readers unfamiliar with a particular topic will find explanations on relevant basics in one of the many general texts on operating systems; for example, in Tanenbaum’s outstanding 1It is not the intention of this book to participate in ideological discussions such as whether Linux can be regarded as a full operating system, although it is, in fact, just a kernel that cannot function productively without relying on other com- ponents. When I speak of Linux as an operating system without explicitly mentioning the acronyms of similar projects (primarily the GNU project, which despite strong initial resistance regarding the kernel reacts extremely sensitively when Linux is used instead of GNU/Linux), this should not be taken to mean that I do not appreciate the importance of the work done by this project. Our reasons are simple and pragmatic. Where do we draw the line when citing those involved without generating such lengthy constructs as GNU/IBM/RedHat/HP/KDE/Linux? If this footnote makes little sense, refer to www.gnu.org/gnu/linux-and-gnu.html, where you will find a summary of the positions of the GNU project. After all ideological questions have been settled, I promise to refrain from using half-page footnotes in the rest of this book. Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 2 Chapter 1: Introduction and Overview introductions ([TW06] and [Tan07]). A solid foundation of C programming is required. Because the kernel makes use of many advanced techniques of C and, above all, of many special features of the GNU C compiler, Appendix C discusses the finer points of C with which even good programmers may not be familiar. A basic knowledge of computer structures will be useful as Linux necessarily interacts very directly with system hardware — particularly with the CPU. There are also a large number of introduc- tory works dealing with this subject; some are listed in the reference section. When I deal with CPUs in greater depth (in most cases I take the IA-32 or AMD64 architecture as an example because Linux is used predominantly on these system architectures), I explain the relevant hardware details. When I dis- cuss mechanisms that are not ubiquitous in daily live, I will explain the general concept behind them, butexpectthatreaderswillalsoconsultthequoted manual pages for more advice on how a particular feature is used from userspace. The present chapter is designed to provide an overview of the various areas of the kernel and to illustrate their fundamental relationships before moving on to lengthier descriptions of the subsystems in the following chapters. Since the kernel evolves quickly, one question that naturally comes to mind is which version is cov- ered in this book. I have chosen kernel 2.6.24, which was released at the end of January 2008. The dynamic nature of kernel development implies that a new kernel version will be available by the time you read this, and naturally, some details will have changed — this is unavoidable. If it were not the case, Linux would be a dead and boring system, and chances are that you would not want to read the book. While some of the details will have changed, concepts will not have varied essentially. This is particularly true because 2.6.24 has seen some very fundamental changes as compared to earlier versions. Developers do not rip out such things overnight, naturally. 1.1 Tasks of the Kernel On a purely technical level, the kernel is an intermediary layer between the hardware and the software. Its purpose is to pass application requests to the hardware and to act as a low-level driver to address the devices and components of the system. Nevertheless, there are other interesting ways of viewing the kernel. ❑ The kernel can be regarded as an enhanced machine that, in the view of the application, abstracts the computer on a high level. For example, when the kernel addresses a hard disk, it must decide which path to use to copy data from disk to memory, where the data reside, which commands must be sent to the disk via which path, and so on. Applications, on the other hand, need only issue the command that data are to be transferred. How this is done is irrelevant to the appli- cation — the details are abstracted by the kernel. Application programs have no contact with the hardware itself,2 only with the kernel, which, for them, represents the lowest level in the hierarchy they know — and is therefore an enhanced machine. ❑ Viewing the kernel as a resource manager is justified when several programs are run concurrently on a system. In this case, the kernel is an instance that shares available resources — CPU time, disk space, network connections, and so on — between the various system processes while at the same time ensuring system integrity. 2The CPU is an exception since it is obviously unavoidable that programs access it. Nevertheless, the full range of possible instruc- tions is not available for applications. 2 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 3 Chapter 1: Introduction and Overview ❑ Another view of the kernel is as a library providing a range of system-oriented commands. As is generally known, system calls are used to send requests to the computer; with the help of the C standard library, these appear to the application programs as normal functions that are invoked in the same way as any other function. 1.2 Implementation Strategies Currently, there are two main paradigms on which the implementation of operating systems is based: 1. Microkernels — In these, only the most elementary functions are implemented directly in a central kernel — the microkernel. All other functions are delegated to autonomous processes that communicate with the central kernel via clearly defined communication interfaces — for example, various filesystems, memory management, and so on. (Of course, the most elementary level of memory management that controls communication with the system itself is in the microkernel. However, handling on the system call level is implemented in external servers.) Theoretically, this is a very elegant approach because the individual parts are clearly segregated from each other, and this forces programmers to use ‘‘clean‘‘ programming techniques. Other benefits of this approach are dynamic extensibility and the ability to swap important components at run time. However, owing to the additional CPU time needed to support complex communication between the components, microkernels have not really established themselves in practice although they have been the subject of active and varied research for some time now. 2. Monolithic Kernels — They are the alternative, traditional concept. Here, the entire code of the kernel — including all its subsystems such as memory management, filesystems, or device drivers — is packed into a single file. Each function has access to all other parts of the kernel; this can result in elaborately nested source code if programming is not done with great care. Because, at the moment, the performance of monolithic kernels is still greater than that of microkernels, Linux was and still is implemented according to this paradigm. However, one major innovation has been introduced. Modules with kernel code that can be inserted or removed while the system is up-and-running support the dynamic addition of a whole range of functions to the kernel, thus compensating for some of the disadvantages of monolithic kernels. This is assisted by elaborate means of communication between the kernel and userland that allows for implementing hotplugging and dynamic loading of modules. 1.3 Elements of the Kernel This section provides a brief overview of the various elements of the kernel and outlines the areas we will examine in more detail in the following chapters. Despite its monolithic approach, Linux is surprisingly well structured. Nevertheless, it is inevitable that its individual elements interact with each other; they share data structures, and (for performance reasons) cooperate with each other via more functions than would be necessary in a strictly segregated system. In the following chapters, I am obliged to make frequent reference to the other elements of the kernel and therefore to other chapters, although I have tried to keep the number of forward references to a minimum. For this reason, I introduce the individual elements briefly here so that you can form an impression of their role and their place in the overall 3 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 4 Chapter 1: Introduction and Overview concept. Figure 1-1 provides a rough initial overview about the layers that comprise a complete Linux system, and also about some important subsystems of the kernel as such. Notice, however, that the individual subsystems will interact in a variety of additional ways in practice that are not shown in the figure. Applications Userspace C Library Kernel space Hardware Device driversCore kernel System Calls Networking Device Drivers FilesystemsVFS Memory mgmt Architecture specific code Process mgmt Figure 1-1: High-level overview of the structure of the Linux kernel and the layers in a complete Linux system. 1.3.1 Processes, Task Switching, and Scheduling Applications, servers, and other programs running under Unix are traditionally referred to as processes. Each process is assigned address space in the virtual memory of the CPU. The address spaces of the indi- vidual processes are totally independent so that the processes are unaware of each other — as far as each process is concerned, it has the impression of being the only process in the system. If processes want to communicate to exchange data, for example, then special kernel mechanisms must be used. Because Linux is a multitasking system, it supports what appears to be concurrent execution of several processes. Since only as many processes as there are CPUs in the system can really run at the same time, the kernel switches (unnoticed by users) between the processes at short intervals to give them the impression of simultaneous processing. Here, there are two problem areas: 1. The kernel, with the help of the CPU, is responsible for the technical details of task switch- ing. Each individual process must be given the illusion that the CPU is always available. This is achieved by saving all state-dependent elements of the process before CPU resources are withdrawn and the process is placed in an idle state. When the process is reactivated, the exact saved state is restored. Switching between processes is known as task switching. 2. The kernel must also decide how CPU time is shared between the existing processes. Impor- tant processes are given a larger share of CPU time, less important processes a smaller share. The decision as to which process runs for how long is known as scheduling. 1.3.2 UNIX Processes Linux employs a hierarchical scheme in which each process depends on a parent process. The kernel starts the init program as the first process that is responsible for further system initialization actions and display of the login prompt or (in more widespread use today) display of a graphical login interface. init is therefore the root from which all processes originate, more or less directly, as shown graphically 4 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 5 Chapter 1: Introduction and Overview by the pstree program. init is the top of a tree structure whose branches spread further and further down. wolfgang@meitner> pstree init-+-acpid |-bonobo-activati |-cron |-cupsd |-2*[dbus-daemon] |-dbus-launch |-dcopserver |-dhcpcd |-esd |-eth1 |-events/0 |-gam_server |-gconfd-2 |-gdm---gdm-+-X | ‘-startkde-+-kwrapper | ‘-ssh-agent |-gnome-vfs-daemo |-gpg-agent |-hald-addon-acpi |-kaccess |-kded |-kdeinit-+-amarokapp---2*[amarokapp] | |-evolution-alarm | |-kinternet | |-kio_file | |-klauncher | |-konqueror | |-konsole---bash-+-pstree | | ‘-xemacs | |-kwin | |-nautilus | ‘-netapplet |-kdesktop |-kgpg |-khelper |-kicker |-klogd |-kmix |-knotify |-kpowersave |-kscd |-ksmserver |-ksoftirqd/0 |-kswapd0 |-kthread-+-aio/0 | |-ata/0 | |-kacpid | |-kblockd/0 | |-kgameportd | |-khubd 5 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 6 Chapter 1: Introduction and Overview | |-kseriod | |-2*[pdflush] | ‘-reiserfs/0 ... How this tree structure spreads is closely connected with how new processes are generated. For this purpose, Unix uses two mechanisms called fork and exec. 1. fork — Generates an exact copy of the current process that differs from the parent process only in its PID (process identification). After the system call has been executed, there are two processes in the system, both performing the same actions. The memory contents of the ini- tial process are duplicated — at least in the view of the program. Linux uses a well-known technique known as copy on write that allows it to make the operation much more efficient by deferring the copy operations until either parent or child writes to a page — read-only accessed can be satisfied from the same page for both. A possible scenario for using fork is, for example, when a user opens a second browser win- dow. If the corresponding option is selected, the browser executes a fork to duplicate its code and then starts the appropriate actions to build a new window in the child process. 2. exec — Loads a new program into an existing content and then executes it. The memory pages reserved by the old program are flushed, and their contents are replaced with new data. The new program then starts executing. Threads Processes are not the only form of program execution supported by the kernel. In addition to heavy-weight processes — another name for classical Unix processes — there are also threads, sometimes referred to as light-weight processes. They have also been around for some time, and essentially, a process may consist of several threads that all share the same data and resources but take different paths through the program code. The thread concept is fully integrated into many modern languages — Java, for instance. In simple terms, a process can be seen as an executing program, whereas a thread is a program function or routine running in parallel to the main program. This is useful, for example, when Web browsers need to load several images in parallel. Usually, the browser would have to execute several fork and exec calls to generate parallel instances; these would then be responsible for loading the images and making data received available to the main program using some kind of communication mechanisms. Threads make this situation easier to handle. The browser defines a routine to load images, and the routine is started as a thread with multiple strands (each with different arguments). Because the threads and the main program share the same address space, data received automatically reside in the main program. There is therefore no need for any communication effort whatsoever, except to prevent the threads from stepping onto their feet mutually by accessing identical memory locations, for instance. Figure 1-2 illustrates the difference between a program with and without threads. W/O Threads With Threads Address Space Control Flow Figure 1-2: Processes with and without threads. 6 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 7 Chapter 1: Introduction and Overview Linux provides the clone method to generate threads. This works in a similar way to fork but enables a precise check to be made of which resources are shared with the parent process and which are generated independently for the thread. This fine-grained distribution of resources extends the classical thread concept and allows for a more or less continuous transition between thread and processes. Namespaces During the development of kernel 2.6, support for namespaces was integrated into numerous subsystems. This allows different processes to have different views of the system. Traditionally, Linux (and Unix in general) use numerous global quantities, for instance, process identifiers: Every process in the system is equipped with a unique identifier (ID), and this ID can be employed by users (or other processes) to refer to the process — by sending it a signal, for instance. With namespaces, formerly global resources are grouped differently: Every namespace can contain a specific set of PIDs, or can provide different views of the filesystem, where mounts in one namespace do not propagate into different namespaces. Namespaces are useful; for example, they are beneficial for hosting providers: Instead of setting up one physical machine per customer, they can instead use containers implemented with namespaces to create multiple views of the system where each seems to be a complete Linux installation from within the container and does not interact with other containers: They are separated and segregated from each other. Every instance looks like a single machine running Linux, but in fact, many such instances can operate simultaneously on a physical machine. This helps use resources more effectively. In contrast to full virtualization solutions like KVM, only a single kernel needs to run on the machine and is responsible to manage all containers. Not all parts of the kernel are yet fully aware of namespaces, and I will discuss to what extent support is available when we analyze the various subsystems. 1.3.3 Address Spaces and Privilege Levels Before we start to discuss virtual address spaces, there are some notational conventions to fix. Through- out this book I use the abbreviations KiB, MiB, and GiB as units of size. The conventional units KB, MB, and GB are not really suitable in information technology because they represent decimal powers  103 , 106,and109  although the binary system is the basis ubiquitous in computing. Accordingly KiB stands for 210,MiBfor220,andGiBfor230 bytes. Because memory areas are addressed by means of pointers, the word length of the CPU determines the maximum size of the address space that can be managed. On 32-bit systems such as IA-32, PPC, and m68k, these are 232 = 4 GiB, whereas on more modern 64-bit processors such as Alpha, Sparc64, IA-64, and AMD64, 264 bytes can be managed. The maximal size of the address space is not related to how much physical RAM is actually available, and therefore it is known as the virtual address space. One more reason for this terminology is that every process in the system has the impression that it would solely live in this address space, and other pro- cesses are not present from their point of view. Applications do not need to care about other applications and can work as if they would run as the only process on the computer. Linux divides virtual address space into two parts known as kernel space and userspace as illustrated in Figure 1-3. 7 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 8 Chapter 1: Introduction and Overview 0 TASK_SIZE 232 respectively 264 Userspace Kernel- space Figure 1-3: Division of virtual address space. Every user process in the system has its own virtual address range that extends from 0 to TASK_SIZE. The area above (from TASK_SIZE to 232 or 264) is reserved exclusively for the kernel — and may not be accessed by user processes. TASK_SIZE is an architecture-specific constant that divides the address space in a given ratio — in IA-32 systems, for instance, the address space is divided at 3 GiB so that the virtual address space for each process is 3 GiB; 1 GiB is available to the kernel because the total size of the virtual address space is 4 GiB. Although actual figures differ according to architecture, the general concepts do not. I therefore use these sample values in our further discussions. This division does not depend on how much RAM is available. As a result of address space virtualization, each user process thinks it has 3 GiB of memory. The userspaces of the individual system processes are totally separate from each other. The kernel space at the top end of the virtual address space is always the same, regardless of the process currently executing. Notice that the picture can be more complicated on 64-bit machines because these tend to use less than 64 bits to actually manage their huge principal virtual address space. Instead of 64 bits, they employ a smaller number, for instance, 42 or 47 bits. Because of this, the effectively addressable portion of the address space is smaller than the principal size. However, it is still larger than the amount of RAM that will ever be present in the machine, and is therefore completely sufficient. As an advantage, the CPU can save some effort because less bits are required to manage the effective address space than are required to address the complete virtual address space. The virtual address space will contain holes that are not addressable in principle in such cases, so the simple situation depicted in Figure 1-3 is not fully valid. We will come back to this topic in more detail in Chapter 4. Privilege Levels The kernel divides the virtual address space into two parts so that it is able to protect the individual system processes from each other. All modern CPUs offer several privilege levels in which processes can reside. There are various prohibitions in each level including, for example, execution of certain assembly language instructions or access to specific parts of virtual address space. The IA-32 architecture uses a system of four privilege levels that can be visualized as rings. The inner rings are able to access more functions, the outer rings less, as shown in Figure 1-4. Whereas the Intel variant distinguishes four different levels, Linux uses only two different modes — kernel mode and user mode. The key difference between the two is that access to the memory area above TASK_SIZE — that is, kernel space — is forbidden in user mode. User processes are not able to manipulate or read the data in kernel space. Neither can they execute code stored there. This is the sole domain 8 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 9 Chapter 1: Introduction and Overview of the kernel. This mechanism prevents processes from interfering with each other by unintentionally influencing each other’s data. 1 0 2 3 Kernel- mode User- mode Less Privileges IA-32 Linux Figure 1-4: Ring system of privilege levels. The switch from user to kernel mode is made by means of special transitions known as system calls;these are executed differently depending on the system. If a normal process wants to carry out any kind of action affecting the entire system (e.g., manipulating I/O devices), it can do this only by issuing a request to the kernel with the help of a system call. The kernel first checks whether the process is permitted to perform the desired action and then performs the action on its behalf. A return is then made to user mode. Besides executing code on behalf of a user program, the kernel can also be activated by asynchronous hardware interrupts, and is then said to run in interrupt context. The main difference to running in process context is that the userspace portion of the virtual address space must not be accessed. Because interrupts occur at random times, a random userland process is active when an interrupt occurs, and since the interrupt will most likely be unconnected with the cause of the interrupt, the kernel has no business with the contents of the current userspace. When operating in interrupt context, the kernel must be more cautious than normal; for instance, it must not go to sleep. This requires extra care when writing interrupt handlers and is discussed in detail in Chapter 2. An overview of the different execution contexts is given in Figure 1-5. Besides normal processes, there can also be kernel threads running on the system. Kernel threads are also not associated with any particular userspace process, so they also have no business dealing with the user portion of the address space. In many other respects, kernel threads behave much more like regular userland applications, though: In contrast to a kernel operating in interrupt context, they may go to sleep, and they are also tracked by the scheduler like every regular process in the system. The kernel uses them for various purposes that range from data synchronization of RAM and block devices to helping the scheduler distribute processes among CPUs, and we will frequently encounter them in the course of this book. Notice that kernel threads can be easily identified in the output of ps because their names are placed inside brackets: wolfgang@meitner> ps fax PID TTY STAT TIME COMMAND 2 ? S< 0:00 [kthreadd] 3 ? S< 0:00 _ [migration/0] 4 ? S< 0:00 _ [ksoftirqd/0] 9 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 10 Chapter 1: Introduction and Overview 5 ? S< 0:00 _ [migration/1] 6 ? S< 0:00 _ [ksoftirqd/1] 7 ? S< 0:00 _ [migration/2] 8 ? S< 0:00 _ [ksoftirqd/2] 9 ? S< 0:00 _ [migration/3] 10 ? S< 0:00 _ [ksoftirqd/3] 11 ? S< 0:00 _ [events/0] 12 ? S< 0:00 _ [events/1] 13 ? S< 0:00 _ [events/2] 14 ? S< 0:00 _ [events/3] 15 ? S< 0:00 _ [khelper] ... 15162 ? S< 0:00 _ [jfsCommit] 15163 ? S< 0:00 _ [jfsSync] System call Return from system call Must not be accessedUser Kernel Interrupt Arrows indicate that CPU executes here() Figure 1-5: Execution in kernel and user mode. Most of the time, the CPU executes code in userspace. When the application performs a system call, a switch to kernel mode is employed, and the kernel fulfills the request. During this, it may access the user portion of the virtual address space. After the system call completes, the CPU switches back to user mode. A hardware interrupt also triggers a switch to kernel mode, but this time, the userspace portion must not be accessed by the kernel. On multiprocessor systems, many threads are started on a per-CPU basis and are restricted to run on only one specific processor. This is represented by a slash and the number of the CPU that are appended to the name of the kernel thread. Virtual and Physical Address Spaces In most cases, a single virtual address space is bigger than the physical RAM available to the system. And the situation does not improve when each processhasitsownvirtualaddressspace.ThekernelandCPU must therefore consider how the physical memory actually available can be mapped onto virtual address areas. The preferred method is to use page tables to allocate virtual addresses to physical addresses. Whereas virtual addresses relate to the combined user and kernel space of a process, physical addresses are used to address the RAM actually available. This principle is illustrated in Figure 1-6. The virtual address spaces of both processes shown in the figure are divided into portions of equal size by the kernel. These portions are known as pages. Physical memory is also divided into pages of the same size. 10 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 11 Chapter 1: Introduction and Overview Process A RAM Process B Page Frame Figure 1-6: Virtual and physical addresses. The arrows in Figure 1-6 indicate how the pages in the virtual address spaces are distributed across the physical pages. For example, virtual page 1 of process A is mapped to physical page 4, while virtual page 1 of process B is mapped to the fifth physical page. This shows that virtual addresses change their meaning from process to process. Physical pages are often called page frames. In contrast, the term page is reserved for pages in virtual address space. Mapping between virtual address spaces and physical memory also enables the otherwise strict sep- aration between processes to be lifted. Our example includes a page frame explicitly shared by both processes. Page 5 of A and page 1 of B both point to the physical page frame 5. This is possible because entries in both virtual address spaces (albeit at different positions) point to the same page. Since the ker- nel is responsible for mapping virtual address space to physical address space, it is able to decide which memory areas are to be shared between processes and which are not. The figure also shows that not all pages of the virtual address spaces are linked with a page frame. This may be because either the pages are not used or because data have not been loaded into memory because they are not yet needed. It may also be that the page has been swapped out onto hard disk and will be swapped back in when needed. Finally, notice that there are two equivalent terms to address the applications that run on behalf of the user. One of them is userland, and this is the nomenclature typically preferred by the BSD community for all things that do not belong to the kernel. The alternative is to say that an application runs in userspace.It should be noted that the term userland will always mean applications as such, whereas the term userspace can additionally not only denote applications, but also the portion of the virtual address space in which they are executed, in contrast to kernel space. 1.3.4 Page Tables Data structures known as page tables are used to map virtual address space to physical address space. The easiest way of implementing the association between both would be to use an array containing an entry for each page in virtual address space. This entry would point to the associated page frame. But there is a problem. IA-32 architecture uses, for example, 4 KiB pages — given a virtual address space of 4 GiB, this would produce an array with a million entries. On 64-bit architectures, the situation is much worse. Because each process needs its own page tables, this approach is impractical because the entire RAM of the system would be needed to hold the page tables. 11 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 12 Chapter 1: Introduction and Overview As most areas of virtual address spaces are not used and are therefore not associated with page frames, a far less memory-intensive model that fulfills the same purpose can be used: multilevel paging. To reduce the size of page tables and to allow unneeded areas to be ignored, the architectures split each virtual address into multiple parts, as shown in Figure 1-7 (the bit positions at which the address is split differ according to architecture, but this is of no relevance here). In the example, I use a split of the virtual address into four components, and this leads to a three-level page table. This is whatmostarchitectures offer. However, some employ four-level page tables, and Linux also adopts four levels of indirection. To simplify the picture, I stick to a three-level variant here. PGD PTEPMD Offset Global Page Table + Middle Page Table Page Table Virtual Address + + Page Frame + Figure 1-7: Splitting a virtual address. The first part of the virtual address is referred to as a page global directory or PGD. It is used as an index in an array that exists exactly once for each process. Its entries are pointers to the start of further arrays called page middle directories or PMD. Once the corresponding array has been found by reference to the PGD and its contents, the PMD is used as an index for the array. The page middle directory likewise consists of pointers to further arrays known as page tables or page directories. The PTE (or page table entry) part of the virtual address is used as an index to the page table. Mapping between virtual pages and page frames is achieved because the page table entries point to page frames. The last part of the virtual address is known as an offset. It is used to specify a byte position within the page; after all, each address points to a uniquely defined byte in address space. A particular feature of page tables is that no page middle tables or page tables need be created for areas of virtual address space that are not needed. This saves a great deal of RAM as compared to the single-array method. Of course, this method also has a downside. Each time memory is accessed, it is necessary to run through the entire chain to obtain the physical address from the virtual address. CPUs try to speed up this process in two ways: 1. A special part of the CPU known as a memory management unit (MMU) is optimized to per- form referencing operations. 12 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 13 Chapter 1: Introduction and Overview 2. The addresses that occur most frequently in address translation are held in a fast CPU cache called a Translation Lookaside Buffer (TLB). Translation is accelerated because the address data in the cache are immediately available without needing to access the page tables and there- fore the RAM. While caches are operated transparently on many architectures, some require special attention from the kernel, which especially implies that their contents must be invalidated whenever the contents of the page tables have been changed. Corresponding calls must be present in every part of the kernel that manipulates page tables. If the kernel is compiled for an architecture that does not require such operations, it automatically ensures that the calls are represented by do-nothing operations. Interaction with the CPU The IA-32 architecture uses a two-level-only method to map virtual addresses to physical addresses. The size of the address space in 64-bit architectures (Alpha, Sparc64, IA-64, etc.) mandates a three-level or four-level method, and the architecture-independent part of the kernel always assumes a four-level page table. The architecture-dependent code of the kernel for two- and three-level CPUs must therefore emulate the missing levels by dummy page tables. Consequently, the remaining memory management code can be implemented independently of the CPU used. Memory Mappings Memory mappings are an important means of abstraction. They are used at many points in the kernel and are also available to user applications. Mapping is the method by which data from an arbitrary source are transferred into the virtual address space of a process. The address space areas in which mapping takes place can be processed using normal methods in the same way as regular memory. However, any changes made are transferred automatically to the original data source. This makes it possible to use identical functions to process totally different things. For example, the contents of a file can be mapped into memory. A process then need only read the contents of memory to access the contents of the file, or write changes to memory in order to modify the contents of the file. The kernel automatically ensures that any changes made are implemented in the file. Mappings are also used directly in the kernel when implementing device drivers. The input and output areas of peripheral devices can be mapped into virtual address space; reads and writes to these areas are then redirected to the devices by the system, thus greatly simplifying driver implementation. 1.3.5 Allocation of Physical Memory When it allocates RAM, the kernel must keep track of which pages have already been allocated and which are still free in order to prevent two processes from using the same areas in RAM. Because memory allocation and release are very frequent tasks, the kernel must also ensure that they are completed as quickly as possible. The kernel can allocate only whole page frames. Dividing memory into smaller portions is delegated to the standard library in userspace. This library splits the page frames received from the kernel into smaller areas and allocates memory to the processes. 13 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 14 Chapter 1: Introduction and Overview The Buddy System Numerous allocation requests in the kernel must be fulfilled by a continuous range of pages. To quickly detect where in memory such ranges are still available, the kernel employs an old, but proven technique: The buddy system. Free memory blocks in the system are always grouped as two buddies. The buddies can be allocated independently of each other; if, however, both remain unused at the same time, the kernel merges them into a larger pair that serves as a buddy on the next level. Figure 1-8 demonstrates this using an example of a buddy pair consisting initially of two blocks of 8 pages. 24 20 23 22 21 24 20 23 22 21 24 20 23 22 21 Allocated Allocated Figure 1-8: The buddy system. All buddies of the same size (1, 2, 4, 8, 16, ... pages) are managed by the kernel in a special list. The buddy pair with two times 8 (16) pages is also in this list. If the system now requires 8 page frames, it splits the block consisting of 16 page frames into two buddies. While one of the blocks is passed to the application that requested memory, the remaining 8 page frames are placed in the list for 8-page memory blocks. If the next request requires only 2 contiguous page frames, the block consisting of 8 blocks is split into 2 buddies, each comprising 4 page frames. One of the blocks is put back into the buddy lists, while the other is again split into 2 buddies consisting of 2 blocks of two pages. One is returned to the buddy system, while the other is passed to the application. 14 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 15 Chapter 1: Introduction and Overview When memory is returned by the application, the kernel can easily see by reference to the addresses whether a buddy pair is reunited and can then merge it into a larger unit that is put back into the buddy list — exactly the reverse of the splitting process. This increases the likelihood that larger memory blocks are available. When systems run for longer periods — it is not unusual for servers to run for several weeks or even months, and many desktop systems also tend to reach long uptime — a memory management problem known as fragmentation occurs. The frequent allocation and release of page frames may lead to a situation in which several page frames are free in the system but they are scattered throughout physical address space — in other words, there are no larger contiguous blocks of page frames, as would be desirable for performance reasons. This effect is reduced to some extent by the buddy system but not completely eliminated. Single reserved pages that sit in the middle of an otherwise large continuous free range can eliminate coalescing of this range very effectively. During the development of kernel 2.6.24, some effec- tive measures were added to prevent memory fragmentation, and I discuss the underlying mechanisms in more detail in Chapter 3. The Slab Cache Often the kernel itself needs memory blocks much smaller than a whole page frame. Because it cannot use the functions of the standard library, it must define its own, additional layer of memory management that builds on the buddy system and divides the pages supplied by the buddy system into smaller portions. The method used not only performs allocation but also implements a generic cache for frequently used small objects; this cache is known as a slab cache. It can be used to allocate memory in two ways: 1. For frequently used objects, the kernel defines its own cache that contains only instances of the desired type. Each time one of the objects is required, it can be quickly removed from the cache (and returned there after use); the slab cache automatically takes care of interaction with the buddy system and requests new page frames when the existing caches are full. 2. For the general allocation of smaller memory blocks, the kernel defines a set of slab caches for various object sizes that it can access using the same functions with which we are familiar from userspace programming; a prefixed k indicates that these functions are associated with the kernel: kmalloc and kfree. While the slab allocator provides good performance across a wide range of workloads, some scalability problems with it have arisen on really large supercomputers. On the other hand of the scale, the overhead of the slab allocator may be too much for really tiny embedded systems. The kernel comes with two drop- in replacements for the slab allocator that provide better performance in these use cases, but offer the same interface to the rest of the kernel such that it need not be concerned with which low-level allocator is actually compiled in. Since slab allocation is still the standard methods of the kernel, I will, however, not discuss these alternatives in detail. Figure 1-9 summarizes the connections between buddy system, slab allocator, and the rest of the kernel. Swapping and Page Reclaim Swapping enables available RAM to be enlarged virtually by using disk space as extended memory. Infrequently used pages can be written to hard disk when the kernel requires more RAM. Once the data 15 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 16 Chapter 1: Introduction and Overview are actually needed, the kernel swaps them back into memory. The concept of page faults is used to make this operation transparent to applications. Swapped-out pages are identified by a special entry in the page table. When a process attempts to access a page of this kind, the CPU initiates a page fault that is intercepted by the kernel. The kernel then has the opportunity to swap the data on disk into RAM. The user process then resumes. Because it is unaware of the page fault, swapping in and out of the page is totally invisible to the process. Generic kernel code Buddy allocator Small boxes indicate page frames Slab allocator Figure 1-9: Page frame allocation is performed by the buddy system, while the slab allocator is responsible for small-sized allocations and generic kernel caches. Page reclaim is used to synchronize modified mappings with underlying block devices — for this reason, it is sometimes referred to simply as writing back data. Once data have been flushed, the page frame can be used by the kernel for other purposes (as with swapping). After all, the kernel data structures contain all the information needed to find the corresponding data on the hard disk when they are again required. 1.3.6 Timing The kernel must be capable of measuring time and time differences at various points — when scheduling processes, for example. Jiffies are one possible time base. A global variable named jiffies_64 and its 32-bit counterpart jiffies are incremented periodically at constant time intervals. The various timer mechanisms of the underlying architectures are used to perform these updates — each computer archi- tecture provides some means of executing periodic actions, usually in the form of timer interrupts. Depending on architecture, jiffies is incremented with a frequency determined by the central constant HZ of the kernel. This is usually on the range between 1,000 and 100; in other words, the value of jiffies is incremented between 1,000 and 100 times per second. Timing based on jiffies is relatively coarse-grained because 1,000 Hz is not an excessively large fre- quency nowadays. With high-resolution timers, the kernel provides additional means that allows for keeping time in the regime of nanosecond precision and resolution, depending on the capabilities of the underlying hardware. It is possible to make the periodic tick dynamic. When there is little to do and no need for frequent periodic actions, it does not make sense to periodically generate timer interrupts that prevent the processor from powering down into deep sleep states. This is helpful in systems where power is scarce, for instance, laptops and embedded systems. 16 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 17 Chapter 1: Introduction and Overview 1.3.7 System Calls System calls are the classical method of enabling user processes to interact with the kernel. The POSIX standard defines a number of system calls and their effect as implemented on all POSIX-compliant sys- tems including Linux. Traditional system calls are grouped into various categories: ❑ Process Management — Creating new tasks, querying information, debugging ❑ Signals — Sending signals, timers, handling mechanisms ❑ Files — Creating, opening, and closing files, reading from and writing to files, querying infor- mation and status ❑ Directories and Filesystem — Creating, deleting, and renaming directories, querying informa- tion, links, changing directories ❑ Protection Mechanisms — Reading and changing UIDs/GIDs, and namespace handling ❑ Timer Functions — Timer functions and statistical information Demands are placed on the kernel in all these functions. They cannot be implemented in a normal user library because special protection mechanisms are needed to ensure that system stability and/or security are not endangered. In addition, many calls are reliant on kernel-internal structures or functions to yield desired data or results — this also dictates against implementation in userspace. When a system call is issued, the processor must change the privilege level and switch from user mode to system mode. There is no standardized way of doing this in Linux as each hardware platform offers specific mechanisms. In some cases, different approaches are implemented on the same architecture but depend on processor type. Whereas Linux uses a special software interrupt to execute system calls on IA-32 processors, the software emulation (iBCS emulator) of other Unix systems on IA-32 employs a different method to execute binary programs (for assembly language aficionados: the lcall7 or lcall27 gate). Modern variants of IA-32 also have their own assembly language statement for executing system calls; this was not available on old systems and cannot therefore be used on all machines. What all variants have in common is that system calls are the only way of enabling user processes to switch in their own incentive from user mode to kernel mode in order to delegate system-critical tasks. 1.3.8 Device Drivers, Block and Character Devices The role of device drivers is to communicate with I/O devices attached to the system; for example, hard disks, floppies, interfaces, sound cards, and so on. In accordance with the classical Unix maxim that ‘‘everything is a file,’’ access is performed using device files that usually reside in the /dev directory and can be processed by programs in the same way as regular files. The task of a device driver is to support application communication via device files; in other words, to enable data to be read from and written to a device in a suitable way. Peripheral devices belong to one of the following two groups: 1. Character Devices — Deliver a continuous stream of data that applications read sequen- tially; generally, random access is not possible. Instead, such devices allow data to be read and written byte-by-byte or character-by-character. Modems are classical examples of char- acter devices. 17 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 18 Chapter 1: Introduction and Overview 2. Block Devices — Allow applications to address their data randomly and to freely select the position at which they want to read data. Typical block devices are hard disks because appli- cations can address any position on the disk from which to read data. Also, data can be read or written only in multiples of block units (usually 512 bytes); character-based addressing, as in character devices, is not possible. Programming drivers for block devices is much more complicated than for character devices because extensive caching mechanisms are used to boost system performance. 1.3.9 Networks Network cards are also controlled by device drivers but assume a special status in the kernel because they cannot be addressed using device files. This is because data are packed into various protocol layers during network communication. When data are received, the layers must be disassembled and analyzed by the kernel before the payload data are passed to the application. When data are sent, the kernel must first pack the data into the various protocol layers prior to dispatch. However, to support work with network connections via the file interface (in the view of applications), Linux uses sockets from the BSD world; these act as agents between the application, file interface, and network implementation of the kernel. 1.3.10 Filesystems Linux systems are made up of many thousands or even millions of files whose data are stored on hard disks or other block devices (e.g., ZIP drives, floppies, CD-ROMs, etc.). Hierarchical filesystems are used; these allow stored data to be organized into directory structures and also have the job of linking other meta-information (owners, access rights, etc.) with the actual data. Many different filesystem approaches are supported by Linux — the standard filesystems Ext2 and Ext3, ReiserFS, XFS, VFAT (for reasons of compatibility with DOS), and countless more. The concepts on which they build differ drastically in part. Ext2 is based on inodes, that is, it makes a separate management structure known as an inode available on disk for each file. The inode contains not only all meta-information but also pointers to the associated data blocks. Hierarchical structures are set up by representing directories as regular files whose data section includes pointers to the inodes of all files contained in the directory. In contrast, ReiserFS makes extensive use of tree structures to deliver the same functionality. The kernel must provide an additional software layer to abstract the special features of the various low- level filesystems from the application layer (and also from the kernel itself). This layer is referred to as the VFS (virtual filesystem or virtual filesystem switch). It acts as an interface downward (this interface must be implemented by all filesystems) and upward (for system calls via which user processes are ultimately able to access filesystem functions). This is illustrated in Figure 1-10. 1.3.11 Modules and Hotplugging Modules are used to dynamically add functionality to the kernel at run time — device drivers, filesys- tems, network protocols, practically any subsystem3 of the kernel can be modularized. This removes one of the significant disadvantages of monolithic kernels as compared with microkernel variants. 3With the exception of basic functions, such as memory management, which are always needed. 18 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 19 Chapter 1: Introduction and Overview Modules can also be unloaded from the kernel at run time, a useful aspect when developing new kernel components. Virtual file system System calls ExtN Block layer Device drivers XFS ProcFS Page cache Buffer cache Applications and Libc Various subsystems Hard disks Figure 1-10: Overview of how the virtual filesystem layer, filesystem implementations, and the block layer interoperate. Basically, modules are simply normal programs that execute in kernel space rather than in userspace. They must also provide certain sections that are executed when the module is initialized (and terminated) in order to register and de-register the module functions with the kernel. Otherwise, module code has the same rights (and obligations) as normal kernel code and can access all the same functions and data as code that is permanently compiled into the kernel. Modules are an essential requisite to support for hotplugging. Some buses (e.g., USB and FireWire) allow devices to be connected while the system is running without requiring a system reboot. When the sys- tem detects a new device, the requisite driver can be automatically added to the kernel by loading the corresponding module. Modules also enable kernels to be built to support all kinds of devices that the kernel can address without unnecessarily bloating kernel size. Once attached hardware has been detected, only the requisite modules are loaded, and the kernel remains free of superfluous drivers. A long-standing issue in the kernel community revolves around the support of binary-only modules, that is, modules for which no source code is available. While binary-only modules are omnipresent on most proprietary operating systems, many kernel developers see them (at least!) as an incarnation of the devil: Since the kernel is developed as open-source software, they believe that modules should also be published as open source, for a variety of both legal and technical reasons. There are, indeed, strong arguments to support this reasoning (and besides, I also follow these), but they are not shared by some commercial companies that tend to think that opening up their driver sources would weaken their business position. It is currently possible to load binary-only modules into the kernel, although numerous restrictions apply for them. Most importantly, they may not access any functions that are explicitly only made available to GPL-licensed code. Loading a binary-only module taints the kernel, and whenever something bad occurs, the fault is naturally attributed to the tainting module. If a kernel is tainted, this will be marked on crash dumps, for instance, and kernel developers will be very unsupportive in solving the issue that led to the crash — since the binary module could have given every part of the kernel a good shaking, it cannot 19 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 20 Chapter 1: Introduction and Overview be assumed that the kernel still works as intended, and support is better left to the manufacturer of the offending module. Loading binary-only modules is not the only possibility for tainting a kernel. This happens also when, for instance, the machine has experienced certain bad exceptions, when a SMP system is built with CPUs that do not officially support multiprocessing by their specification, and other similar reasons. 1.3.12 Caching The kernel uses caches to improve system performance. Data read from slow block devices are held in RAM for a while, even if they are no longer needed at the time. When an application next accesses the data, they can be read from fast RAM, thus bypassing the slow block device. Because the kernel implements access to block devices by means of page memory mappings, caches are also organized into pages, that is, whole pages are cached, thus giving rise to the name page cache. The far less important buffer cache is used to cache data that are not organized into pages. On traditional Unix systems, the buffer cache serves as the main system cache, and the same approach was used by Linux a long, long time ago. By now, the buffer cache has mostly been superseded by the page cache. 1.3.13 List Handling A recurring task in C programs is the handling of doubly linked lists. The kernel too is required to handle such lists. Consequently, I will make frequent mention of the standard list implementation of the kernel in the following chapters. At this point, I give a brief introduction to the list handling API. Standard lists as provided by the kernel can be used to link data structures of any type with each other. It is explicitly not type-safe. The data structures to be listed must contain an element of the list_head type; this accommodates the forward and back pointers. If a data structure is to be organized in several lists — and this is not unusual — several list_head elements are needed. struct list_head { struct list_head *next, *prev; }; This element could be placed in a data structure as follows: struct task_struct { ... struct list_head run_list; ... }; The starting point for linked lists is again an instance of list_head that is usually declared and initial- ized by the LIST_HEAD(list_name) macro. In this way, the kernel produces a cyclic list, as shown in Figure 1-11. It permits access to the first and last element of a list in O(1), that is, in always the same, constant time regardless of the list size. 20 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 21 Chapter 1: Introduction and Overview next prev next prev next prev next prev Figure 1-11: Doubly linked standard list. struct list_head is called a list element when it is held in a data structure. An element that serves as the starting point for a list is called a list head. Pointers that connect head and tail elements of a list tend to clutter up images and often obstruct the principal intention of a figure, namely, to briefly summarize the connections of various kernel data structures. I thus usually omit the connection between list head and list tail in figures. The above list is in the remainder of this book therefore represented as shown in Figure 1-12. This allows for concentrating on the essential details without having to waste space for irrelevant list pointers. Figure 1-12: Simplified illustration of a doubly linked list. Notice that the connection between list head and list tail is not displayed, although it is present in kernel memory. There are several standard functions for handling and processing lists. We will come across them again and again in the following chapters (the data type of their arguments is struct list_head). ❑ list_add(new, head) inserts new right after the existing head element. ❑ list_add_tail(new, head) inserts new right before the element specified by head. If the list head is specified for head, this causes the new element to be inserted at the end of the list because of the cyclic nature of the list (this gives the function its name). ❑ list_del(entry) deletes an entry from a list. ❑ list_empty(head) checks if a list is empty, that is, if it does not contain any elements. ❑ list_splice(list, head) combines two lists by inserting the list in list after the head element of an existing list. ❑ list_entry must be used to find a list element; at first glance, its call syntax appears to be quite complicated: list_entry(ptr, type, member). ptr is a pointer to the list_head instance of the 21 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 22 Chapter 1: Introduction and Overview data structure, type is its type, and member is the element name used for the list element. The following sample call would be needed to find a task_struct instance of a list: struct task_struct = list_entry(ptr, struct task_struct, run_list) Explicit type specification is required because list implementation is not type-safe. The list ele- ment must be specified to find the correct element if there are data structures that are included in several lists.4 ❑ list_for_each(pos, head) must be used to iterate through all elements of a list. pos indicates the current position in the list, while head specifies the list head. struct list_head *p; list_for_each(p, &list) if (condition) return list_entry(p, struct task_struct, run_list); return NULL; 1.3.14 Object Management and Reference Counting All over the kernel, the need to keep track of instances of C structures arises. Despite the fact that these objects will be used in the most varying forms, some operations are very similar across subsystems — just consider reference counting. This leads to code duplication. Since this is a bad thing, the kernel has adopted generic methods to manage kernel objects during the development of 2.5. The framework is, however, not just required to prevent code duplication. It also allows for providing a coherent view on objects managed by different parts of the kernel, and this information can be brought to good use in many parts of the kernel, for instance, for power management. The generic kernel object mechanism can be used to perform the following operations on objects: ❑ Reference counting ❑ Management of lists (sets) of objects ❑ Locking of sets ❑ Exporting object properties into userspace (via the sysfs filesystem) Generic Kernel Objects The following data structure that is embedded in other data structures is used as a basis. struct kobject { const char * k_name; struct kref kref; struct list_head entry; struct kobject * parent; struct kset * kset; struct kobj_type * ktype; struct sysfs_dirent * sd; }; 4Even if there is only one list element in the structure, this entry is used to find the correct start address of the instance by means of pointer arithmetic; the address is translated into the required data type by means of type conversion. I deal with this in more detail in the appendix on C programming. 22 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 23 Chapter 1: Introduction and Overview It is essential that kobjects are not linked with other data structures by means of pointers but are directly embedded. Managing the kernel object itself amounts to managing the whole containing object this way. Since struct kobject is embedded into many data structures of the kernel, the developers take care to keep it small. Adding a single new element to this data structure results in a size increase of many other data structures. Embedded kernel objects look as follows: struct sample { ... struct kobject kobj; ... }; The meanings of the individual elements of struct kobject are as follows: ❑ k_name is a text name exported to userspace using sysfs. Sysfs is a virtual filesystem that allows for exporting various properties of the system into userspace. Likewise sd supports this connec- tion, and I will come back to this in Chapter 10. ❑ kref holds the general type struct kref designed to simplify reference management. I discuss this below. ❑ entry is a standard list element used to group several kobjects in a list (known as a set in this case). ❑ kset is required when an object is grouped with other objects in a set. ❑ parent is a pointer to the parent element and enables a hierarchical structure to be established between kobjects. ❑ ktype provides more detailed information on the data structure in which a kobject is embedded. Of greatest importance is the destructor function that returns the resources of the embedding data structure. The similarity between the name kobject and the object concept of, well, object-oriented languages like C++ or Java is by no means coincidental: The kobject abstraction indeed allows for using object- oriented techniques in the kernel, but without requiring all the extra mechanics (and bloat, and overhead) of C++. Table 1-1 lists the standard operations provided by the kernel to manipulate kobject instances, and therefore effectively act on the embedding structure. The layout of the kref structure used to manage references is as follows: struct kref { atomic_t refcount; }; refcount is an atomic data type to specify the number of positions in the kernel at which an object is currently being used. When the counter reaches 0, the object is no longer needed and can therefore be removed from memory. 23 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 24 Chapter 1: Introduction and Overview Table 1-1: Standard Methods for Processing kobjects Function Meaning kobject_get, kobject_put Increments or decrements the reference counter of a kobject kobject_(un)register Registers or removes obj from a hierarchy (the object is added to the existing set (if any) of the parent element; a corresponding entry is cre- ated in the sysfs filesystem). kobject_init Initializes a kobject; that is, it sets the reference counter to its initial value and initializes the list elements of the object. kobect_add Initializes a kernel object and makes it visible in sysfs kobject_cleanup Releases the allocated resources when a kobject (and therefore the embedding object) is no longer needed Encapsulation of the single value in a structure was chosen to prevent direct manipulation of the value. kref_init must always be used for initialization. If an object is in use, kref_get must be invoked beforehand to increment the reference counter. kref_put decrements the counter when the object is no longer used. Sets of Objects In many cases, it is necessary to group different kernel objects into a set — for instance, the set of all character devices or the set of all PCI-based devices. The data structure provided for this purpose is defined as follows: struct kset { struct kobj_type * ktype; struct list_head list; ... struct kobject kobj; struct kset_uevent_ops * uevent_ops; }; Interestingly, the kset serves as the first example for the use of kernel objects. Since the management structure for sets is nothing other than a kernel object, it can be managed via the previously discussed struct kobj. Indeed, an instance is embedded via kobj. It has nothing to do with the kobjects collected in the set, but only serves to manage the properties of the kset object itself. The other members have the following meaning: ❑ ktype points to a further object that generalizes the behavior of the kset. ❑ list is used to build a list of all kernel objects that are a member of the set. ❑ uevent_ops provides several function pointers to methods that relay information about the state of the set to userland. This mechanism is used by the core of the driver model, for instance, to format messages that inform about the addition of new devices. 24 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 25 Chapter 1: Introduction and Overview Another structure is provided to group common features of kernel objects. It is defined as follows: struct kobj_type { ... struct sysfs_ops * sysfs_ops; struct attribute ** default_attrs; }; Note that a kobj_type is not used to collect various kernel objects — this is already managed by ksets. Instead, it provides an interface to the sysfs filesystem (discussed in Section 10.3). If multiple objects export similar information via the filesystem, then this can be simplified by using a single ktype to pro- vide the required methods. Reference Counting Reference counting is used to detect from how many places in the kernel an object is used. Whenever one part of the kernel needs information contained in one object, it increments the reference count, and when it does not need the information anymore, the count is decremented. Once the count has dropped to 0, the kernel knows that the object is not required anymore, and that it is safe to release it from memory. The kernel provides the following data structure to handle reference counting: struct kref { atomic_t refcount; }; The data structure is really simple in that it only provides a generic, atomic reference count. ‘‘Atomic’’ means in this context that incrementing and decrementing the variable is also safe on multiprocessor systems, where more than one code path can access an object at the same time. Chapter 5 discusses the need for this in more detail. The auxiliary methods kref_init, kref_get,andkref_put are provided to initialize, increment, or decrement the reference counter. This might seem trivial at a first glance. Nevertheless, it helps to avoid excessive code duplication because such reference counts together with the aforementioned operations are used all over the kernel. Although manipulating the reference counter this way is safe against concurrency issues, this does not imply that the surrounding data structure is safe against concurrent access! Kernel code needs to employ further means to ensure that access to data structures does not cause any problems when this can happen from multiple processors simultaneously, and I discuss these issues in Chapter 5. Finally, notice that the kernel contains some documentation related to kernel objects in Documentation/ kobject.txt. 1.3.15 Data Types Some issues related to data types are handled differently in the kernel in comparison to userland programs. 25 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 26 Chapter 1: Introduction and Overview Type Definitions The kernel uses typedef to define various data types in order to make itself independent of architecture- specific features because of the different bit lengths for standard data types on individual processors. The definitions have names such as sector_t (to specify a sector number on a block device), pid_t (to indicate a process identifier), and so on, and are defined by the kernel in architecture-specific code in such a way as to ensure that they represent the applicable value range. Because it is not usually important to know on which fundamental data types the definitions are based, and for simplicity’s sake, I do not always discuss the exact definitions of data types in the following chapters. Instead, I use them without further explanation — after all, they are simply non-compound standard data types under a different name. typedef’d variables must not be accessed directly, but only via auxiliary functions that I introduce when we encounter the type. This ensures that they are properly manipulated, although the type definition is transparent to the user. At certain points, the kernel must make use of variables with an exact, clearly defined number of bits — for example, when data structures need to be stored on hard disk. To allow data to be exchanged between various systems (e.g., on USB sticks), the same external format must always be used, regardless of how data are represented internally in the computer. To this end, the kernel defines several integer data types that not only indicate explicitly whether they are signed or unsigned, but also specify the exact number of bits they comprise. __s8 and __u8 are, for example, 8-bit integers that are either signed (__s8) or unsigned (__u8). __u16 and __s16, __u32 and __s32,and__u64 and __s64 are defined in the same way. Byte Order To represent numbers, modern computers use either the big endian or little endian format. The format indicates how multibyte data types are stored. With big endian ordering, the most significant byte is stored at the lowest address and the significance of the bytes decreases as the addresses increase. With little endian ordering, the least significant byte is stored at the lowest address and the significance of the bytes increases as the addresses increase (some architectures such as MIPS support both variants). Figure 1-13 illustrates the issue. 0–7 0–7 0–7 8–15 8–15 16–23 24–31 Little endian char short int char short int Byte 0 1 2 3 0–7 8–15 24–31 16–23 0–7 8–15 0–7 Big endian Figure 1-13: Composition of elementary data types depending on the endianness of the underlying architecture. 26 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 27 Chapter 1: Introduction and Overview The kernel provides various functions and macros to convert between the format used by the CPU and specific representations: cpu_to_le64 converts a 64-bit data type to little endian format, and le64_to_cpu does the reverse (if the architecture works with little endian format, the routines are, of course, no-ops; otherwise, the byte positions must be exchanged accordingly). Conversion routines are available for all combinations of 64, 32, and 16 bits for big and little endian. Per-CPU Variables A particularity that does not occur in normal userspace programming is per-CPU variables. They are declared with DEFINE_PER_CPU(name, type),wherename is the variable name and type is the data type (e.g., int[3], struct hash, etc.). On single-processor systems, this is not different from regular variable declaration. On SMP systems with several CPUs, an instance of the variable is created for each CPU. The instance for a particular CPU is selected with get_cpu(name, cpu),wheresmp_processor_id(),which returns the identifier of the active processor, is usually used as the argument for cpu. Employing per-CPU variables has the advantage that the data required are more likely to be present in the cache of a processor and can therefore be accessed faster. This concept also skirts round several communication problems that would arise when using variables that can be accessed by all CPUs of a multiprocessor system. Access to Userspace At many points in the source code there are pointers labeled __user; these are also unknown in userspace programming. The kernel uses them to identify pointers to areas in user address space that may not be de-referenced without further precautions. This is because memory is mapped via page tables into the userspace portion of the virtual address space and not directly mapped by physical memory. Therefore the kernel needs to ensure that the page frame in RAM that backs the destination is actually present —I discuss this in further detail in Chapter 2. Explicit labeling supports the use of an automatic checker tool (sparse) to ensure that this requirement is observed in practice. 1.3.16 ... and Beyond the Infinite Although a wide range of topics are covered in this book, they inevitably just represent a portion of what Linux is capable of: It is simply impossible to discuss all aspects of the kernel in detail. I have tried to choose topics that are likely to be most interesting for a general audience and also present a representative cross-section of the whole kernel ecosystem. Besides going through many important parts of the kernel, one of my concerns is also to equip you with the general idea of why the kernel is designed as it is, and how design decisions are made by interacting developers. Besides a discussion of numerous fields that are not directly related to the kernel (e.g., how the GNU C compiler works), but that support kernel development as such, I have also included a discussion about some nontechnical but social aspects of kernel development in Appendix F. Finally, please note Figure 1-14, which shows the growth of the kernel sources during the last couple of years. Kernel development is a highly dynamical process, and the speed at which the kernel acquires new features and continues to improve is sometimes nothing short of miraculous. As a study by the Linux Foundation has shown [KHCM], roughly 10,000 patches go into each kernel release, and this massive 27 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 28 Chapter 1: Introduction and Overview amount of code is created by nearly 1,000 developers per release. On average, 2.83 changes are integrated every hour, 24 hours a day, and 7 days a week! This can only be handled with mature means of source code management and communication between developers; I come back to these issues in Appendices B and F. 120 140 160 180 200 220 240 260 280 2.5.0 2.5.12 2.5.50 2.6.0 2.6.7 2.6.10 2.6.14 2.6.17 2.6.19 2.6.22 2.6.25 01-Jan-02 01-Jan-03 01-Jan-04 01-Jan-05 01-Jan-06 01-Jan-07 01-Jan-08 Uncompressed Tree Size [MiB] Kernel Release Figure 1-14: Evolution of the core kernel distribution’s size during the last years. 1.4 Why the Kernel Is Special The kernel is an amazing place — but after all, it is just a big C program with some assembler portions (and a drop or two of black magic added sometimes). So what makes the kernel so fascinating? Several factors contribute to this. First and foremost, the kernel is written by the best programmers in the world, and this shows in the code. It is well structured, written with meticulous attention for detail, and contains clever solutions all over the place. In one word: It is code as it ought to be. This, however, does not mean that the kernel is the product of a textbook-style programming methodology: While it employs cleanly designed abstractions to keep the code modular and manageable, it’s the mix with the other face of the kernel that makes the code so interesting and unique: If it need be, the kernel does not back off from reusing bit positions in a context-dependent manner, overloading structure elements multiple times, squeezing yet another storage bit out of the aligned portion of pointers, using gotos freely, and numer- ous other things that would make any structured programmer scream miserably in agony and pain. 28 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 29 Chapter 1: Introduction and Overview Techniques that would be unthinkable in many textbook solutions can not only be good, but are simply required for a proper real-world working kernel. It’s the small path that keeps the balance between these totally opposite faces of the kernel that makes the whole thing so interesting, challenging, and fun! Having praised the kernel sources, there are a number of more sober issues distinct from userland programs: ❑ Debugging the kernel is usually harder than debugging userland programs. While for the latter a multitude of debuggers exist, this is much harder to realize for the kernel. There are various mechanisms to employ debuggers in kernel development as discussed in Appendix B, but these require more effort than their userland counterparts. ❑ The kernel provides numerous auxiliary functions that resemble the standard C library found in userspace, but things are much more frugal in the kernel domain. ❑ Errors in userland applications lead to a segmentation fault or a core dump, but kernel errors will take the whole system down. Or, what is even worse: They will keep the kernel happily running, but manifest themselves in a weird system crash hours after the error occurred. Because debugging in kernel space is harder than for userland applications as mentioned above, it is essential that kernel code receives more thought and judicious consideration than userland code before it is brought into use. ❑ It must be taken into account that many architectures on which the kernel runs do not support unaligned memory access without further ado. This also affects portability of data structures across architectures because of padding that is inserted by the compiler. This issue is discussed further in Appendix C. ❑ All kernel code must be protected against concurrency. Owing to the support of multiprocessor machines, Linux kernel code must be both reentrant and thread-safe. That is, routines must allow being executed simultaneously, and data must be protected against parallel access. ❑ Kernel code must work both on machines with little and big endianness. ❑ Most architectures do not allow performing floating-point calculations in the kernel without further ado, so you need to find a way to do your calculations with integer types. You will see how to deal with these issues in the further course of this book. 1.5 Some Notes on Presentation Before we start to dive right into the kernel, I need to make some remarks on how I present the material, and why I have chosen my particular approach. Notice that this book is specifically about understanding the kernel. Examples of how to write code have intentionally and explicitly been left out, considering that this book is already very comprehensive and voluminous. The works by Corbet et al. [CRKH05], Venkateswaran [Ven08], and Quade/Kunst [QK06] fill in this gap and discuss how to create new code, especially for drivers, by countless practical examples. While I discuss how the kernel build system, which is responsible to create a kernel that precisely suits your needs works, I won’t discuss the plethora of configuration options in detail, especially because they are mostly concerned with driver configuration. However, the book by Kroah-Hartman [KH07] can be avaluableaidhere. 29 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 30 Chapter 1: Introduction and Overview Usually I start my discussion with a general overview about the concepts of the topic that I am going to present, and then go down to data structures and their interrelation in the kernel. Code is usually discussed last, because this requires the highest level of detail. I have chosen this top-down approach because it is in our opinion the most accessible and easiest way to understand the kernel. Notice that it would also be possible to discuss things from the bottom up, that is, start deep down in the kernel and then work slowly up to the C library and userspace level. Notice, however, that presenting something in inverse order does not automatically make it better. In my experience, more forward references are required for a bottom-up than for a top-down strategy, so I stick to the latter throughout this book. When I directly present C source code, I sometimes take the liberty to rewrite it slightly to highlight more important elements and remove less important ‘‘due diligence’’ work. For example, it is very important for the kernel to check the return value of every memory allocation. While allocations will succeed in nearly almost all cases, it is essential to take care of cases in which not enough memory is available for a particular task. The kernel has to deal with this situation somehow, usually by returning an error return code to userspace if a task is performed as a response to a request by an application, or by omitting a warning message to the system log. However, details of this kind will in general obstruct the view of what is really important. Consider the following code, which sets up namespaces for a process: kernel/nsproxy.c static struct nsproxy *create_new_namespaces(unsigned long flags, struct task_struct *tsk, struct fs_struct *new_fs) { struct nsproxy *new_nsp; int err; new_nsp = clone_nsproxy(tsk->nsproxy); if (!new_nsp) return ERR_PTR(-ENOMEM); new_nsp->mnt_ns = copy_mnt_ns(flags, tsk->nsproxy->mnt_ns, new_fs); if (IS_ERR(new_nsp->mnt_ns)) { err = PTR_ERR(new_nsp->mnt_ns); goto out_ns; } new_nsp->uts_ns = copy_utsname(flags, tsk->nsproxy->uts_ns); if (IS_ERR(new_nsp->uts_ns)) { err = PTR_ERR(new_nsp->uts_ns); goto out_uts; } new_nsp->ipc_ns = copy_ipcs(flags, tsk->nsproxy->ipc_ns); if (IS_ERR(new_nsp->ipc_ns)) { err = PTR_ERR(new_nsp->ipc_ns); goto out_ipc; } ... return new_nsp; out_ipc: if (new_nsp->uts_ns) put_uts_ns(new_nsp->uts_ns); out_uts: if (new_nsp->mnt_ns) 30 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 31 Chapter 1: Introduction and Overview put_mnt_ns(new_nsp->mnt_ns); out_ns: kmem_cache_free(nsproxy_cachep, new_nsp); return ERR_PTR(err); } What the code does in detail is irrelevant right now; I come back to this in the following chapter. What is essential is that the routine tries to clone various parts of the namespace depending on some flags that control the cloning operation. Each type of namespace is handled in a separate function, for instance, in copy_mnt_ns for the filesystem namespace. Each time the kernel copies a namespace, errors can occur, and these must be detected and passed on to the calling function. Either the error is detected directly by the return code of a function, as for clone_nsproxy, or the error is encoded in a pointer return value, which can be detected using the ERR_PTR macro, which allows for decoding the error value (I also discuss this mechanism below). In many cases, it is not sufficient to just detect an error and return this information to the caller. It is also essen- tial that previously allocated resources that are not required anymore because of the error are released again. The standard technique of the kernel to handle this situation is as follows: Jump to a special label and free all previously allocated resources, or put down references to objects by decrementing the refer- ence count. Handling such cases as this is one of the valid applications for the goto statement. There are various possibilities to describe what is going on in the function: ❑ Talk the reader directly through thecodeinhugestep-by-steplists: 1. create_new_namespace calls clone_nsproxy. If this fails, return -ENOMEM;otherwise, continue. 2. create_new_namespace then calls copy_mnt_ns. If this fails, obtain the error value encoded in the return value of copy_mnt_ns and jump to the label out_ns; otherwise, proceed. 3. create_new_namespace then calls copy_utsname. If this fails, obtain the error value encoded in the return value of copy_utsname and jump to the label out_ns;otherwise, proceed. 4. ... While this approach is favored by a number of kernel texts, it conveys only little information in addition to what is directly visible from the source code anyway. It is appropriate to discuss some of the most complicated low-level parts of the kernel this way, but this will foster an under- standing of neither the big picture in general nor the code snippet involved in particular. ❑ Summarize what the function does with words, for instance, by remarking that ‘‘create_new_ namespaces is responsible to create copies or clones of the parent namespaces.’’ We use this approach for less important tasks of the kernel that need to be done somehow, but do not pro- vide any specific insights or use particularly interesting tricks. ❑ Use a flow diagram to illustrate what is going on in a function. With more than 150 code flow diagrams in this book, this is one of my preferred ways of dealing with code. It is important to note that these diagrams are not supposed to be a completely faithful representation of the opera- tion. This would hardly simplify matters. Consider Figure 1-15, which illustrates how a faithful representation of copy_namespaces could look. It is not at all simpler to read than the source itself, so there is not much purpose in providing it. 31 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 32 Chapter 1: Introduction and Overview create_new_namespace clone_nsproxy copy_mnt_ns put_uts_ns put_mnt_ns copy_utsname Error? Error? Return error NO NO YES YES YES NO new_ns->put_ns not NULL? new_ns-> mnt_ns not NULL? copy_utsname return result return-ENOMEM kmem_cache_free Error? Error? 1 1 Figure 1-15: Example of a faithful, but unclear and convoluted code flow diagram. Instead I employ code flow diagrams that illustrate the essential tasks performed by a function. Figure 1-16 shows the code flow diagram that I would have employed instead of Figure 1-15. copy_namespaces clone_nsproxy copy_mnt_ns copy_utsname copy_ipcs Figure 1-16: Example of the style of code flow diagrams used in this book. They allow immediately catching all essential actions without being distracted by nonessential standard tasks. The diagram omits several things, but this is on purpose, and also essential. By looking at the figure, you will not see every detail of the function implementation, but you will instead imme- diately realize that the kernel uses a specific routine to create a clone of each namespace, and the function names provide a sufficient hint of which namespace is copied. This is much more important! 32 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 33 Chapter 1: Introduction and Overview Handling error return codes is something that we assume goes without saying, and so we will not pay particular attention to it. This does not imply that it is not an important thing to do, and in fact it is: Linux would be a lousy kernel if it did not handle this issue properly. But handling errors also obfuscates most operations without introducing any new insights, and does not make it easier to understand the general principles of the kernel, so it’s usually better to sacrifice some thoroughness for clarity. The kernel sources are always available for all the gory details! ❑ It is also often important to discuss kernel code directly if it is packed with important deci- sions, and I do so when I deem it necessary. However, I often take the liberty of omitting less interesting or purely mechanical parts, so don’t be astonished if the code presented in the book sometimes differs slightly from the code seen in the kernel. With respect to the source code, this book is self-contained, but it certainly helps if it is not read on a desolate island, but next to a computer where the Linux source code is available and can be inspected. Besides that, being on a desolate island is not much fun anyway. Since I base many machine-specific examples on IA-32 and AMD64, some words about these terms are in order. ‘‘IA-32‘‘ includes all Intel-compatible CPUs such as Pentium, Athlon, and so on. AMD64 also includes the Intel variant EM64T. For the sake of simplicity, I use only the abbreviations IA-32 and AMD64 in this book. Since Intel undoubtedly invented IA-32 and AMD came up first with the 64-bit extensions, this seems a fair compromise. It is also interesting to note that starting with kernel 2.6.23, both architectures are unified to the generic x86 architecture within the Linux kernel. This makes the code easier to maintain for the developers because many elements can be shared between both variants, but nevertheless still distinguishes between 32- and 64-bit capabilities of the processors. 1.6 Summary The Linux kernel is one of the most interesting and fascinating pieces of software ever written, and I hope this chapter has succeeded in whetting your appetite for the things to come in the following chapters, where I discuss many subsystems in detail. For now, I have provided a bird’s eye view of the kernel to present the big picture of how responsibilities are distributed, which part of the kernel has to deal with which problems, and how the components interact with each other. Since the kernel is a huge system, there are some issues related to the presentation of the complex mate- rial, and I have introduced you to the particular approach chosen for this book. 33 Mauerer runc01.tex V2 - 09/04/2008 4:13pm Page 34 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 35 Process Management and Scheduling All modern operating systems are able to run several processes at the same time — at least, this is the impression users get. If a system has only one processor, only one program can run on it at a given time. In multiprocessor systems, the number of processes that can truly run in parallel is determined by the number of physical CPUs. The kernel and the processor create the illusion of multitasking — the ability to perform several operations in parallel — by switching repeatedly between the different applications running on the system at very rapid intervals. Because the switching intervals are so short, users do not notice the intervening brief periods of inactivity and gain the impression that the computer is actually doing several things at once. This kind of system management gives rise to several issues that the kernel must resolve, the most important of which are listed below. ❑ Applications must not interfere with each other unless this is expressly desired. For example, an error in application A must not be propagated to application B. Because Linux is a multiuser system, it must also be ensured that programs are not able to read or modify the memory contents of other programs — otherwise, it would be extremely easy to access the private data of other users. ❑ CPU time must be shared as fairly as possible between the various applications, whereby some programs are regarded as more important than others. I deal with the first requirement — memory protection — in Chapter 3. In the present chapter, I focus my attention on the methods employed by the kernel to share CPU time and to switch between processes. This twofold task is split into two parts that are performed relatively indepen- dently of each other. Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 36 Chapter 2: Process Management and Scheduling ❑ The kernel must decide how much time to devote to each process and when to switch to the next process. This begs the question as to which process is actually the next. Decisions of this kind are not platform-dependent. ❑ When the kernel switches from process A to process B, it must ensure that the execution envi- ronment of B is exactly the same as when it last withdrew processor resources. For example, the contents of the processor registers and the structure of virtual address space must be identical. This latter task is extremely dependent on processor type. It cannot be implemented with C only, but requires help by pure assembler portions. Both tasks are the responsibility of a kernel subsystem referred to as the scheduler. How CPU time is allo- cated is determined by the scheduler policy, which is totally separate from the task switching mechanism needed to switch between processes. 2.1 Process Priorities Not all processes are of equal importance. In addition to process priority, with which most readers will be familiar, there are different criticality classes to satisfy differing demands. In a first coarse distinction, processes can be split into real-time processes and non-real-time processes. ❑ Hard real-time processes are subject to strict time limits during which certain tasks must be com- pleted. If the flight control commands of an aircraft are processed by computer, they must be forwarded as quickly as possible — within a guaranteed period of time. For example, if an air- craft is on its landing approach and the pilot wants to pull up the nose, it serves little purpose if the computer forwards the command a few seconds later. By this time, the aircraft may well be buried — nose first — in the ground. The key characteristic of hard real-time processes is that they must be processed within a guaranteed time frame. Note that this does not imply that the time frame is particularly short. Instead, the system must guarantee that a certain time frame is never exceeded, even when unlikely or adverse conditions prevail. Linux does not support hard real-time processing, at least not in the vanilla kernel. There are, however, modified versions such as RTLinux, Xenomai, or RATI that offer this feature. The Linux kernel runs as a separate ‘‘process’’ in these approaches and handles less important soft- ware, while real-time work is done outside the kernel. The kernel may run only if no real-time critical actions are performed. Since Linux is optimized for throughput and tries to handle common cases as fast as possible, guaranteed response times are only very hard to achieve. Nevertheless quite a bit of progress has been made during the last years to decrease the overall kernel latency, that is, the time that elapses between making a request and its fulfillment. The efforts include the preemptible kernel mechanism, real-time mutexes, and the new completely fair scheduler discussed in this book. ❑ Soft real-time processes are a softer form of hard real-time processes. Although quick results are still required, it is not the end of the world if they are a little late in arriving. An example of a soft real-time process is a write operation to a CD. Data must be received by the CD writer at a certain rate because data are written to the medium in a continuous stream. If system loading is too high, the data stream may be interrupted briefly, and this may result in an unusable CD, far less drastic than a plane crash. Nevertheless, the write process should always be granted CPU time when needed — before all other normal processes. 36 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 37 Chapter 2: Process Management and Scheduling ❑ Most processes are normal processes that have no specific time constraints but can still be classified as more important or less important by assigning priorities to them. For example, a long compiler run or numerical calculations need only very low priority because it is of little consequence if computation is interrupted occasionally for a second or two — users are unlikely to notice. In contrast, interactive applications should respond as quickly as possible to user commands because users are notoriously impatient. The allocation of CPU time can be portrayed in much simplified form as in Figure 2-1. Processes are spread over a time slice, and the share of the slice allocated to them corresponds to their relative impor- tance. The time flow in the system corresponds to the turning of the circle, and the CPU is represented by a ‘‘scanner‘‘ at the circumference of the circle. The net effect is that important processes are granted more CPU time than less important processes, although all eventually have their turn. CPU A B C D Figure 2-1: Allocation of CPU time by means of time slices. In this scheme, known as preemptive multitasking, each process is allocated a certain time period during which it may execute. Once this period has expired, the kernel withdraws control from the process and lets a different process run — regardless of the last task performed by the previous process. Its runtime environment — essentially, the contents of all CPU registers and the page tables — is, of course, saved so that results are not lost and the process environment is fully reinstated when its turn comes around again. The length of the time slice varies depending on the importance of the process (and therefore on the priority assigned to it). Figure 2-1 illustrates this by allocating segments of different sizes to the individual processes. This simplified model does not take into account several important issues. For example, processes may not be ready to execute at certain times because they have nothing to do. Because it is essential to use CPU time as profitably as possible, such processes must be prevented from executing. This is not evident in Figure 2-1 because it is assumed that all processes are always ready to run. Also ignored is the fact that Linux supports different scheduling classes (completely fair scheduling between processes, and real- time scheduling), and these must also be taken into consideration during scheduling. Neither is there an option to replace the current process with an important process that has become ready to run. Note that process scheduling causes very fervid and excited discussion among kernel developers, espe- cially when it comes to picking the best possible algorithm. Finding a quantitative measure for the quality of a scheduler is a very hard — if not impossible — task. It is also a very challenging task for a sched- uler to fulfill the requirements imposed by the many different workloads that Linux systems have to face: Small embedded systems for automated control usually have very different requirements than large 37 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 38 Chapter 2: Process Management and Scheduling number crunchers, while these in turn differ considerably from multimedia systems. In fact, the scheduler code has seen two complete rewrites in recent years: 1. During the development series 2.5, a so-called O(1) scheduler replaced the previous sched- uler. One particular property of this scheduler was that it could perform its work in constant time independent of the number of processes that were running on a system. The design broke quite radically with the previously employed scheduling architecture. 2. The completely fair scheduler was merged during the development of kernel 2.6.23. The new code again marked a complete departure from previous principles by abandoning, for instance, many heuristics that were required in previous schedulers to ensure that interactive tasks would respond quickly. The key feature of this scheduler is that it tries to resemble ideal fair scheduling as close as possible. Besides, it cannot only schedule individual tasks, but works with more general scheduling entities. This allows, for instance, for distribution the available time between all processes of different users, and then among the processes of each user. I discuss the implementation of this scheduler below in detail. Before we concern ourselves with how scheduling is implemented in the kernel, it is useful to discuss the states that a process may have. 2.2 Process Life Cycle A process is not always ready to run. Occasionally, it has to wait for events from external sources beyond its control — for keyboard input in a text editor, for example. Until the event occurs, the process cannot run. The scheduler must know the status of every process in the system when switching between tasks; it obviously doesn’t make sense to assign CPU time to processes that have nothing to do. Of equal impor- tance are the transitions between individual process states. For example, if a process is waiting for data from a peripheral device, it is the responsibility of the scheduler to change the state of the process from waiting to runnable once the data have arrived. A process may have one of the following states: ❑ Running — The process is executing at the moment. ❑ Waiting — The process is able to run but is not allowed to because the CPU is allocated to another process. The scheduler can select the process, if it wants to, at the next task switch. ❑ Sleeping — The process is sleeping and cannot run because it is waiting for an external event. The scheduler cannot select the process at the next task switch. The system saves all processes in a process table — regardless of whether they are running, sleeping, or waiting. However, sleeping processes are specially ‘‘marked‘‘ so that the scheduler knows they are not ready to run (see how this is implemented in Section 2.3). There are also a number of queues that group sleeping processes so that they can be woken at a suitable time — when, for example, an external event that the process has been waiting for takes place. 38 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 39 Chapter 2: Process Management and Scheduling Figure 2-2 shows several process states and transitions. Running Stopped Waiting Sleeping 5 4 2 1 3 Figure 2-2: Transitions between process states. Let’s start our examination of the various transitions with a queued runnable process; the process is ready to run but is not allowed to because the CPU is allocated to a different process (its state is therefore ‘‘waiting‘‘). It remains in this state until the scheduler grants it CPU time. Once this happens, its state changes to ‘‘running‘‘ (path 4). When the scheduler decides to withdraw CPU resources from the process — I deal with the possible rea- sons why shortly — the process state changes from ‘‘running‘‘ to ‘‘waiting‘‘ (path 2), and the cycle starts anew. There are, in fact, two ‘‘sleeping‘‘ states that differ according to whether they can be interrupted by signals or not. At the moment, this difference is not important, but it is of relevance when we examine implementation more closely. If the process has to wait for an event, its state changes (path 1) from ‘‘running‘‘ to ‘‘sleeping.’’ However, it cannot change directly from ‘‘sleeping‘‘ to ‘‘running‘‘; once the event it was waiting for has taken place, the process changes back to the ‘‘waiting‘‘ state (path 3) and then rejoins the normal cycle. Once program execution terminates (e.g., the user closes the the application), the process state changes from ‘‘running‘‘ to ‘‘stopped‘‘ (path 5). A special process state not listed above is the ‘‘zombie‘‘state. As the name suggests, such processes are defunct but are somehow still alive. In reality, they are dead because their resources (RAM, connections to peripherals, etc.) have already been released so that they cannot and never will run again. However, they are still alive because there are still entries for them in the process table. How do zombies come about? The reason lies in the process creation and destruction structure under Unix. A program terminates when two events occur — first, the program must be killed by another process or by a user (this is usually done by sending a SIGTERM or SIGKILL signal, which is equivalent to terminating the process regularly); second, the parent process from which the process originates must invoke or have already invoked the wait4 (read: wait for) system call when the child process terminates. This confirms to the kernel that the parent process has acknowledged the death of the child. The system call enables the kernel to free resources reserved by the child process. A zombie occurs when only the first condition (the program is terminated) applies but not the second (wait4). A process always switches briefly to the zombie state between termination and removal of its data from the process table. In some cases (if, e.g., the parent process is badly programmed and does not issue a wait call), a zombie can firmly lodge itself in the process table and remain there until the next 39 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 40 Chapter 2: Process Management and Scheduling reboot. This can be seen by reading the output of process tools such as ps or top.Thisishardlyaproblem as the residual data take up little space in the kernel. 2.2.1 Preemptive Multitasking The structure of Linux process management requires two further process state options — user mode and kernel mode. These reflect the fact that all modern CPUs have (at least) two different execution modes, one of which has unlimited rights while the other is subject to various restrictions — for example, access to certain memory areas can be prohibited. This distinction is an important prerequisite for creating locked ‘‘cages,’’ which hold existing processes and prevent them from interfering with other parts of the system. Normally the kernel is in user mode in which it may access only its own data and cannot therefore inter- fere with other applications in the system — it usually doesn’t even notice that there are other programs besides itself. If a process wants to access system data or functions (the latter manage the resources shared between all processes, e.g., filesystem space), it must switch to kernel mode. Obviously, this is possible only under control — otherwise all established protection mechanisms would be superfluous — and via clearly defined paths. Chapter 1 mentioned briefly that ‘‘system calls‘‘ are one way to switch between modes. Chapter 13 discusses the implementation of such calls in depth. A second way of switching from user mode to kernel mode is by means of interrupts — switching is then triggered automatically. Unlike system calls, which are invoked intentionally by user applications, interrupts occur more or less arbitrarily. Generally, the actions needed to handle interrupts have nothing to do with the process executing when the interrupt occurred. For example, an interrupt is raised when an external block device has transferred data to RAM, although these data may be intended for any process running on the system. Similarly, incoming network packages are announced by means of an interrupt. Again, it is unlikely that the inbound package is intended for the process currently running. For this reason, Linux performs these actions in such a way that the running process is totally unaware of them. The preemptive scheduling model of the kernel establishes a hierarchy that determines which process states may be interrupted by which other states. ❑ Normal processes may always be interrupted — even by other processes. When an important process becomes runnable — for example, an editor receives long-awaited keyboard input — the scheduler can decide whether to execute the process immediately, even if the current process is still happily running. This kind of preemption makes an important contribution to good interac- tive behavior and low system latency. ❑ If the system is in kernel mode and is processing a system call, no other process in the system is able to cause withdrawal of CPU time. The scheduler is forced to wait until execution of the system call has terminated before it can select another process. However, the system call can be suspended by an interrupt.1 ❑ Interrupts can suspend processes in user mode and in kernel mode. They have the highest prior- ity because it is essential to handle them as soon as possible after they are issued. 1It is possible to disable almost all interrupts for important kernel actions. 40 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 41 Chapter 2: Process Management and Scheduling One option known as kernel preemption was added to the kernel during the development of kernel 2.5. This option supports switches to another process, if this is urgently required, even during the execution of system calls in kernel mode (but not during interrupts). Although the kernel attempts to execute system calls as quickly as possible, the time needed may be too long for some applications that are reliant on constant data streams. Kernel preemption can reduce such wait times and thus ensure ‘‘smoother‘‘ program execution. However, this is at the expense of increased kernel complexity because many data structures then need to be protected against concurrent access even on single-processor systems. This technique is discussed in Section 2.8.3. 2.3 Process Representation All algorithms of the Linux kernel concerned with processes and programs are built around a data struc- ture named task_struct and defined in include/sched.h. This is one of the central structures in the system. Before we move on to deal with the implementation of the scheduler, it is essential to examine how Linux manages processes. The task structure includes a large number of elements that link the process with the kernel subsystems which I discuss below. I therefore make frequent reference to later chapters because it is difficult to explain the significance of some elements without detailed knowledge of them. The task structure is defined as follows — in simplified form: struct task_struct { volatile long state; /* -1 unrunnable, 0 runnable, >0 stopped */ void *stack; atomic_t usage; unsigned long flags; /* per process flags, defined below */ unsigned long ptrace; int lock_depth; /* BKL lock depth */ int prio, static_prio, normal_prio; struct list_head run_list; const struct sched_class *sched_class; struct sched_entity se; unsigned short ioprio; unsigned long policy; cpumask_t cpus_allowed; unsigned int time_slice; #if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT) struct sched_info sched_info; #endif struct list_head tasks; /* * ptrace_list/ptrace_children forms the list of my children * that were stolen by a ptracer. */ struct list_head ptrace_children; 41 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 42 Chapter 2: Process Management and Scheduling struct list_head ptrace_list; struct mm_struct *mm, *active_mm; /* task state */ struct linux_binfmt *binfmt; long exit_state; int exit_code, exit_signal; int pdeath_signal; /* The signal sent when the parent dies */ unsigned int personality; unsigned did_exec:1; pid_t pid; pid_t tgid; /* * pointers to (original) parent process, youngest child, younger sibling, * older sibling, respectively. (p->father can be replaced with * p->parent->pid) */ struct task_struct *real_parent; /* real parent process (when being debugged) */ struct task_struct *parent; /* parent process */ /* * children/sibling forms the list of my children plus the * tasks I’m ptracing. */ struct list_head children; /* list of my children */ struct list_head sibling; /* linkage in my parent’s children list */ struct task_struct *group_leader; /* threadgroup leader */ /* PID/PID hash table linkage. */ struct pid_link pids[PIDTYPE_MAX]; struct list_head thread_group; struct completion *vfork_done; /* for vfork() */ int __user *set_child_tid; /* CLONE_CHILD_SETTID */ int __user *clear_child_tid; /* CLONE_CHILD_CLEARTID */ unsigned long rt_priority; cputime_t utime, stime, utimescaled, stimescaled;; unsigned long nvcsw, nivcsw; /* context switch counts */ struct timespec start_time; /* monotonic time */ struct timespec real_start_time; /* boot based time */ /* mm fault and swap info: this can arguably be seen as either mm-specific or thread-specific */ unsigned long min_flt, maj_flt; cputime_t it_prof_expires, it_virt_expires; unsigned long long it_sched_expires; struct list_head cpu_timers[3]; /* process credentials */ uid_t uid,euid,suid,fsuid; gid_t gid,egid,sgid,fsgid; struct group_info *group_info; kernel_cap_t cap_effective, cap_inheritable, cap_permitted; 42 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 43 Chapter 2: Process Management and Scheduling unsigned keep_capabilities:1; struct user_struct *user; char comm[TASK_COMM_LEN]; /* executable name excluding path - access with [gs]et_task_comm (which lock it with task_lock()) - initialized normally by flush_old_exec */ /* file system info */ int link_count, total_link_count; /* ipc stuff */ struct sysv_sem sysvsem; /* CPU-specific state of this task */ struct thread_struct thread; /* filesystem information */ struct fs_struct *fs; /* open file information */ struct files_struct *files; /* namespace */ struct nsproxy *nsproxy; /* signal handlers */ struct signal_struct *signal; struct sighand_struct *sighand; sigset_t blocked, real_blocked; sigset_t saved_sigmask; /* To be restored with TIF_RESTORE_SIGMASK */ struct sigpending pending; unsigned long sas_ss_sp; size_t sas_ss_size; int (*notifier)(void *priv); void *notifier_data; sigset_t *notifier_mask; #ifdef CONFIG_SECURITY void *security; #endif /* Thread group tracking */ u32 parent_exec_id; u32 self_exec_id; /* journalling filesystem info */ void *journal_info; /* VM state */ struct reclaim_state *reclaim_state; struct backing_dev_info *backing_dev_info; struct io_context *io_context; unsigned long ptrace_message; siginfo_t *last_siginfo; /* For ptrace use. */ ... }; 43 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 44 Chapter 2: Process Management and Scheduling Admittedly, it is difficult to digest the amount of information in this structure. However, the structure contents can be broken down into sections, each of which represents a specific aspect of the process: ❑ State and execution information such as pending signals, binary format used (and any emulation information for binary formats of other systems), process identification number (pid), pointers to parents and other related processes, priorities, and time information on program execution (e.g., CPU time). ❑ Information on allocated virtual memory. ❑ Process credentials such as user and group ID, capabilities,2 and so on. System calls can be used to query (or modify) these data; I deal with these in greater detail when describing the specific subsystems. ❑ Files used: Not only the binary file with the program code but also filesystem information on all files handled by the process must be saved. ❑ Thread information, which records the CPU-specific runtime data of the process (the remaining fields in the structure are not dependent on the hardware used). ❑ Information on interprocess communication required when working with other applications. ❑ Signal handlers used by the process to respond to incoming signals. Many members of the task structure are not simple variables but pointers to other data structures examined and discussed in the following chapters. In the present chapter, I consider some elements of task_struct that are of particular significance in process management implementation. state specifies the current state of a process and accepts the following values (these are pre-processor constants defined in ): ❑ TASK_RUNNING means that a task is in a runnable state. It does not mean that a CPU is actually allocated. The task can wait until it is selected by the scheduler. This state guarantees that the process really is ready to run and is not waiting for an external event. ❑ TASK_INTERRUPTIBLE is set for a sleeping process that is waiting for some event or other. When the kernel signals to the process that the event has occurred, it is placed in the TASK_RUNNING state and may resume execution as soon as it is selected by the scheduler. ❑ TASK_UNINTERRUPTIBLE is used for sleeping processes disabled on the instructions of the kernel. Theymaynotbewokenbyexternalsignals,onlybythekernelitself. ❑ TASK_STOPPED indicates that the process was stopped on purpose — by a debugger, for example. ❑ TASK_TRACED is not a process state per se — it is used to distinguish stopped tasks that are cur- rently being traced (using the ptrace mechanism) from regular stopped tasks. The following constants can be used both in the task state field of struct task_struct, but also in the field exit_state, which is specifically for exiting processes. ❑ EXIT_ZOMBIE is the zombie state described above. ❑ EXIT_DEAD is the state after an appropriate wait system call has been issued and before the task is completely removed from the system. This state is only of importance if multiple threads issue wait calls for the same task. 2Capabilities are special permissions that can be granted to a process. They allow the process to perform certain operations that nor- mally may be performed only by root processes. 44 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 45 Chapter 2: Process Management and Scheduling Linux provides the resource limit (rlimit) mechanism to impose certain system resource usage limits on processes. The mechanism makes use of the rlim array in task_struct, whose elements are of the struct rlimit type. struct rlimit { unsigned long rlim_cur; unsigned long rlim_max; } The definition is purposely kept very general so that it can accept many different resource types. ❑ rlim_cur is the current resource limit for the process. It is also referred to as the soft limit. ❑ rlim_max is the maximum allowed value for the limit. It is therefore also referred to as the hard limit. The setrlimit system call is used to increase or decrease the current limit. However, the value specified in rlim_max may not be exceeded. getrlimits is used to check the current limit. The limitable resources are identified by reference to their position in the rlim array, which is why the kernel defines pre-processor constants to associate resource and position. Table 2-1 lists the possible constants and their meanings. Textbooks on system programming provide detailed explanations on best use of the various limits in practice, and the manual page setrlimit(2) contains more detailed descriptions of all limits. The numeric values differ between architectures because Linux tries to establish binary compatibility with the specific native Unix systems. Because the limits relate to very different parts of the kernel, the kernel must check that the limits are observed in the corresponding subsystems. This is why we encounter rlimit time and time again in later chapters of this book. If a resource type may be used without limits (the default setting for almost all resources), RLIM_INFINITY is used as the value for rlim_max. Exceptions are, among others: ❑ The number of open files (RLIMIT_NOFILE, limited to 1,024 by default). ❑ The maximum number of processes per user (RLIMIT_NPROC), defined as max_threads/2. max_threads is a global variable whose value specifies how many threads may be generated so that an eighth of available RAM is used only for management of thread information, given a minimum possible memory usage of 20 threads. The boot-time limits for the init task are defined in INIT_RLIMITS in include/asm-generic-resource.h. Notice that kernel 2.6.25, which was still under development when this book was written, will contain one file per process in the proc filesystem, which allows for inspecting the current rlimit values: wolfgang@meitner> cat /proc/self/limits Limit Soft Limit Hard Limit Units 45 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 46 Chapter 2: Process Management and Scheduling Max cpu time unlimited unlimited ms Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1024 1024 files Max locked memory unlimited unlimited bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals unlimited unlimited signals Max msgqueue size unlimited unlimited bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Table 2-1: Process-Specific Resource Limits. Constant Meaning RLIMIT_CPU Maximum CPU time in milliseconds. RLIMIT_FSIZE Maximum file size allowed. RLIMIT_DATA Maximum size of the data segment. RLIMIT_STACK Maximum size of the (user mode) stack. RLIMIT_CORE Maximum size for core dump files. RLIMIT_RSS Maximum size of the resident size set; in other words, the maximum number of page frames that a process uses. Not used at the moment. RLIMIT_NPROC Maximum number of processes that the user associated with the real UID of a process may own. RLIMIT_NOFILE Maximum number of open files. RLIMIT_MEMLOCK Maximum number of non-swappable pages. RLIMIT_AS Maximum size of virtual address space that may be occupied by a process. RLIMIT_LOCKS Maximum number of file locks. RLIMIT_SIGPENDING Maximum number of pending signals. RLIMIT_MSGQUEUE Maximum number of message queues. RLIMIT_NICE Maximum nice level for non-real-time processes. RLIMIT_RTPRIO Maximum real-time priority. 46 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 47 Chapter 2: Process Management and Scheduling Most of the code to generate the information is already present in kernel 2.6.24, but the final connection with /proc will only be made in the following kernel release. 2.3.1 Process Types A classical Unix process is an application that consists of binary code, a chronological thread (the com- puter follows a single path through the code, no other paths run at the same time), and a set of resources allocated to the application — for example, memory, files, and so on. New processes are generated using the fork and exec system calls: ❑ fork generates an identical copy of the current process; this copy is known as a child process.All resources of the original process are copied in a suitable way so that after the system call there are two independent instances of the original process. These instances are not linked in any way but have, for example, the same set of open files, the same working directory, the same data in memory (each with its own copy of the data), and so on.3 ❑ exec replaces a running process with another application loaded from an executable binary file. In other words, a new program is loaded. Because exec does not create a new process, an old program must first be duplicated using fork,andthenexec must be called to generate an addi- tional application on the system. Linux also provides the clone system call in addition to the two calls above that are available in all Unix flavors and date back to very early days. In principle, clone works in the same way as fork,butthenew process is not independent of its parent process and can share some resources with it. It is possible to specify which resources are to be shared and which are to be copied — for example, data in memory, open files, or the installed signal handlers of the parent process. clone is used to implement threads. However, the system call alone is not enough to do this. Libraries are also needed in userspace to complete implementation. Examples of such libraries are Linuxthreads and Next Generation Posix Threads. 2.3.2 Namespaces Namespaces provide a lightweight form of virtualization by allowing us to view the global properties of a running system under different aspects. The mechanism is similar to zones in Solaris or the jail mech- anism in FreeBSD. After a general overview of the concept, I discuss the infrastructure provided by the namespace framework. Concept Traditionally, many resources are managed globally in Linux as well as other Unix derivatives. For instance, all processes in the system are conventionally identified by their PID, which implies that a global list of PIDs must be managed by the kernel. Likewise, the information about the system returned by the uname system call (which includes the system name and some information about the kernel) is the same for all callers. User IDs are managed in a similar fashion: Each user is identified by a UID number that is globally unique. 3In Section 2.4.1, you will see that Linux does use the copy-on-write mechanism to not copy memory pages of the forked process until the new process performs a write access to the pages — this is more efficient than blindly copying all memory pages immedi- ately on execution of fork. The link between the memory pages of the parent and child process needed to do this is visible to the kernel only and is transparent to the applications. 47 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 48 Chapter 2: Process Management and Scheduling Global identifiers allow the kernel to selectively grant or deny certain privileges. While the root user with UID 0 is essentially allowed to do anything, higher user IDs are more confined. A user with PID n may, for instance, not kill processes that belong to user m = n. However, this does not prevent users from seeing each other: User n can see that another user m is also active on the machine. This is no problem: As long as users can only fiddle with their own processes, there is no reason why they should not be allowed to observe that other users have processes as well. There are cases, though, where this can be undesired. Consider that a web provider wants to give full access to Linux machines to customers, including root access. Traditionally, this would require setting up one machine per customer, which is a costly business. Using virtualized environments as provided by KVM or VMWare is one way to solve the problem, but does not distribute resources very well: One separate kernel is required for each customer on the machine, and also one complete installation of the surrounding userland. A different solution that is less demanding on resources is provided by namespaces. Instead of using virtualized systems such that one physical machine can run multiple kernels — which may well be from different operating systems — in parallel, a single kernel operates on a physical machine, and all previ- ously global resources are abstracted in namespaces. This allows for putting a group of processes into a container, and one container is separated from other containers. The separation can be such that members of one container have no connection whatsoever with other containers. Is is, however, also possible to loosen the separation of containers by allowing them to share certain aspects of their life. For instance, containers could be set up to use their own set of PIDs, but still share portions of filesystems with each other. Namespaces essentially create different views of the system. Every formerly global resource must be wrapped up in a container data structure, and only tuples of the resource and the containing namespace are globally unique. While the resource alone is enough inside a given container, it does not provide a unique identity outside the container. An overview of the situation is given in Figure 2-3. Consider a case in which three different namespaces are present on the system. Namespaces can be hierarchically related, and I consider this case here. One namespace is the parent namespace, which has spawned two child namespaces. Assume that the containers are used in a hosting setup where each container must look like a single Linux machine. Each of them therefore has its own init task with PID 0, and the PIDs of other tasks are assigned in increasing order. Both child namespaces have an init task with PID 0, and two processes with PIDs 2 and 3, respectively. Since PIDs with identical values appear multiple times on the system, the numbers are not globally unique. While none of the child containers has any notion about other containers in the system, the parent is well informed about the children, and consequently sees all processes they execute. They are mapped to the PID range 4 to 9 in the parent process. Although there are 9 processes on the system, 15 PIDs are required to represent them because one process can be associated with more than one PID. The ‘‘right’’ one depends on the context in which the process is observed. Namespaces can also be non-hierarchical if they wrap simpler quantities, for instance, like the UTS namespace discussed below. In this case, there is no connection between parent and child namespaces. Notice that support for namespaces in a simple form has been available in Linux for quite a long time in the form of the chroot system call. This method allows for restricting processes to a certain part of 48 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 49 Chapter 2: Process Management and Scheduling 4 5 6 7 8 9 1 2 3 1 2 3 1 2 3 parent parent Parent namespace Child namespaces Figure 2-3: Namespaces can be related in a hierarchical order. Each namespace has a parent from which it originates, and a parent can have multiple children. the filesystem and is thus a simple namespace mechanism. True namespaces do, however, allow for controlling much more than just the view on the filesystem. New namespaces can be established in two ways: 1. When a new process is created with the fork or clone system call, specific options control if namespaces will be shared with the parent process, or if new namespaces are created. 2. The unshare system call dissociates parts of a process from the parent, and this also includes namespaces. See the manual page unshare(2) for more information. Once a process has been disconnected from the parent namespace using any of the two mechanisms above, changing a — from its point of view — global property will not propagate into the parent names- pace, and neither will a change on the parent side propagate into the child, at least for simple quantities. The situation is more involved for filesystems where the sharing mechanisms are very powerful and allow a plethora of possibilities, as discussed in Chapter 8. Namespaces are currently still marked as experimental in the standard kernel, and development to make all parts of the kernel fully namespace-aware are still going on. As of kernel 2.6.24, the basic framework is, however, set up and in place.4 The file Documentation/namespaces/compatibility-list.txt provides information about some problems that are still present in the current state of the implementation. 4This, however, does not imply that the approach was only recently developed. In fact, the methods have been used in production systems over many years, but were only available as external kernel patches. 49 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 50 Chapter 2: Process Management and Scheduling Implementation The implementation of namespaces requires two components: per-subsystem namespace structures that wrap all formerly global components on a per-namespace basis, and a mechanism that associates a given process with the individual namespaces to which it belongs. Figure 2-4 illustrates the situation. struct task_struct struct nsproxy struct nsproxy uts_namespc user_namespc mnt_namespc uts_namespc user_namespc mnt_namespc UTS Namespace User Namespace 0 0 1 Tasks struct task_struct struct task_struct Figure 2-4: Connection between processes and namespaces. Formerly global properties of subsystems are wrapped up in namespaces, and each process is associated with a particular selection of namespaces. Each kernel subsystem that is aware of namespaces must provide a data structure that collects all objects that must be available on a per-namespace basis. struct nsproxy is used to collect pointers to the subsystem-specific namespace wrappers: struct nsproxy { atomic_t count; struct uts_namespace *uts_ns; struct ipc_namespace *ipc_ns; struct mnt_namespace *mnt_ns; struct pid_namespace *pid_ns; struct user_namespace *user_ns; struct net *net_ns; }; Currently the following areas of the kernel are aware of namespaces: ❑ The UTS namespace contains the name of the running kernel, and its version, the underlying architecture type, and so on. UTS is a shorthand for Unix Timesharing System. ❑ All information related to inter-process communication (IPC) is stored in struct ipc_namespace. 50 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 51 Chapter 2: Process Management and Scheduling ❑ The view on the mounted filesystem is given in struct mnt_namespace. ❑ struct pid_namespace provides information about process identifiers. ❑ struct user_namespace is required to hold per-user information that allows for limiting resource usage for individual users. ❑ struct net_ns contains all networking-related namespace parameters. There is, however, still quite a lot of effort required to make this area fully aware of namespaces as you will see in Chapter 12. I introduce the contents of the individual namespace containers when I discuss the respective subsystem. In this chapter, we will be concerned about UTS and user namespaces. Since fork can be instructed to open a new namespace when a new task is created, appropriate flags to control the behavior must be provided. One flag is available for each individual namespace: #define CLONE_NEWUTS 0x04000000 /* New utsname group? */ #define CLONE_NEWIPC 0x08000000 /* New ipcs */ #define CLONE_NEWUSER 0x10000000 /* New user namespace */ #define CLONE_NEWPID 0x20000000 /* New pid namespace */ #define CLONE_NEWNET 0x40000000 /* New network namespace */ Each task is associated with his own view of the namespaces: struct task_struct { ... /* namespaces */ struct nsproxy *nsproxy; ... } Because a pointer is used, a collection of sub-namespaces can be shared among multiple processes. This way, changes in a given namespace will be visible in all processes that belong to this namespace. Notice that support for namespaces must be enabled at compile time on a per-namespace basis. Generic support for namespaces is, however, always compiled in. This allows the kernel to avoid using different code for systems with and without namespaces. By providing a default namespace that is associated with every process unless specified differently, the namespace-aware code can always be used, but the results will be identical to a situation in which all properties are global and not wrapped up in namespaces if no active support for namespaces is compiled in. The initial global namespace is defined by init_nsproxy, which keeps pointers to the initial objects of the per-subsystem namespaces: struct nsproxy init_nsproxy = INIT_NSPROXY(init_nsproxy); #define INIT_NSPROXY(nsproxy) { \ .pid_ns = &init_pid_ns, \ .count = ATOMIC_INIT(1), \ 51 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 52 Chapter 2: Process Management and Scheduling .uts_ns = &init_uts_ns, \ .mnt_ns = NULL, \ INIT_NET_NS(net_ns) \ INIT_IPC_NS(ipc_ns) \ .user_ns = &init_user_ns, \ } The UTS Namespace The UTS namespace can be handled with particularly little effort because it only has to manage simple quantities and does not require a hierarchical organization. All relevant information is collected in an instance of the following structure: struct uts_namespace { struct kref kref; struct new_utsname name; }; kref is an embedded reference counter that can be used to track from how many places in the kernel an instance of struct uts_namespace is used (recall that Chapter 1 provides more information about the generic framework to handle reference counting). The information proper is contained in struct new_utsname: struct new_utsname { char sysname[65]; char nodename[65]; char release[65]; char version[65]; char machine[65]; char domainname[65]; }; The individual strings store the name of the system (Linux...), the kernel release, the machine name, and so on. The current values can be determined using the uname tool, but are also visible in /proc/sys/kernel/: wolfgang@meitner> cat /proc/sys/kernel/ostype Linux wolfgang@meitner> cat /proc/sys/kernel/osrelease 2.6.24 The initial settings are stored in init_uts_ns: init/version.c struct uts_namespace init_uts_ns = { ... .name = { .sysname = UTS_SYSNAME, .nodename = UTS_NODENAME, .release = UTS_RELEASE, .version = UTS_VERSION, 52 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 53 Chapter 2: Process Management and Scheduling .machine = UTS_MACHINE, .domainname = UTS_DOMAINNAME, }, }; The pre-processor constants are defined on various places across the kernel. UTS_RELEASE is, for instance, set in , which is dynamically generated at build time by the top-level Makefile. Notice that some parts of the UTS structure cannot be changed. For instance, it would not make sense to exchange sysname by anything else than Linux. It is, however, possible to change the machine name, for example. How does the kernel go about creating a new UTS namespace? This falls under the responsibility of the function copy_utsname. The function is called when a process is forked and the flag CLONE_NEWUTS specifies that a new UTS namespace is to be established. In this case, a copy of the previous instance of uts_namespace is generated, and a corresponding pointer is installed into the nsproxy instance of the current task. Nothing more is required! Since the kernel makes sure to always operate on the task-specific uts_namespace instance whenever a UTS value is read or set, changes for the current process will not be reflected in the parent, and changes in the parent will also not propagate toward the children. The User Namespace The user namespace is handled similarly in terms of data structure management: When a new user namespace is requested, a copy of the current user namespace is generated and associated with the nsproxy instance of the current task. However, the representation of a user namespace itself is slightly more complex: struct user_namespace { struct kref kref; struct hlist_head uidhash_table[UIDHASH_SZ]; struct user_struct *root_user; }; As before, kref is a reference counter that tracks in how many places a user_namespace instance is required. For each user in the namespace, an instance of struct user_struct keeps track of the individ- ual resource consumption, and the individual instances are accessible via the hash table uidhash_table. The exact definition of user_struct is not interesting for our purposes. It suffices to know that some sta- tistical elements like the number of open files or processes a user has are kept in there. What is much more interesting is that each user namespace accounts resource usage for its users completely detached from other namespaces — including accounting for the root user. This is possible because a new user_struct both for the current user and the root is created when a user namespace is cloned: kernel/user_namespace.c static struct user_namespace *clone_user_ns(struct user_namespace *old_ns) { struct user_namespace *ns; struct user_struct *new_user; ... ns = kmalloc(sizeof(struct user_namespace), GFP_KERNEL); ... 53 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 54 Chapter 2: Process Management and Scheduling ns->root_user = alloc_uid(ns, 0); /* Reset current->user with a new one */ new_user = alloc_uid(ns, current->uid); switch_uid(new_user); return ns; } alloc_uid is a helper function that allocates an instance of user_struct for a user with a given UID in the current namespace if none exists yet. Once an instance has been set up for both root and the current user, switch_uid ensures that the new user_struct will be used to account resources from now on. This essentially works by setting the user element of struct task_struct to the new user_struct instance. Notice that if support for user namespaces is not compiled in, cloning a user namespace is a null opera- tion: The default namespace is always used. 2.3.3 Process Identification Numbers Unix processes are always assigned a number to uniquely identify them in their namespace. This number is called the process identification number or PID for short. Each process generated with fork or clone is automatically assigned a new unique PID value by the kernel. Process Identifiers Each process is, however, not only characterized by its PID but also by other identifiers. Several types are possible: ❑ All processes in a thread group (i.e., different execution contexts of a process created by call- ing clone with CLONE_THREAD as we will see below) have a uniform thread group id (TGID). If a process does not use threads, its PID and TGID are identical. The main process in a thread group is called the group leader.Thegroup_leader element of the task structures of all cloned threads points to the task_struct instance of the group leader. ❑ Otherwise, independent processes can be combined into a process group (using the setpgrp sys- tem call). The pgrp elements of their task structures all have the same value, namely, the PID of the process group leader. Process groups facilitate the sending of signals to all members of the group, which is helpful for various system programming applications (see the literature on sys- tem programming, e.g., [SR05]). Notice that processes connected with pipes are contained in a process group. ❑ Several process groups can be combined in a session. All processes in a session have the same session ID which is held in the session element of the task structure. The SID can be set using the setsid system call. It is used in terminal programming but is of no particular relevance to us here. Namespaces add some additional complexity to how PIDs are managed. Recall that PID namespaces are organized in a hierarchy. When a new namespace is created, all PIDs that are used in this namespace are visible to the parent namespace, but the child namespace does not see PIDs of the parent name- space. However this implies that some tasks are equipped with more than one PID, namely, one per 54 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 55 Chapter 2: Process Management and Scheduling namespace they are visible in. This must be reflected in the data structures. We have to distinguish between local and global IDs: ❑ Global IDs are identification numbers that are valid within the kernel itself and in the initial namespace to which the init tasks started during boot belongs. For each ID type, a given global identifier is guaranteed to be unique in the whole system. ❑ Local IDs belong to a specific namespace and are not globally valid. For each ID type, they are valid within the namespace to which they belong, but identifiers of identical type may appear with the same ID number in a different namespace. The global PID and TGID are directly stored in the task struct, namely, in the elements pid and tgid: struct task_struct { ... pid_t pid; pid_t tgid; ... } Both are of type pid_t, which resolves to the type __kernel_pid_t; this, in turn, has to be defined by each architecture. Usually an int is used, which means that 232 different IDs can be used simultaneously. The session and process group IDs are not directly contained in the task structure itself, but in the struc- ture used for signal handling. task_struct->signal->__session denotes the global SID, while the global PGID is stored in task_struct->signal->__pgrp. The auxiliary functions set_task_session and set_task_pgrp are provided to modify the values. Managing PIDs In addition to these two fields, the kernel needs to find a way to manage all local per-namespace quanti- ties, as well as the other identifiers like TID and SID. This requires several interconnected data structures and numerous auxiliary functions that are discussed in the following. Data Structures Below I use the term ID to refer to any process identifier. I specify the identifier type explicitly (e.g., TGID for ‘‘thread group identifier’’) where this is necessary. A small subsystem known as a pid allocator is available to speed up the allocation of new IDs. Besides, the kernel needs to provide auxiliary functions that allow for finding the task structure of a process by reference to an ID and its type, and functions that convert between the in-kernel representation of IDs and the numerical values visible to userspace. Before I introduce the data structures required to represent IDs themselves, I need to discuss how PID namespaces are represented. The elements required for our purposes are as follows: struct pid_namespace { ... struct task_struct *child_reaper; 55 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 56 Chapter 2: Process Management and Scheduling ... int level; struct pid_namespace *parent; }; In reality, the structure also contains elements that are needed by the PID allocator to produce a stream of unique IDs, but these do not concern us now. What is interesting are the following elements: ❑ Every PID namespace is equipped with a task that assumes the role taken by init in the global picture. One of the purposes of init is to call wait4 for orphaned tasks, and this must likewise be done by the namespace-specific init variant. A pointer to the task structure of this task is stored in child_reaper. ❑ parent is a pointer to the parent namespace, and level denotes the depth in the namespace hier- archy. The initial namespace has level 0, any children of this namespace are in level 1, children of children are in level 2, and so on. Counting the levels is important because IDs in higher levels must be visible in lower levels. From a given level setting, the kernel can infer how many IDs must be associated with a task. Recall from Figure 2-3 that namespaces are hierarchically related. This clarifies the above definitions. PID management is centered around two data structures: struct pid is the kernel-internal representation of a PID, and struct upid represents the information that is visible in a specific namespace. The definition of both structures is as follows: struct upid { int nr; struct pid_namespace *ns; struct hlist_node pid_chain; }; struct pid { atomic_t count; /* lists of tasks that use this pid */ struct hlist_head tasks[PIDTYPE_MAX]; int level; struct upid numbers[1]; }; Since these and some other data structures are comprehensively interconnected, Figure 2-5 provides an overview about the situation before I discuss the individual components. As for struct upid, nr represents the numerical value of an ID, and ns is a pointer to the namespace to which the value belongs. All upid instances are kept on a hash table to which we will come in a moment, and pid_chain allows for implementing hash overflow lists with standard methods of the kernel. The definition of struct pid is headed by a reference counter count. tasks is an array with a hash list head for every ID type. This is necessary because an ID can be used for several processes. All task_struct instances that share a given ID are linked on this list. PIDTYPE_MAX denotes the number of ID types: 56 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 57 Chapter 2: Process Management and Scheduling enum pid_type { PIDTYPE_PID, PIDTYPE_PGID, PIDTYPE_SID, PIDTYPE_MAX }; level 0 level 1 level 2 1 PID_TYPE_PID 2 PID_TYPE_PGID 3 3 2 1 PID_TYPE_SID tasks struct task_struct task_structtask_struct struct pid pids [1] pid pid pid mode mode mode pid mode pid mode pids [2] pids [3] parent PID namespace pid_ hask struct upid numbers[level] struct upid Hashed by pid and namespace level n +1 entries struct pid_namespace Figure 2-5: Overview of data structures used to implement a namespace-aware representation of IDs. Notice that thread group IDs are not contained in this collection! This is because the thread group ID is simply given by the PID of the thread group leader, so a separate entry is not necessary. A process can be visible in multiple namespaces, and the local ID in each namespace will be different. level denotes in how many namespaces the process is visible (in other words, this is the depth of the containing namespace in the namespace hierarchy), and numbers contains an instance of upid for each level. Note that the array consists formally of one element, and this is true if a process is contained only in the global namespace. Since the element is at the end of the structure, additional entries can be added to the array by simply allocating more space. 57 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 58 Chapter 2: Process Management and Scheduling Since all task structures that share an identifier are kept on a list headed by tasks, a list element is required in struct task_struct: struct task_struct { ... /* PID/PID hash table linkage. */ struct pid_link pids[PIDTYPE_MAX]; ... }; The auxiliary data structure pid_link permits linking of task structures on the lists headed from struct pid: struct pid_link { struct hlist_node node; struct pid *pid; }; pid points to a pid instance to which the task belongs, and node is used as list element. A hash table is used to find the pid instance that belongs to a numeric PID value in a given namespace: kernel/pid.c static struct hlist_head *pid_hash; hlist_head is a kernel standard data element used to create doubly linked hash lists (Appendix C describes the structure of such lists and introduces several auxiliary functions for processing them). pid_hash is used as an array of hlist_heads. The number of elements is determined by the RAM con- figuration of the machine and lies between 24 = 16 and 212 = 4, 096. pidhash_init computes the apt size and allocates the required storage. Suppose that a new instance of struct pid has been allocated and set up for a given ID type type.Itis attached to a task structure as follows: kernel/pid.c int fastcall attach_pid(struct task_struct *task, enum pid_type type, struct pid *pid) { struct pid_link *link; link = &task->pids[type]; link->pid = pid; hlist_add_head_rcu(&link->node, &pid->tasks[type]); return 0; } A connection is made in both directions: The task structure can access the pid instance via task_struct->pids[type]->pid. Starting from the pid instance, the task can be found by iterating over 58 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 59 Chapter 2: Process Management and Scheduling the tasks[type] list. hlist_add_head_rcu is a standard function to traverse a list that additionally ensures as per the RCU mechanism (see Chapter 5) that the iteration is safe against race conditions that could arise when other kernel components manipulate the list concurrently. Functions The kernel provides a number of auxiliary functions to manipulate and scan the data structures described above. Essentially the kernel must be able to fulfill two different tasks: 1. Given a local numerical ID and the corresponding namespace, find the task structure that is described by this tuple. 2. Given a task structure, an ID type, and a namespace, obtain the local numerical ID. Let us first concentrate on the case in which a task_struct instance must be converted into a numerical ID. This is a two-step process: 1. Obtain the pid instance associated with the task structure. The auxiliary functions task_pid, task_tgid, task_pgrp,andtask_session are provided for the different types of IDs. This is simple for PIDs: static inline struct pid *task_pid(struct task_struct *task) { return task->pids[PIDTYPE_PID].pid; } Obtaining a TGID works similarly because it is nothing other than the PID of the tread group leader. The element to grab is task->group_leader->pids[PIDTYPE_PID].pid. Finding out a process group ID requires using PIDTYPE_PGID as array index. However, it must again be taken from the pid instance of the process group leader: static inline struct pid *task_pgrp(struct task_struct *task) { return task->group_leader->pids[PIDTYPE_PGID].pid; } 2. Once the pid instance is available, the numerical ID can be read off from the uid information available in the numbers array in struct pid: kernel/pid.c pid_t pid_nr_ns(struct pid *pid, struct pid_namespace *ns) { struct upid *upid; pid_t nr = 0; if (pid && ns->level <= pid->level) { upid = &pid->numbers[ns->level]; if (upid->ns == ns) nr = upid->nr; } return nr; } 59 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 60 Chapter 2: Process Management and Scheduling Because a parent namespace sees PIDs in child namespaces, but not vice versa, the kernel has to ensure that the current namespace level is less than or equal to the level in which the local PID was generated. It is also important to note that the kernel need only worry about generating global PIDs: All other ID types in the global namespace will be mapped to PIDs, so there is no need to generate, for instance, global TGIDs or SIDs. Instead of using pid_nr_ns in the second step, the kernel could also employ one of these auxiliary functions: ❑ pid_vnr returns the local PID seen from the namespace to which the ID belongs. ❑ pid_nr obtains the global PID as seen from the init process. Both rely on pid_nr_ns and automatically select the proper level: 0 for the global PID, and pid->level for the local one. The kernel provides several helper functions that combine the described steps: kernel/pid.c pid_t task_pid_nr_ns(struct task_struct *tsk, struct pid_namespace *ns) pid_t task_tgid_nr_ns(struct task_struct *tsk, struct pid_namespace *ns) pid_t task_pgrp_nr_ns(struct task_struct *tsk, struct pid_namespace *ns) pid_t task_session_nr_ns(struct task_struct *tsk, struct pid_namespace *ns) Their meaning is obvious from the function names, so we need not add anything further. Now let us turn our attention to how the kernel can convert a numerical PID together with the namespace into a pid instance. Again two steps are required: 1. To determine the pid instance (the in-kernel representation of a PID) given the local numer- ical PID of a process and the associated namespace (the userspace representation of a PID), the kernel must employ a standard hashing scheme: First, the array index in pid_hash is computed from the PID and namespace pointers,5 and then the hash list is traversed until the desired element has been found. This is handled by the auxiliary function find_pid_ns: kernel/pid.c struct pid * fastcall find_pid_ns(int nr, struct pid_namespace *ns) Instances of struct upid are kept on the hash, but since these are directly contained in struct pid, the kernel can infer the desired information using the container_of mechanism (see Appendix C). 2. pid_task extracts the first task_struct instance that is queued on the list pid->tasks[type]. These two steps are performed by the auxiliary function find_task_by_pid)_type_ns: kernel/pid.c struct task_struct *find_task_by_pid_type_ns(int type, int nr, struct pid_namespace *ns) 5For this purpose, the kernel uses multiplicative hashing with a prime number that is in a golden ratio to the largest number that can be represented in a machine word. For details, refer to [Knu97]. 60 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 61 Chapter 2: Process Management and Scheduling { return pid_task(find_pid_ns(nr, ns), type); } Some simpler auxiliary functions build on the most general find_task_by_pid_type_ns: ❑ find_task_by_pid_ns(pid_t nr, struct pid_namespace *ns) finds a task_struct instance given a numerical PID and the namespace of the task. ❑ find_task_by_vpid(pid_t vnr) finds a task by its local numerical PID. ❑ find_task_by_pid(pid_t nr) finds a task by its global numerical PID. find_task_by_pid is required at many points in the kernel sources because a large number of process- specific operations (e.g., sending a signal using kill) identify their target process by means of its PID. Generating Unique PIDs In addition to managing PIDs, the kernel is also responsible for providing a mechanism to generate unique PIDs that have not yet been assigned. In this case, the differences between the various PID types can be ignored because unique numbers need only be generated for PIDs in the classical Unix sense. All other identifiers can be derived from the PID, as we will see when discussing fork and clone below. In the sections that follow, the term PID once again refers to the classical Unix process identifier (PIDTYPE_PID). To keep track of which PIDs have been allocated and which are still free, the kernel uses a large bitmap in which each PID is identified by a bit. The value of the PID is obtained from the position of the bit in the bitmap. Allocating a free PID is then restricted essentially to looking for the first bit in the bitmap whose value is 0; this bit is then set to 1. Conversely, freeing a PID can be implemented by ‘‘toggling‘‘ the corresponding bit from 1 to 0. These operations are implemented using kernel/pid.c static int alloc_pidmap(struct pid_namespace *pid_ns) to reserve a PID, and kernel/pid.c static fastcall void free_pidmap(struct pid_namespace *pid_ns, int pid) to free a PID. How they are implemented does not concern us here, but naturally, they must work on a per-namespace basis. When a new process is created, it may be visible in multiple namespaces. For each of them a local PID must be generated. This is handled in alloc_pid: kernel/pid.c struct pid *alloc_pid(struct pid_namespace *ns) { struct pid *pid; enum pid_type type; int i, nr; 61 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 62 Chapter 2: Process Management and Scheduling struct pid_namespace *tmp; struct upid *upid; ... tmp = ns; for (i = ns->level; i >= 0; i--) { nr = alloc_pidmap(tmp); ... pid->numbers[i].nr = nr; pid->numbers[i].ns = tmp; tmp = tmp->parent; } pid->level = ns->level; ... Starting at the level of the namespace in which the process is created, the kernel goes down to the initial, global namespace and creates a local PID for each. All upids that are contained in struct pid are filled with the newly generated PIDs. Each upid instance must be placed on the PID hash: kernel/pid.c for (i = ns->level; i >= 0; i--) { upid = &pid->numbers[i]; hlist_add_head_rcu(&upid->pid_chain, &pid_hash[pid_hashfn(upid->nr, upid->ns)]); } ... return pid; } 2.3.4 Task Relationships In addition to the relationships resulting from ID links, the kernel is also responsible for managing the ‘‘family relationships‘‘ established on the basis of the Unix model of process creation. The following terminology is used in this context: ❑ If process A forks to generate process B, A is known as the parent process and B as the child process.6 If process B forks again to create a further process C, the relationship between A and C is sometimes referred to as a grandparent and grandchild relationship. ❑ If process A forks several times therefore generating several child processes B1, B2, ..., Bn,the relationship between the Bi processes is known as a siblings relationship. Figure 2-6 illustrates the possible family relationships graphically. The task_struct task data structure provides two list heads to help implement these relationships: struct task_struct { ... struct list_head children; /* list of my children */ 6Unlike natural families, a process has only one parent. 62 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 63 Chapter 2: Process Management and Scheduling struct list_head sibling; /* linkage in my parent’s children list */ ... } ❑ children is the list head for the list of all child elements of the process. ❑ siblings is used to link siblings with each other. Parent prcess Child prcess Children sibling sibling sibling Figure 2-6: Family relationships between processes. New children are placed at the start of the siblings list, meaning that the chronological sequence of forks can be reconstructed.7 2.4 Process Management System Calls In this section, I discuss the implementation of the fork and exec system call families. Normally, these calls are not issued directly by applications but are invoked via an intermediate layer — the C standard library — that is responsible for communication with the kernel. The methods used to switch from user mode to kernel mode differ from architecture to architecture. In Appendix A, I describe at length the mechanisms used to switch between these two modes and also explain how parameters are exchanged between userspace and kernel space. For the time being, it is sufficient to regard the kernel as a ‘‘program library‘‘ used by the C standard library as mentioned briefly in Chapter 1. 2.4.1 Process Duplication The traditional Unix system call to duplicate a process is fork. However, it is not the only call imple- mented by Linux for this purpose — in fact, there are three: 7Kernel versions before 2.6.21 had three helper functions: younger_sibling, older_sibling,andeldest_child,which gave some aid in accessing the described lists and their elements. They were used to produce debugging output, which had, how- ever, not proved very useful, so it was removed. Patch author Ingo Molnar noticed that the corresponding code was among the oldest elements of the kernel and noted this accordingly. This led another well-known developer to sign off the patch as Linus ’snif’ Torvalds ... 63 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 64 Chapter 2: Process Management and Scheduling 1. fork is the heavy-weight call because it creates a full copy of the parent process that then executes as a child process. To reduce the effort associated with this call, Linux uses the copy- on-write technique, discussed below. 2. vfork is similar to fork but does not create a copy of the data of the parent process. Instead, it shares the data between the parent and child process. This saves a great deal of CPU time (and if one of the processes were to manipulate the shared data, the other would notice auto- matically). vfork is designed for the situation in which a child process just generated immediately exe- cutes an execve system call to load a new program. The kernel also guarantees that the parent process is blocked until the child process exits or starts a new program. Quoting the manual page vfork(2), it is ‘‘rather unfortunate that Linux revived this specter from the past.’’ Since fork uses copy-on-write, the speed argument for vfork does not really count anymore, and its use should therefore be avoided. 3. clone generates threads and enables a decision to be made as to exactly which elements are to be shared between the parent and the child process and which are to be copied. Copy on Write The kernel uses the copy-on-write technique (COW) to prevent all data of the parent process from being copied when fork is executed. This technique exploits the fact that processes normally use only a fraction of their pages in memory.8 When fork is called, the kernel would usually create an identical copy of each memory page of the parent process for the child process. This has two very negative effects: 1. A large amount of RAM, a scarce resource, is used. 2. The copy operation takes a long time. The negative impact is even greater if the application loads a new program using exec immediately after process duplication. This means, in effect, that the preceding copy operation was totally superfluous as the process address space is reinitialized and the data copied are no longer needed. The kernel can get around this problem by using a trick. Not the entire address space of the process but only its page tables are copied. These establish the link between virtual address space and physical pages as described briefly in Chapter 1 and at length in Chapters 3 and 4. The address spaces of parent and child processes then point to the same physical pages. Of course, parent and child processes must not be allowed to modify each other’s pages,9 which is why the page tables of both processes indicate that only read access is allowed to the pages — even though they could be written to in normal circumstances. Providing that both processes have only read access to their pages in memory, data sharing between the two is not a problem because no changes can be made. As soon as one of the processes attempts to write to the copied pages, the processor reports an access error to the kernel (errors of this kind are called page faults). The kernel then references additional memory management data structures (see Chapter 4) to check whether the page can be accessed in Read and Write mode or in Read mode only — if the latter is true, a segmentation fault must be reported to the 8The pages most frequently accessed by the process are called the working set. 9With the exception of pages explicitly shared by both processes. 64 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 65 Chapter 2: Process Management and Scheduling process. As you see in Chapter 4, the actual implementation of the page fault handler is more complicated because other aspects, such as swapped-out pages, must also be taken into account. The condition in which a page table entry indicates that a page is ‘‘Read Only’’ although normally it would be writable allows the kernel to recognize that the page is, in fact, a COW page. It therefore creates a copy of the page that is assigned exclusively to the process — and may therefore also be used for write operations. How the copy operation is implemented is not discussed until Chapter 4 because extensive background knowledge of memory management is required. The COW mechanism enables the kernel to delay copying of memory pages for as long as possible and — more importantly — to make copying unnecessary in many cases. This saves a great deal of time. Executing System Calls The entry points for the fork, vfork,andclone system calls are the sys_fork, sys_vfork,andsys_clone functions. Their definitions are architecture-dependent because the way in which parameters are passed between userspace and kernel space differs on the various architectures (see Chapter 13 for further infor- mation). The task of the above functions is to extract the information supplied by userspace from the registers of the processors and then to invoke the architecture-independent do_fork function responsible for process duplication. The prototype of the function is as follows. kernel/fork.c long do_fork(unsigned long clone_flags, unsigned long stack_start, struct pt_regs *regs, unsigned long stack_size, int __user *parent_tidptr, int __user *child_tidptr) The function requires the following arguments: ❑ Aflagset(clone_flags) to specify duplication properties. The low byte specifies the signal num- ber to be sent to the parent process when the child process terminates. The higher bytes hold various constants discussed below. ❑ The start address of the user mode stack (start_stack)tobeused. ❑ A pointer to the register set holding the call parameters in raw form (regs). The data type used is the architecture-specific struct pt_regs structure, which holds all registers in the order in which they are saved on the kernel stack when a system call is executed (more information is provided in Appendix A). ❑ The size of the user mode stack (stack_size). This parameter is usually unnecessary and set to 0. ❑ Two pointers to addresses in userspace (parent_tidptr and child_tidptr) that hold the TIDs of the parent and child processes. They are needed for the thread implementation of the NPTL (Native Posix Threads Lilbrary) library. I discuss their meaning below. The different fork variants are distinguished primarily by means of the flag set. On most architectures,10 the classical fork call is implemented in the same way as on IA-32 processors. 10Exception: Sparc(64) systems that access do_fork via sparc_do_fork. IA-64 kernels only provide a single system call, sys_clone2, which is used to implement fork, vfork,andclone in userspace. Both sys_clone2 and sparc_do_fork eventually rely on do_fork. 65 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 66 Chapter 2: Process Management and Scheduling arch/x86/kernel/process_32.c asmlinkage int sys_fork(struct pt_regs regs) { return do_fork(SIGCHLD, regs.esp, ®s, 0, NULL, NULL); } The only flag used is SIGCHLD. This means that the SIGCHLD signal informs the parent process once the child process has terminated. Initially, the same stack (whose start address is held in the esp register on IA-32 systems) is used for the parent and child processes. However, the COW mechanism creates a copy of the stack for each process if it is manipulated and therefore written to. If do_fork was successful, the PID of the newly created task is returned as the result of the system call. Otherwise the (negative) error code is returned. The implementation of sys_vfork differs only slightly from that of sys_fork in that additional flags are used (CLONE_VFORK and CLONE_VM whose meaning is discussed below). sys_clone is also implemented in a similar way to the above calls with the difference that do_fork is invoked as follows: arch/x86/kernel/process_32.c asmlinkage int sys_clone(struct pt_regs regs) { unsigned long clone_flags; unsigned long newsp; int __user *parent_tidptr, *child_tidptr; clone_flags = regs.ebx; newsp = regs.ecx; parent_tidptr = (int __user *)regs.edx; child_tidptr = (int __user *)regs.edi; if (!newsp) newsp = regs.esp; return do_fork(clone_flags, newsp, ®s, 0, parent_tidptr, child_tidptr); } The clone flags are no longer permanently set but can be passed to the system call as parameters in various registers. Thus, the first part of the function deals with extracting these parameters. Also, the stack of the parent process is not copied; instead, a new address (newsp) can be specified for it. (This is required to generate threads that share the address space with the parent process but use their own stack in this address space.) Two pointers (parent_tidptr and child_tidptr) in userspace are also specified for purposes of communication with thread libraries. Their meaning is discussed in Section 2.4.1. Implementation of do_fork All three fork mechanisms end up in do_fork in kernel/fork.c (an architecture-independent function), whose code flow diagram is shown in Figure 2-7. do_fork begins with an invocation of copy_process, which performs the actual work of generating a new process and reusing the parent process data specified by the flags. Once the child process has been generated, the kernel must carry out the following concluding operations: 66 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 67 Chapter 2: Process Management and Scheduling Determine PID do_fork copy_process Initialize vfork completion handler (only with CLONE_VFORK) and ptrace flags wake_up_new_task CLONE_VFORK set? wait_for_completion Figure 2-7: Code flow diagram for do_fork. ❑ Since fork returns the PID of the new task, it must be obtained. This is complicated because the fork operation could have opened a new PID namespace if the flag CLONE_NEWPID was set. If this is the case, then task_pid_nr_ns is required to obtain the PID that was selected for the new process in the parent namespace, that is, the namespace of the process that issued fork. If the PID namespace remains unchanged, calling task_pid_vnr is enough to obtain the local PID because old and new processes will live in the same namespace. kernel/fork.c nr = (clone_flags & CLONE_NEWPID) ? task_pid_nr_ns(p, current->nsproxy->pid_ns) : task_pid_vnr(p); ❑ If the new process is to be monitored with Ptrace (see Chapter 13), the SIGSTOP signal is sent to the process immediately after generation to allow an attached debugger to examine its data. ❑ The child process is woken using wake_up_new_task; in other words, the task structure is added to the scheduler queue. The scheduler also gets a chance to specifically handle newly started tasks, which, for instance, allows for implementing a policy that gives new tasks a good chance to run soon, but also prevents processes that fork over and over again to consume all CPU time. If a child process begins to run before the parent process, this can greatly reduce copying effort, especially if the child process issues an exec call after fork. However, keep in mind that enqueu- ing a process in the scheduler data structures does not mean that the child process begins to execute immediately but rather that it is available for selection by the scheduler. ❑ If the vfork mechanism was used (the kernel recognizes this by the fact that the CLONE_VFORK flag is set), the completions mechanism of the child process must be enabled. The vfork_done element of the child process task structure is used for this purpose. With the help of the wait_for_completion function, the parent process goes to sleep on this variable until the child process exits. When a process terminates (or a new application is started with execve), the kernel automatically invokes complete(vfork_done). This wakes all processes sleeping on it. In Chapter 14, I discuss the implementation of completions in greater detail. By adopting this approach, the kernel ensures that the parent process of a child process gener- ated using vfork remains inactive until either the child process exits or a new process is exe- cuted. The temporary inactivity of the parent process also ensures that both processes do not interfere with each other or manipulate each other’s address space. 67 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 68 Chapter 2: Process Management and Scheduling Copying Processes In do_fork the bulk of the work is done by the copy_process function, whose code flow diagram is shown in Figure 2-8. Notice that the function has to handle the main work for the three system calls fork, vfork,andclone. Check resource limits Check flags copy_process dup_task_struct sched_fork copy_semundo copy_files copy_fs copy_sighand copy_signal copy_mm copy_namespaces copy_thread Initialize task structure Copy/share process components Set IDs, task relationships, etc. Figure 2-8: Code flow diagram for copy_process. Because the kernel has to deal with a large number of special and very specific situations, let’s restrict our description to a slightly simplified version of the function so as not to lose sight of the most important aspects in a myriad of details. Quite a number of flags control the behavior of process duplication. They are all well documented in the clone(2) man page, and instead of repeating them here, I advise you to just take a look into it — or, for that matter, any good text on Linux systems programming. More interesting is that there are some flag combinations that do not make sense, and the kernel has to catch these. For instance, it does not make sense to request creation of a new namespace (CLONE_NEWNS), on the one hand, but also express the 68 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 69 Chapter 2: Process Management and Scheduling desire to share all filesystem information with the parent (CLONE_FS). It’s not complicated to catch this combination and return an error code: kernel/fork.c static struct task_struct *copy_process(unsigned long clone_flags, unsigned long stack_start, struct pt_regs *regs, unsigned long stack_size, int __user *child_tidptr, struct pid *pid) { int retval; struct task_struct *p; int cgroup_callbacks_done = 0; if ((clone_flags & (CLONE_NEWNS|CLONE_FS)) == (CLONE_NEWNS|CLONE_FS)) return ERR_PTR(-EINVAL); ... This is also a good place to recall from the introduction that Linux sometimes has to return a pointer if an operation succeeds, and an error code if something fails. Unfortunately, the C language only allows a single direct return value per function, so any information about possible errors has to be encoded into the pointer. While pointers can in general point to arbitrary locations in memory, each architecture supported by Linux has a region in virtual address space that starts from virtual address 0 and goes at least 4 KiB far where no senseful information can live. The kernel can thus reuse this pointer range to encode error codes: If the return value of fork points to an address within the aforementioned range, then the call has failed, and the reason can be determined by the numerical value of the pointer. ERR_PTR is a helper macro to perform the encoding of the numerical constant -EINVAL (invalid operation) into a pointer. Some further flag checks are required: ❑ When a thread is created with CLONE_THREAD, signal sharing must be activated with CLONE_SIGHAND. Individual threads in a thread group cannot be addressed by a signal. ❑ Shared signal handlers can only be provided if the virtual address space is shared between par- ent and child (CLONE_VM). Transitive thinking reveals that threads, therefore, also have to share the address space with the parent. Once the kernel has established that the flag set does not contradict itself, dup_task_struct is used to create an identical copy of the task structure of the parent process. The new task_struct instance for the child can be allocated at any point in kernel memory that happens to be free (see Chapter 3, in which the allocation mechanisms used for this purpose are described). The task structures for parent and child differ only in one element: A new kernel mode stack is allocated for the new process. A pointer to it is stored in task_struct->stack. Usually the stack is stored in a union with thread_info, which holds all required processor-specific low-level information about the thread. union thread_union { struct thread_info thread_info; unsigned long stack[THREAD_SIZE/sizeof(long)]; }; 69 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 70 Chapter 2: Process Management and Scheduling In principle, individual architectures are, however, free to store whatever they like in the stack pointer if they signal this to the kernel by setting the pre-processor constant __HAVE_THREAD_FUNCTIONS.Inthis case, they must provide their own implementations of task_thread_info and task_stack_page,which allows for obtaining the thread information and the kernel mode stack for a given task_struct instance. Additionally, they must implement the function setup_thread_stack that is called in dup_task_struct to create a destination for stack. Currently, only IA-64 and m68k do not rely on the default methods of the kernel. On most architectures, one or two memory pages are used to hold an instance of thread_union.On IA-32, two pages are the default setting, and thus the available kernel stack size is slightly less than 8 KiB because part is occupied by the thread_info instance. Note, though, that the configuration option 4KSTACKS decreases the stack size to 4 KiB and thus to one page. This is advantageous if a large number of processes is running on the system because one page per process is saved. On the other hand, it can lead to problems with external drivers that often tend to be ‘‘stack hogs,’’ for example, use too much stack space. All central parts of the kernel that are part of the standard distribution have been designed to operate smoothly also with a stack size of 4 KiB, but problems can arise (and unfortunately have in the past) if binary-only drivers are required, which often have a tendency to clutter up the available stack space. thread_info holds process data that needs to be accessed by the architecture-specific assembly language code. Although the structure is defined differently from processor to processor, its contents are similar to the following on most systems. struct thread_info { struct task_struct *task; /* main task structure */ struct exec_domain *exec_domain; /* execution domain */ unsigned long flags; /* low level flags */ unsigned long status; /* thread-synchronous flags */ __u32 cpu; /* current CPU */ int preempt_count; /* 0 => preemptable, <0 => BUG */ mm_segment_t addr_limit; /* thread address space */ struct restart_block restart_block; } ❑ task is a pointer to the task_struct instance of the process. ❑ exec_domain is used to implement execution domains with which different ABIs (Application Binary Interfaces) can be implemented on a machine type (e.g., to run 32-bit applications on an AMD64 system in 64-bit mode). ❑ flags can hold various process-specific flags, two of which are of particular interest to us: ❑ TIF_SIGPENDING is set if the process has pending signals. ❑ TIF_NEED_RESCHED indicates that the process should be or would like to be replaced with another process by the scheduler. Other possible constants — some hardware-specific — which are, however, hardly ever used, are available in . ❑ cpu specifies the number of the CPU on which a process is just executing (important on multi- processor systems — very easy to determine on single-processor systems). 70 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 71 Chapter 2: Process Management and Scheduling ❑ preempt_count is a counter needed to implement kernel preemption, discussed in Section 2.8.3. ❑ addr_limit specifies up to which address in virtual address space a process may use. As already noted, there is a limit for normal processes, but kernel threads may access the entire virtual address space, including the kernel-only portions. (This does not represent any kind of restric- tion on how much RAM a process may allocate.) Recall that I have touched on the separation between user and kernel address space in the Introduction, and will come back to the details in Section 4. ❑ restart_block is needed to implement the signal mechanism (see Chapter 5). Figure 2-9 shows the relationship between task_struct, thread_info and the kernel stack. When a particular component of the kernel uses too much stack space, the kernel stack will crash into the thread information, and this will most likely lead to severe failures. Besides, this can also lead to wrong informa- tion when an emergency stack trace is printed, so the kernel provides the function kstack_end to decide if a given address is within the valid portion of the stack or not. Kernel stack thread_info task_struct thread_info->task task_struct->stack INIT_THREAD_SIZE Figure 2-9: Relationship between task_struct, thread_info,andthe kernel stack of a process. dup_task_struct copies the contents of task_struct and thread_info instances of the parent process, but the stack pointer is set to the new thread_info instance. This means that the task structures of parent and child processes are absolutely identical at this point except for the stack pointer. The child will, however, be modified in the course of copy_process. There are also two symbols named current and current_thread_info that are defined as macros or functions by all architectures. Their meanings are as follows: ❑ current_thread_info delivers a pointer to the thread_info instance of the process currently executing. The address can be determined from the kernel stack pointer because the instance is always located at the top of the stack.11Because a separate kernel stack is used for each process, the process to stack assignment is unique. ❑ current specifies the address of the task_struct instance of the current process. This function appears very frequently in the sources. The address can be determined using get_thread_info: current = current_thread_info()->task. 11The pointer to the kernel stack is usually held in a specially reserved register. Some architectures, especially IA-32 and AMD64, use a different solution discussed in Section A.10.3. 71 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 72 Chapter 2: Process Management and Scheduling Let us return to copy_process.Afterdup_task_struct has succeeded, the kernel checks if the maximam number of processes allowed for a particular user are exceeded with the creation of the new task: kernel/fork.c if (atomic_read(&p->user->processes) >= p->signal->rlim[RLIMIT_NPROC].rlim_cur) { if (!capable(CAP_SYS_ADMIN) && !capable(CAP_SYS_RESOURCE) && p->user != current->nsproxy->user_ns->root_user) goto bad_fork_free; } ... The per-user resource counters for the user owning the current process are kept in an instance of user_struct that is accessible via task_struct->user, and the number of processes currently held by a particular user is stored in user_struct->processes. If this value exceeds the limit set by rlimit, task creation is aborted — unless the current user is assigned special capabilities (CAP_SYS_ADMIN or CAP_SYS_RESOURCE) or is the root user. Checking for the root user is interesting: Recall from above that each PID namespace has its own root user. This must now be taken into account in the above check. If resource limits do not prevent process creation, the interface function sched_fork is called to give the scheduler a chance to set up things for the new task. Before the introduction of the CFQ scheduler in kernel 2.6.23, this was more complicated because the remaining time slice of the parent had to be distributed between parent and child. Since the new scheduler does not require time slices anymore, things are a lot simpler now. Essentially, the routines initialize statistical fields and on multi-processor systems probably re-balance the available processes between the CPUs if this is necessary. Besides, the task state is set to TASK_RUNNING — which is not really true since the new process is, in fact, not yet running. However, this prevents any other part of the kernel from trying to change the process state from non-running to running and scheduling the new process before its setup has been completely finished. Alargenumberofcopy_xyz routines are then invoked to copy or share the resources of specific kernel subsystems. The task structure contains pointers to instances of data structures that describe a sharable or cloneable resource. Because the task structure of the child starts out as an exact copy of the parent’s task structure, both point to the same resource-specific instances initially. This is illustrated in Figure 2-10. Suppose we have two resources: res_abc and res_def. Initially the corresponding pointers in the task structure of the parent and child process point to the same instance of the resource-specific data structure in memory. If CLONE_ABC is set, then both processes will share res_abc. This is already the case, but it is additionally necessary to increment the reference counter of the instance to prevent the associated memory space from being freed too soon — memory may be relinquished to memory management only when it is no longer being used by a process. If either parent or child modifies the shared resource, the change will be visible in both processes. If CLONE_ABC is not set, then a copy of res_abc is created for the child process, and the resource counter of the new copy is initialized to 1. Consequently, if parent or child modifies the resource, then changes will not propagate to the other process in this case. As a general rule, the fewer the number of CLONE flags set, the less work there is to do. However, this gives parent and child processes more opportunities to mutually manipulate their data structures — and this must be taken into consideration when programming applications. 72 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 73 Chapter 2: Process Management and Scheduling res_abc res_def count = m count = m task_struct(parent) task_struct(child) !CLONE_ABCCLONE_ABC count = 1 count = n count = n+1 Figure 2-10: When a new thread is created, resources of the parent can either be shared or copied. Deciding if a resource can be cloned or must be copied is done through numerous helper routines — one for each resource. It is not my intention here to discuss the (rather uninteresting) implementations of the various copy_xyz functions, but I summarize their effects below. I will introduce the data structures asso- ciated with every process component when I discuss the respective subsystem in detail in the following chapters. ❑ copy_semundo uses the System V semaphores of the parent process if COPY_SYSVSEM is set (see Chapter 5). ❑ copy_files uses the file descriptors of the parent process if CLONE_FILES is set. Otherwise, a new files structure is generated (see Chapter 8) that contains the same information as the parent process. This information can be modified independently of the original structure. ❑ copy_fs uses the filesystem context (task_struct->fs) of the parent process if CLONE_FS is set. This is an fs_struct type structure that holds, for example, the root directory and the current working directory of the process (see Chapter 8 for detailed information). ❑ copy_sighand uses the signal handlers of the parent process (task_struct->sighand)if CLONE_SIGHAND or CLONE_THREAD is set. Chapter 5 discusses the struct sighand_struct structure used in more detail. ❑ copy_signal uses the non-handler-specific part of signal handling (task_struct->signal,see Chapter 5) together with the parent process if CLONE_THREAD is set. ❑ copy_mm causes the parent process and child process to share the same address space if COPY_MM is set. In this case, both processes use the same instance of mm_struct (see Chapter 4) to which task_struct->mm points. If copy_mm is not set, it does not mean that the entire address space of the parent process is copied. The kernel does, in fact, create a copy of the page tables but does not copy the actual contents of the pages. This is done using the COW mechanism only if one of the two processes writes to one of the pages. ❑ copy_namespaces has special call semantics. It is used to set up namespaces for the child process. Recall that several CLONE_NEWxyz flags control which namespaces are shared with the 73 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 74 Chapter 2: Process Management and Scheduling parent. However, the semantics are opposite to all other flags: If CLONE_NEWxyz is not specified, then the specific namespace is shared with the parent. Otherwise, a new namespace is generated. copy_namespace is a dispatcher that executes a copy routine for each possible namespace. The individual copy routines, however, are not too interesting because they essentially copy data or make already existing instances shared by means of reference counter management, so I will not discuss their implementation in detail. ❑ copy_thread is — in contrast to all other copy operations discussed here — an architecture- specific function that copies the thread-specific data of a process. Thread-specific in this context does not refer to any of the CLONE flags or to the fact that the operation is performed for threads only and not for full processes. It simply means that all data that contribute to the architecture-specific execution context are copied (the term thread is used with more than one meaning in the kernel). What is important is to fill the elements of task_struct->thread.Thisisastructureofthe thread_struct type whose definition is architecture-dependent. It holds all registers (plus other information) needed by the kernel to save and restore process contents during low-level switch- ing between tasks. Intimate knowledge of the various CPUs is needed to understand the layout of the individual thread_struct structures. A full discussion of these structures is beyond the scope of this book. However, Appendix A includes some information relating to the contents of the structures on several systems. Back in copy_process, the kernel must fill in various elements of the task structure that differ between parent and child. These include the following: ❑ The various list elements contained in task_struct, for instance, sibling and children. ❑ The interval timer elements cpu_timers (see Chapter 15). ❑ The list of pending signals (pending) discussed in Chapter 5. After allocating a new pid instance for the task with the mechanisms described before, they are stored in the task structure. For threads, the thread group ID is the same as that of the forking process: kernel/fork.c p->pid = pid_nr(pid); p->tgid = p->pid; if (clone_flags & CLONE_THREAD) p->tgid = current->tgid; ... Recall that pid_nr computes the global numerical PID for a given pid instance. For regular processes, the parent process is the forking process. This is different for threads: Since they are seen as the second (or third, or fourth,...) line of execution within the generating process, their parent is the parent’s parent. This is easier to express in code than in words: kernel/fork.c if (clone_flags & (CLONE_PARENT|CLONE_THREAD)) p->real_parent = current->real_parent; else 74 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 75 Chapter 2: Process Management and Scheduling p->real_parent = current; p->parent = p->real_parent; Regular processes that are not threads can trigger the same behavior by setting CLONE_PARENT.Another correction is required for threads: The thread group leader of a regular process is the process itself. For a thread, the group leader is the group leader of the current process: kernel/fork.c p->group_leader = p; if (clone_flags & CLONE_THREAD) { p->group_leader = current->group_leader; list_add_tail_rcu(&p->thread_group, &p->group_leader->thread_group); ... } The new process must then be linked with its parent process by means of the children list. This is handled by the auxiliary macro add_parent. Besides, the new process must be included in the ID data structure network as described in Section 2.3.3. kernel/fork.c add_parent(p); if (thread_group_leader(p)) { if (clone_flags & CLONE_NEWPID) p->nsproxy->pid_ns->child_reaper = p; set_task_pgrp(p, task_pgrp_nr(current)); set_task_session(p, task_session_nr(current)); attach_pid(p, PIDTYPE_PGID, task_pgrp(current)); attach_pid(p, PIDTYPE_SID, task_session(current)); } attach_pid(p, PIDTYPE_PID, pid); ... return p; } thread_group_leader checks only whether pid and tgid of the new process are identical. If so, the process is the leader of a thread group. In this case, some more work is necessary: ❑ Recall that processes in a process namespace that is not the global namespace have their own init task. If a new PID namespace was opened by setting CLONE_NEWPID,thisrolemustbe assumed by the task that called clone. ❑ The new process must be added to the current task group and session. This allows for bringing some of the functions discussed above to good use. Finally, the PID itself is added to the ID network. This concludes the creation of a new process! Special Points When Generating Threads Userspace thread libraries use the clone system call to generate new threads. This call supports flags (other than those discussed above) that produce certain special effects in the copy_process (and in the 75 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 76 Chapter 2: Process Management and Scheduling associated invoked functions). For the sake of simplicity, I omitted these flags above. However, it should be remembered that the differences between a classical process and a thread in the Linux kernel are relatively fluid and both terms are often used as synonyms (thread is also frequently used to mean the architecture-dependent part of a process as mentioned above). In this section, I concentrate on the flags used by user thread libraries (above all, NPTL) to implement multithreading capabilities. ❑ CLONE_PARENT_SETTID copies the PID of the generated thread to a point in userspace specified in the clone call (parent_tidptr, the pointer is passed to clone)12: kernel/fork.c if (clone_flags & CLONE_PARENT_SETTID) put_user(nr, parent_tidptr); The copy operation is performed in do_fork before the task structure of the new thread is initial- ized and before its data are created with the copy operations. ❑ CLONE_CHILD_SETTID first causes a further userspace pointer (child_tidptr) passed to clone to be stored in the task structure of the new process. kernel/fork.c p->set_child_tid = (clone_flags & CLONE_CHILD_SETTID) ? child_tidptr : NULL; The schedule_tail function invoked when the new process is executed for the first time copies the current PID to this address. kernel/schedule.c asmlinkage void schedule_tail(struct task_struct *prev) { ... if (current->set_child_tid) put_user(task_pid_vnr(current), current->set_child_tid); ... } ❑ CLONE_CHILD_CLEARTID has the initial effect in copy_process that the userspace pointer child_tidptr is stored in the task structure — but this time in a different element. kernel/fork.c p->clear_child_tid = (clone_flags & CLONE_CHILD_CLEARTID) ? child_tidptr: NULL; When the process terminates,13 0 is written to the address defined in clear_child_tid.14 kernel/fork.c void mm_release(struct task_struct *tsk, struct mm_struct *mm) { if (tsk->clear_child_tid && atomic_read(&mm->mm_users) > 1) { u32 __user * tidptr = tsk->clear_child_tid; tsk->clear_child_tid = NULL; put_user(0, tidptr); 12put_user is used to copy data between kernel address space and user address space as discussed in Chapter 4. 13Or, more accurately, when it automatically frees its memory management data structures using mm_release at process termination. 14The condition mm->mm_users > 1 means that the memory management data structure must be used by at least one other process in the system. The current process is therefore a thread in the classical sense — it takes its address space from another process and has just one control flow. 76 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 77 Chapter 2: Process Management and Scheduling sys_futex(tidptr, FUTEX_WAKE, 1, NULL, NULL, 0); } ... } In addition, sys_futex, a fast userspace mutex, is used to wake processes waiting for this event, namely, the end of the thread. The above flags can be used from within userspace to check when threads are generated and destroyed in the kernel. CLONE_CHILD_SETTID and CLONE_PARENT_SETTID are used to check when a thread is gen- erated; CLONE_CHILD_CLEARTID is used to pass information on the death of a thread from the kernel to userspace. These checks can genuinely be performed in parallel on multiprocessor systems. 2.4.2 Kernel Threads Kernel threads are processes started directly by the kernel itself. They delegate a kernel function to a separate process and execute it there in ‘‘parallel‘‘ to the other processes in the system (and, in fact, in parallel to execution of the kernel itself).15 Kernel threads are often referred to as (kernel) daemons.They are used to perform, for example, the following tasks: ❑ To periodically synchronize modified memory pages with the block device from which the pages originate (e.g., files mapped using mmap). ❑ To write memory pages into the swap area if they are seldom used. ❑ To manage deferred actions. ❑ To implement transaction journals for filesystems. Basically, there are two types of kernel thread: ❑ Type 1 — The thread is started and waits until requested by the kernel to perform a specific action. ❑ Type 2 — Once started, the thread runs at periodic intervals, checks the utilization of a specific resource, and takes action when utilization exceeds or falls below a set limit value. The kernel uses this type of thread for continuous monitoring tasks. The kernel_thread function is invoked to start a kernel thread. Its definition is architecture-specific, but it always uses the same prototype. int kernel_thread(int (*fn)(void *), void * arg, unsigned long flags) The function passed with the fn pointer is executed in the generated thread, and the argument specified in arg is automatically passed to the function.16 CLONE flags can be specified in flags. The first task of kernel_thread is to construct a pt_regs instance in which the registers are supplied with suitable values, as would be the case with a regular fork system call. Then the familiar do_fork function is invoked. p = do_fork(flags | CLONE_VM | CLONE_UNTRACED, 0, ®s, 0, NULL, NULL); 15On multiprocessor systems, the processes genuinely execute in parallel; on single-processor systems, the scheduler simulates par- allel execution. 16Arguments allow the function to be used for different purposes by indicating what needs to be done. 77 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 78 Chapter 2: Process Management and Scheduling Because kernel threads are generated by the kernel itself, two special points should be noted: 1. They execute in the supervisor mode of the CPU, not in the user mode (see Chapter 1). 2. They may access only the kernel part of virtual address space (all addresses above TASK_SIZE) but not the virtual user area. Recall from above that the two pointers to mm_structs are contained in the task structure: struct task_struct { ... struct mm_struct *mm, *active_mm; ... } The total virtual address space of a system is separated into two parts on most machines: The lower portion is accessible by userland programs, and the upper part is reserved for the kernel. When the kernel is running on behalf of a userland program to serve a system call, for instance, the userspace portion of the virtual address space is described by the mm_struct instance pointed to by mm (the exact content of this structure is irrelevant for now, but is discussed in Chapter 4). Every time the kernel performs a context switch, the userland portion of the virtual address space must be replaced to match the then-running process. This provides some room for optimization, which goes by the name lazy TLB handling:Sincekernel threads are not associated with any particular userland process, the kernel does not need to rearrange the userland portion of the virtual address space and can just leave the old setting in place. Since any userland process can have been running before a kernel thread, the contents of the userspace part are essentially random, and the kernel thread must not modify it. To signalize that the userspace portion must not be accessed, mm is set to a NULL pointer. However, since the kernel must know what is currently contained in the userspace, a pointer to the mm_struct describing it is preserved in active_mm. Why are processes without an mm pointer called lazy TLB processes? Suppose that the process that runs after a kernel thread is the same process that has run before. In this case, the kernel does not need to modify the userspace address tables, and the information in the translation lookaside buffers is still valid. A switch (and a corresponding clearance of TLB data) is only required when a different userland process from before executes after the kernel thread. Notice that when the kernel is operating in process context, mm and active_mm have identical values. A kernel thread can be implemented in one of two ways. The older variant — which is still in use in some places in the kernel — is to pass a function directly to kernel_thread. The function is then responsible to assist the kernel in the transformation into a daemon by invoking daemonize. This results in the following actions: 1. The function frees all resources (e.g., memory context, file descriptors, etc.) of the user pro- cess as whose child the kernel thread was started because otherwise these would be pinned until the end of the thread — this is not desirable because daemons usually run until the sys- tem is shut down. As each daemon operates only in the address area of the kernel, it does not even need these resources. 78 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 79 Chapter 2: Process Management and Scheduling 2. daemonize blocks the receipt of signals. 3. init is used as the parent process of the daemon. The more modern possibility to create a kernel thread is the auxiliary function kthread_create. kernel/kthread.c struct task_struct *kthread_create(int (*threadfn)(void *data), void *data, const char namefmt[], ...) The function creates a new kernel thread with its name given by namefmt. Initially, the thread will be stopped. To start it, wake_up_process needs to be used. After this, the thread function given in threadfn will be called with data as argument. As an alternative, the macro kthread_run (which uses the same arguments as kthread_create) will call kthread_create to create the new thread, but will wake it up immediately. A kernel thread can also be bound to a particular CPU by using kthread_create_cpu instead of kthread_create. Kernel threads appear in the system process list but are enclosed in square brackets in the output of ps to differentiate them from normal processes. wolfgang@meitner> ps fax PID TTY STAT TIME COMMAND 2? S< 0:00 [kthreadd] 3? S< 0:00 _ [migration/0] 4? S< 0:00 _ [ksoftirqd/0] 5? S< 0:00 _ [migration/1] 6? S< 0:00 _ [ksoftirqd/1] ... 52 ? S< 0:00 _ [kblockd/3] 55 ? S< 0:00 _ [kacpid] 56 ? S< 0:00 _ [kacpi_notify] ... If a kernel thread is bound to a particular CPU, the CPU’s number is noted after the slash. 2.4.3 Starting New Programs New programs are started by replacing an existing program with new code. Linux provides the execve system call for this purpose.17 Implementation of execve The entry point of the system call is the architecture-dependent sys_execve function. This function quickly delegates its work to the system-independent do_execve routine. kernel/exec.c int do_execve(char * filename, char __user *__user *argv, 17There are other exec variants with different names in the C standard library, but ultimately all are based on execve.Asinthe above sections, exec is often used to refer to any of these variants. 79 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 80 Chapter 2: Process Management and Scheduling char __user *__user *envp, struct pt_regs * regs) Not only the register set with the arguments and the name of the executable file (filename)butalso pointers to the arguments and the environment of the program are passed as in system programming.18 The notation is slightly clumsy because argv and envp are arrays of pointers, and both the pointer to the array itself as well as all pointers in the array are located in the userspace portion of the virtual address space. Recall from the Introduction that some precautions are required when userspace memory is accessed from the kernel, and that the __user annotations allow automated tools to check if everything is handled properly. Figure 2-11 shows the code flow diagram for do_execve. Copy environment and arguments Open executable file do_execve bprm_init mm_alloc init_new_context _ _bprm_mm_init prepare_binprm search_binary_handler Figure 2-11: Code flow diagram for do_execve. First, the file to be executed is opened; in other words — as described in Chapter 8 — the kernel finds the associated inode and generates a file descriptor that is used to address the file. bprm_init then handles several administrative tasks: mm_alloc generates a new instance of mm_struct to manage the process address space (see Chapter 4). init_new_context is an architecture-specific function that initializes the instance, and __bprm_mm_init sets up an initial stack. Various parameters of the new process (e.g., euid, egid, argument list, environment, filename, etc.) that are subsequently passed to other functions are, for the sake of simplicity, combined into a structure of type linux_binprm. prepare_binprm is used to supply a number of parent process values (above all, the effective UID and GID); the remaining data — the argument list — are then copied manually into the structure. Note that prepare_binprm also takes care of handling the SUID and SGID bits: 18argv includes all arguments passed to the program in the command line (for ls -l /usr/bin these are, e.g., -l and /usr/bin). The environment encompasses all environment variables defined at program execution time. In most shells, a list of these variables can be output using set. 80 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 81 Chapter 2: Process Management and Scheduling fs/exec.c int prepare_binprm(struct linux_binprm *bprm) { ... bprm->e_uid = current->euid; bprm->e_gid = current->egid; if(!(bprm->file->f_vfsmnt->mnt_flags & MNT_NOSUID)) { /* Set-uid? */ if (mode & S_ISUID) { bprm->e_uid = inode->i_uid; } /* Set-gid? */ /* * If setgid is set but no group execute bit then this * is a candidate for mandatory locking, not a setgid * executable. */ if ((mode & (S_ISGID | S_IXGRP)) == (S_ISGID | S_IXGRP)) { bprm->e_gid = inode->i_gid; } } ... } After making sure that MNT_NOSUID is not set for the mount from which the file originates, the kernel checks if the SUID or SGID bit is set. The first case is simple to handle: If S_ISUID is set, then the effective UID gets the same value as the inode (otherwise, the process’s effective UID is used). The SGID case is similar, but the kernel must additionally make sure that the execute bit is also set for the group. Linux supports various organization formats for executable files. The standard format is ELF (Executable and Linkable Format), which I discuss at length in Appendix E. Other alternatives are the variants shown in Table 2-2 (which lists the names of the corresponding linux_binfmt instances in the kernel). Even though many binary formats can be used on different architectures (ELF was designed explicitly to be as system-independent as possible), this does not mean that programs in a specific binary format are able to run on multiple architectures. The assembler statements used still differ greatly from processor to processor and the binary format only indicates how the different parts of a program — data, code, and so on — are organized in the executable file and in memory. search_binary_handler is used at the end of do_execve to find a suitable binary format for the particular file. Searching is possible because each format can be recognized by reference to special characteristics (usually a ‘‘magic number‘‘ at the beginning of the file). The binary format handler is responsible for loading the data of the new program into the old address space. Appendix E describes the steps needed to do this when the ELF format is used. Generally, a binary format handler performs the following actions: ❑ It releases all resources used by the old process. ❑ It maps the application into virtual address space. The following segments must be taken into account (the variables specified are elements of the task structure and are set to the correct values by the binary format handler): 81 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 82 Chapter 2: Process Management and Scheduling ❑ The text segment contains the executable code of the program. start_code and end_code specify the area in address space where the segment resides. ❑ The pre-initialized data (variables supplied with a specific value at compilation time) are located between start_data and end_data and are mapped from the corresponding seg- ment of the executable file. ❑ The heap used for dynamic memory allocation is placed in virtual address space; start_brk and brk specify its boundaries. ❑ The position of the stack is defined by start_stack; the stack grows downward automati- cally on nearly all machines. The only exception is currently PA-Risc. The inverse direction of stack growth must be noted by the architecture by setting the configuration symbol STACK_GROWSUP. ❑ The program arguments and the environment are mapped into the virtual address space and are located between arg_start and arg_end and env_start and env_end,respec- tively. ❑ The instruction pointer of the process and some other architecture-specific registers are set so that the main function of the program is executed when the scheduler selects the process. How the ELF format populates the virtual address space will be discussed in more detail in Section 4.2.1. Table 2-2: Binary Formats Supported by Linux. Name Meaning flat_format The flat format is used on embedded CPUs without a memory manage- ment unit (MMU). To save space, the data in the executable can also be compressed (if zlib support is available in the kernel). script_format This is a dummy format used to run scripts using the she-bang mechanism. By looking at the first line of the file, the kernel knows which interpreter to use and starts the appropriate application (e.g., Perl for #! /usr/bin/perl). misc_format This is also a dummy format used to start applications requiring an external interpreter. In contrast to the #! mechanism, the interpreter need not be specified explicitly but is determined by reference to special file identifiers (suffix, header, etc.). This format is used, for example, to execute Java byte code or to run Windows programs with Wine. elf_format This is a machine- and architecture-independent format for 32 and 64 bits. It is the standard format under Linux. elf_fdpic_format ELF format with special features for systems without an MMU. irix_format ELF format with Irix-specific features. som_format HP-UX-specific format used on PA-Risc machines. aout_format a.out is the former standard format for Linux used before ELF was intro- duced. It is rarely used today because it is too inflexible. 82 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 83 Chapter 2: Process Management and Scheduling Interpreting Binary Formats Each binary format is represented in the Linux kernel by an instance of the following (simplified) data structure: struct linux_binfmt { struct linux_binfmt * next; struct module *module; int (*load_binary)(struct linux_binprm *, struct pt_regs * regs); int (*load_shlib)(struct file *); int (*core_dump)(long signr, struct pt_regs * regs, struct file * file); unsigned long min_coredump; /* minimal dump size */ }; Each binary format must provide three functions: 1. load_binary to load normal programs. 2. load_shlib to load a shared library, that is, a dynamic library. 3. core_dump to write a core dump if there is a program error. This dump can subsequently be analyzed using a debugger (e.g., gdb) for troubleshooting purposes. min_coredump is a lower bound on the core file size from which a coredump will be generated (usually, this is the size of a single memory page). Each binary format must first be registered in the kernel using register_binfmt. The purpose of this function is to add a new binary format to a linked list whose list head is represented by the formats global variable from fs/exec.c.Thelinux_binfmt instances are linked with each other by means of their next element. 2.4.4 Exiting Processes Processes must terminate with the exit system call. This gives the kernel the opportunity to free the resources used by the processes to the system.19 The entry point for this call is the sys_exit function that requires an error code as its parameter in order to exit the process. Its definition is architecture- independent and is held in kernel/exit.c. Its implementation is not particularly interesting because it immediately delegates its work to do_exit. Suffice it to say that the implementation of this function consists essentially of decrementing reference counters and returning memory areas to memory management once the reference counter has reverted to 0 and the corresponding structure is no longer being used by any process in the system. 2.5 Implementation of the Scheduler A unique description of each process is held in memory and is linked with other processes by means of several structures. This is the situation facing the scheduler, whose task is to share CPU time between the programs to create the illusion of concurrent execution. As discussed above, this task is split into two different parts — one relating to the scheduling policy and the other to context switching. 19exit can be called explicitly by the programmer. However, the compiler automatically adds a corresponding call to the end of the main function (or to the main function used by the particular language). 83 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 84 Chapter 2: Process Management and Scheduling 2.5.1 Overview The kernel must provide a method of sharing CPU time as fairly as possible between the individual processes while at the same time taking into account differing task priorities. There are many ways of doing this, and all have their pros and cons, which we need not discuss here (see [Tan07] for an overview of potential approaches). Our focus is on the solution adopted in the Linux kernel. The schedule function is the starting point to an understanding of scheduling operations. It is defined in kernel/sched.c and is one of the most frequently invoked functions in the kernel code. The implemen- tation of the scheduler is obscured a little by several factors: ❑ On multiprocessor systems, several details (some very subtle) must be noted so that the sched- uler doesn’t get under its own feet. ❑ Not only priority scheduling but also two other soft real-time policies required by the Posix stan- dard are implemented. ❑ gotos are used to generate optimal assembly language code. These jump backward and forward in the C code and run counter to all principles of structured programming. However, this feature can be beneficial if it is used with great care, and the scheduler is one example where gotosmake sense. In the following overview, I consider the completely fair scheduler and neglect real-time tasks for now. I come back to them later. An outstanding feature of the Linux scheduler is that it does not require the concept of time slices, at least not in the traditional way. Classical schedulers compute time slices for each process in the system and allow them to run until their time slice is used up. When all time slices of all processes have been used up, they need to be recalculated again. The current scheduler, in contrast, considers only the wait time of a process — that is, how long it has been sitting around in the run-queue and was ready to be executed. The task with the gravest need for CPU time is scheduled. The general principle of the scheduler is to provide maximum fairness to each task in the system in terms of the computational power it is given. Or, put differently, it tries to ensure that no task is treated unfairly. Now this clearly sounds good, but what do fair and unfair with respect to CPU time mean? Consider an ideal computer that can run an arbitrary number of tasks in parallel: If N processes are present on the system, then each one gets 1 N of the total computational power, and all tasks really execute physically parallel. Suppose that a task requires 10 minutes to complete its work. If 5 such tasks are simultaneously present on a perfect CPU, each will get 20 percent of the computational power, which means that it will be running for 50 instead of 10 minutes. However, all 5 tasks will finish their job after exactly this time span, and none of them will have ever been inactive! This is clearly not achievable on real hardware: If a system has only a single CPU, at most one process can be run simultaneously. Multitasking is only achieved by switching back and forth between the tasks with high frequency. For users, who think considerably more slowly than the switching frequency, this creates the illusion of parallel executing, but in reality, it is not. While more CPUs in the system improve the situation and allow perfect parallel execution of a small number of tasks, there will always be situations in which fewer CPUs than processes that are to be run are available, and the problem starts anew. If multitasking is simulated by running one process after another, then the process that is currently running is favored over those waiting to be picked by the scheduler — the poor waiting processes are being treated unfairly. The unfairness is directly proportional to the waiting time. 84 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 85 Chapter 2: Process Management and Scheduling Every time the scheduler is called, it picks the task with the highest waiting time and gives the CPU to it. If this happens often enough, no large unfairness will accumulate for tasks, and the unfairness will be evenly distributed among all tasks in the system. Figure 2-12 illustrates how the scheduler keeps track of which process has been waiting for how long. Since runnable processes are queued, the structure is known as the run queue. Real clock Virtual clock task picked to run Time ordered Red-black tree Run queue decreasing wait time CPU Figure 2-12: The scheduler keeps track of the waiting time of the available processes by sorting them in a red-black tree. All runnable tasks are time-ordered in a red-black tree, essentially with respect to their waiting time. The task that has been waiting for the CPU for the largest amount of time is the leftmost entry and will be considered next by the scheduler. Tasks that have been waiting less long are sorted on the tree from left to right. If you are not familiar with red-black trees, suffice it to know here that this data structure allows for efficient management of the entries it contains, and that the time required for lookup, insertion, and dele- tion operations will only moderately rise with the number of processes present in the tree.20 Red-black trees are available as a standard data structure of the kernel, and Appendix C provides more information about them. Besides, a discussion of such trees can be found in every textbook on data structures. Besides the red-black tree, a run queue is also equipped with a virtual clock.21 Time passes slower on this clock than in real time, and the exact speed depends on the number of processes that are currently waiting to be picked by the scheduler. Suppose that four processes are on the queue: Then the virtual clock will run at one-quarter of the speed of a real clock. This is the basis to determine how much CPU time a waiting process would have gotten if computational power could be shared in a completely fair manner. Sitting on the run queue for 20 seconds in real time amounts to 5 seconds in virtual time. Four tasks executing for 5 seconds each would keep the CPU occupied for 20 seconds in real time. 20To be precise: Time complexity is O(logn),wheren is the number of elements in the tree. This is worse than for the old scheduler, which was famous for being an O(1) scheduler, that is, its run time was independent of the number of processes it had to deal with. However, the slow-down caused by the linear-logarithmic dependency of the new scheduler is negligible unless a huge number of processes is simultaneously runnable. In practice, such a situation does not occur. 21Notice that the kernel really used the concept of a virtual clock for the scheduling mechanism in kernel 2.6.23, but currently com- putes the virtual time a little differently. Since the method is easier to understand with virtual clocks, I will stick to this now and discuss how the virtual clock is emulated when I discuss the scheduler implementation. 85 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 86 Chapter 2: Process Management and Scheduling Suppose that the virtual time of the run queue is given by fair_clock, while the waiting time of a process is stored in wait_runtime. To sort tasks on the red-black tree, the kernel uses the difference fair_clock - wait_runtime. While fair_clock is a measure for the CPU time a task would have gotten if scheduling were completely fair, wait_runtime is a direct measure for the unfairness caused by the imperfection of real systems. When a task is allowed to run, the interval during which it has been running is subtracted from wait_runtime. This way, it will move rightward in the time-ordered tree at some point, and another process will be the leftmost one — and is consequently selected to run. Notice, however, that the virtual clock in fair_clock will increase when the task is running. This effectively means that the share of CPU time that the task would have received in a perfectly fair system is deducted from the time spent executing on the real CPU. This slows degradation of unfairness: Decrementing wait_runtime is equivalent to lowering the amount of unfairness received by the task, but the kernel must not forget that some portion of the time used to lower the unfairness would have belonged to the process in a completely fair world anyway. Suppose again that four processes sit on the run queue, and that a process has been waiting for 20 real seconds. Now it is allowed to run for 10 seconds: wait_runtime is afterward 10, but since the process would have gotten 10/4 = 2 seconds of this time span anyway, effectively only 8 time units account for the potentially new position on the run queue. Unfortunately, this strategy is complicated by a number of real-world issues: ❑ Different priority levels for tasks (i.e., nice values) must be taken into account, and more impor- tant processes must get a higher share of CPU time than less important ones. ❑ Tasks must not be switched too often because a context switch, that is, changing from one task to another, has a certain overhead. When switching happens too often, too much time is spent with exchanging tasks that is not available for effective work anymore. On the other hand, the time that goes by between task switches must not be too long because large unfairness values could accumulate in this case. Letting tasks run for too long can also lead to larger latencies than desired for multimedia systems. We will see how the scheduler tackles these problems in the following discussion. A good way to understand scheduling decisions is to activate scheduler statistics at compile time. This will generate the file /proc/sched_debug, which contains information on all aspects of the current state of the scheduler. Finally, note that the Documentation/ directory contains some files that relate to various aspects of the scheduler. Keep in mind, however, that some of them still relate to the old O(1) scheduler and are there- fore outdated! 2.5.2 Data Structures The scheduler uses a series of data structures to sort and manage the processes in the system. How the scheduler works is closely linked with the design of these structures. Several components interact with each other in many ways, and Figure 2-13 provides a first overview of the connections. Scheduling can be activated in two ways: either directly if a task goes to sleep or wants to yield the CPU for other reasons, or by a periodic mechanism that is run with constant frequency and that checks from time to time if switching tasks is necessary. I denote these two components generic scheduler or core 86 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 87 Chapter 2: Process Management and Scheduling scheduler in the following. Essentially, the generic scheduler is a dispatcher that interacts with two other components: 1. Scheduling classes are used to decide which task runs next. The kernel supports different scheduling policies (completely fair scheduling, real-time scheduling, and scheduling of the idle task when there is nothing to do), and scheduling classes allow for implementing these policies in a modular way: Code from one class does not need to interact with code from other classes. When the scheduler is invoked, it queries the scheduler classes which task is supposed to run next. 2. After a task has been selected to run, a low-level task switch must be performed. This requires close interaction with the underlying CPU. Main scheduler Periodic scheduler CPU Select task switch Context Tasks Scheduler classes Figure 2-13: Overview of the components of the scheduling subsystem. Every task belongs to exactly one of the scheduling classes, and each scheduling class is responsible to manage their tasks. The generic scheduler itself is not involved in managing tasks at all; this is completely delegated to the scheduler classes. Elements in the Task Structure There are several scheduling-relevant elements in the task structure of each process. struct task_struct { ... int prio, static_prio, normal_prio; unsigned int rt_priority; struct list_head run_list; const struct sched_class *sched_class; struct sched_entity se; unsigned int policy; cpumask_t cpus_allowed; unsigned int time_slice; ... } 87 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 88 Chapter 2: Process Management and Scheduling ❑ Not all processes on a system are equally important: Less urgent tasks should receive less atten- tion, while important work should be done as quickly as possible. To determine the importance of a particular task, it is equipped with a relative priority. However, the task structure employs three elements to denote the priority of a process: prio and normal_prio indicate the dynamic priorities, static_prio the static priority of a process. The static priority is the priority assigned to the process when it was started. It can be modified with the nice and sched_setscheduler system calls, but remains otherwise constant during the process’ run time. normal_priority denotes a priority that is computed based on the static priority and the scheduling policy of the process. Identical static priorities will therefore result in different normal priorities depending on whether a process is a regular or a real-time process. When a process forks, the child process will inherit the normal priority. However, the priority considered by the scheduler is kept in prio. A third element is required because situations can arise in which the kernel needs to temporarily boost the priority of a pro- cess. Since these changes are not permanent, the static and normal priorities are unaffected by this. How the three priorities depend on each other is slightly subtle, and I discuss this in detail below. ❑ rt_priority denotes the priority of a real-time process. Note that this does not replace the pre- viously discussed values! The lowest real-time priority has value 0, whereas the highest priority is 99. Higher values correspond to higher priorities. The convention used here is different from the convention used for nice values. ❑ sched_class denotes the scheduler class the process is in. ❑ The scheduler is not limited to schedule processes, but can also work with larger entities. This allows for implementing group scheduling: This way, the available CPU time can first be dis- tributed between general process groups (e.g., all processes can be grouped according to their owner), and the assigned time is then again distributed within the group. This generality requires that the scheduler does not directly operate on processes but works with schedulable entities. An entity is represented by an instance of sched_entity. In the simplest case, scheduling is performed on a per-process level, and this is the case we con- centrate on initially. Since the scheduler is designed to work on schedulable entities, each process must look to it like such an entity. se therefore embeds an instance of sched_entity on which the scheduler operates in each task struct (notice that se is not a pointer because the entity is embedded in the task!). ❑ policy holds the scheduling policy applied to the process. Linux supports five possible values: ❑ SCHED_NORMAL is used for normal processes on which our description focuses. They are handled by the completely fair scheduler. SCHED_BATCH and SCHED_IDLE are also handled by the completely fair scheduler but can be used for less important tasks. SCHED_BATCH is for CPU-intensive batch processes that are not interactive. Tasks of this type are disfa- vored in scheduling decisions: They will never preempt another process handled by the CF scheduler and will therefore not disturb interactive tasks. The class is well suited for situations in which the static priority of a task is not desired to be decreased with nice,but when the task should nevertheless not influence the interactivity of a system. SCHED_IDLE tasks will also be of low importance in the scheduling decisions, but this time because their relative weight is always minimal (this will become clear when I discuss how the kernel computes task weights that reflect their priority). 88 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 89 Chapter 2: Process Management and Scheduling Note that SCHED_IDLE is, despite its name, not responsible to schedule the idle task. The kernel provides a separate mechanism for this purpose. ❑ SCHED_RR and SCHED_FIFO are used to implement soft real-time processes. SCHED_RR imple- ments a round robin method, while SCHED_FIFO uses a first in, first out mechanism. These are not handled by the completely fair scheduler class, but by the real-time scheduler class, which is discussed in Section 2.7 in greater length. The auxiliary function rt_policy is used to decide if a given scheduling policy belongs to the real-time class (SCHED_RR and SCHED_FIFO)ornot.task_has_rt_policy determines this prop- erty for a given task. kernel/sched.c static inline int rt_policy(int policy) static inline int task_has_rt_policy(struct task_struct *p) ❑ cpus_allowed is a bit field used on multiprocessor systems to restrict the CPUs on which a pro- cess may run.22 ❑ run_list and time_slice are required for the round-robin real-time scheduler, but not for the completely fair scheduler. run_list is a list head used to hold the process on a run list, while time_slice specifies the remaining time quantum during which the process may use the CPU. The TIF_NEED_RESCHED flag discussed above is just as important for the scheduler as the specific sched- uler elements held in the task structure. If this flag is set for an active process, the scheduler knows that the CPU is to be withdrawn from the process — either voluntarily or by force — and granted to a new process. Scheduler Classes Scheduler classes provide the connection between the generic scheduler and individual scheduling methods. They are represented by several function pointers collected in a special data structure. Each operation that can be requested by the global scheduler is represented by one pointer. This allows for cre- ation of the generic scheduler without any knowledge about the internal working of different scheduler classes. Without extensions required for multiprocessor systems (I will come back to these later), the structure looksasfollows: struct sched_class { const struct sched_class *next; void (*enqueue_task) (struct rq *rq, struct task_struct *p, int wakeup); void (*dequeue_task) (struct rq *rq, struct task_struct *p, int sleep); void (*yield_task) (struct rq *rq); void (*check_preempt_curr) (struct rq *rq, struct task_struct *p); struct task_struct * (*pick_next_task) (struct rq *rq); void (*put_prev_task) (struct rq *rq, struct task_struct *p); 22The bitmap can be set using the sched_setaffinity system call. 89 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 90 Chapter 2: Process Management and Scheduling void (*set_curr_task) (struct rq *rq); void (*task_tick) (struct rq *rq, struct task_struct *p); void (*task_new) (struct rq *rq, struct task_struct *p); }; An instance of struct sched_class must be provided for each scheduling class. Scheduling classes are related in a flat hierarchy: Real-time processes are most important, so they are handled before completely fair processes, which are, in turn, given preference to the idle tasks that are active on a CPU when there is nothing better to do. The next element connects the sched_class instances of the different scheduling classes in the described order. Note that this hierarchy is already set up at compile time: There is no mechanism to add new scheduler classes dynamically at run time. The operations that can be provided by each scheduling class are as follows: ❑ enqueue_task adds a new process to the run queue. This happens when a process changes from a sleeping into a runnable state. ❑ dequeue_task provides the inverse operation: It takes a process off a run queue. Naturally, this happens when a process switches from a runnable into an un-runnable state, or when the kernel decides to take it off the run queue for other reasons — for instance, because its priority needs to be changed. Although the term run queue is used, the individual scheduling classes need not represent their processes on a simple queue. In fact, recall from above that the completely fair scheduler uses a red-black tree for this purpose. ❑ When a process wants to relinquish control of the processor voluntarily, it can use the sched_yield system call. This triggers yield_task to be called in the kernel. ❑ check_preempt_curr is used to preempt the current task with a newly woken task if this is necessary. The function is called, for instance, when a new task is woken up with wake_up_new_task. ❑ pick_next_task selects the next task that is supposed to run, while put_prev_task is called before the currently executing task is replaced with another one. Note that these operations are not equivalent to putting tasks on and off the run queue like enqueue_task and dequeue_task. Instead, they are responsible to give the CPU to a task, respectively, take it away. Switching between different tasks, however, still requires performing a low-level context switch. ❑ set_curr_task is called when the scheduling policy of a task is changed. There are also some other places that call the function, but they are not relevant for our purposes. ❑ task_tick is called by the periodic scheduler each time it is activated. ❑ new_task allows for setting up a connection between the fork system call and the scheduler. Each time a new task is created, the scheduler is notified about this with new_task. The standard functions activate_task and deactivate_task are provided to enqueue and dequeue a task by calling the aforementioned functions. Additionally, they keep the kernel statistics up to date. kernel/sched.c static void enqueue_task(struct rq *rq, struct task_struct *p, int wakeup) static void dequeue_task(struct rq *rq, struct task_struct *p, int sleep) When a process is registered on a run queue, the on_rq element of the embedded sched_entity instance is set to 1, otherwise to 0. 90 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 91 Chapter 2: Process Management and Scheduling Besides these, the kernel defines the convenience method check_preempt_curr to call the check_preempt_curr method of the scheduling class that is associated with a given task: kernel/sched.c static inline void check_preempt_curr(struct rq *rq, struct task_struct *p) Userland applications do not directly interact with scheduling classes. They only know of the constants SCHED_xyz as defined above. It is the kernel’s job to provide an appropriate mapping between these con- stants and the available scheduling classes. SCHED_NORMAL, SCHED_BATCH,andSCHED_IDLE are mapped to fair_sched_class, while SCHED_RR and SCHED_FIFO are associated with rt_sched_class.Both fair_sched_class and rt_sched_class are instances of struct sched_class that represent, respec- tively, the completely fair and the realtime scheduler. The contents of these instances will be shown when I discuss the respective scheduler classes in detail. Run Queues The central data structure of the core scheduler that is used to manage active processes is known as the run queue. Each CPU has its own run queue, and each active process appears on just one run queue. It is not possible to run a process on several CPUs at the same time.23 The run queue is the starting point for many actions of the global scheduler. Note, however, that pro- cesses are not directly managed by the general elements of the run queue! This is the responsibility of the individual scheduler classes, and a class-specific sub-run queue is therefore embedded in each run queue.24 Run queues are implemented using the following data structure. To simplify matters, I have omitted several statistical elements that do not directly influence the work of the run queue, and also the elements required on multiprocessor systems. kernel/sched.c struct rq { unsigned long nr_running; #define CPU_LOAD_IDX_MAX 5 unsigned long cpu_load[CPU_LOAD_IDX_MAX]; ... struct load_weight load; struct cfs_rq cfs; struct rt_rq rt; struct task_struct *curr, *idle; u64 clock; ... }; ❑ nr_running specifies the number of runnable processes on the queue — regardless of their pri- ority or scheduling class. 23However, threads originating from the same process can execute on different processors as task management makes no important distinction between processes and threads. 24For readers familiar with earlier versions of the kernel, it might be interesting to know the scheduler class run queues replace the lists of active and expired tasks that were utilized by the previous O(1) scheduler. 91 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 92 Chapter 2: Process Management and Scheduling ❑ load provides a measure for the current load on the run queue. The queue load is essentially proportional to the number of currently active processes on the queue, where each process is additionally weighted by its priority. The speed of the virtual per-run queue clock is based on this information. Since computing the load and other related quantities is an important com- ponent of the scheduling algorithm, I devote Section 2.5.3 below to a detailed discussion of the mechanisms involved. ❑ cpu_load allows for tracking the load behavior back into the past. ❑ cfs and rt are the embedded sub-run queues for the completely fair and real-time scheduler, respectively. ❑ curr points to the task structure of the process currently running. ❑ idle points to the task structure of the idle process called when no other runnable process is available — the idle thread. ❑ clock and prev_raw_clock are used to implement the per-run queue clock. The value of clock is updated each time the periodic scheduler is called. Additionally, the kernel provides the stan- dard function update_rq_clock that is called from many places in the scheduler that manipulate the run queue, for instance, when a new task is woken up in wakeup_new_task. All run queues of the system are held in the runqueues array, which contains an element for each CPU in the system. On single-processor systems, there is, of course, just one element because only one run queue is required. kernel/sched.c static DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); The kernel also defines a number of convenient macros, which are self-explanatory. kernel/sched.c #define cpu_rq(cpu) (&per_cpu(runqueues, (cpu))) #define this_rq() (&__get_cpu_var(runqueues)) #define task_rq(p) cpu_rq(task_cpu(p)) #define cpu_curr(cpu) (cpu_rq(cpu)->curr) Scheduling Entities Since the scheduler can operate with more general entities than tasks, an appropriate data structure is required to describe such an entity. It is defined as follows: struct sched_entity { struct load_weight load; /* for load-balancing */ struct rb_node run_node; unsigned int on_rq; u64 exec_start; u64 sum_exec_runtime; u64 vruntime; u64 prev_sum_exec_runtime; ... } 92 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 93 Chapter 2: Process Management and Scheduling The structure can contain many more statistical elements if support for scheduler statistics has been compiled into the kernel, and also has some more elements if group scheduling is enabled. The part that is interesting for us right now, however, boils down to what you see above. The meaning of the individual elements is as follows: ❑ load specifies a weight for each entity that contributes to the total load of the queue. Comput- ing the load weight is an important task of the scheduler because the speed of the virtual clock required for CFS will ultimately depend on it, so I discuss the method in detail in Section 2.5.3. ❑ run_node is a standard tree element that allows the entity to be sorted on a red-black tree. ❑ on_rq denotes whether the entity is currently scheduled on a run queue or not. ❑ When a process is running, the consumed CPU time needs to be recorded for the completely fair scheduler. sum_exec_runtime is used for this purpose. Tracking the run time is done cumulatively, in update_curr. The function is called from numerous places in the scheduler, for instance, when a new task is enqueued, or from the periodic tick. At each invocation, the difference between the current time and exec_start is computed, and exec_start is updated to the current time. The difference interval is added to sum_exec_runtime. The amount of time that has elapsed on the virtual clock during process execution is accounted in vruntime. ❑ When a process is taken off the CPU, its current sum_exec_runtime value is preserved in prev_exec_runtime. The data will later be required in the context of process preemption. Notice, however, that preserving the value of sum_exec_runtime in prev_exec_runtime does not mean that sum_exec_runtime is reset! The old value is kept, and sum_exec_runtime continues to grow monotonically. Since each task_struct has an instance of sched_entity embedded, a task is a schedulable entity. Notice, however, that the inverse statement is not true in general: A schedulable entity need not nec- essarily be a task. However in the following we are concerned only with task scheduling, so for now we can equate scheduling entities and tasks. Keep in mind that this is not true in general, though! 2.5.3 Dealing with Priorities Priorities are deceptively simple from the userspace point of view: After all, they seem to be just a range of numbers. The in-kernel reality is unfortunately somewhat different, and comparatively much effort is required to work with priorities. Kernel Representation of Priorities The static priority of a process can be set in userspace by means of the nice command, which internally invokes the nice system call.25 Thenicevalueofaprocessisbetween−20 and +19 (inclusive). Lower values mean higher priorities. Why this strange range was chosen is shrouded in history. The kernel uses a simpler scale ranging from 0 to 139 inclusive to represent priorities internally. Again, lower values mean higher priorities. The range from 0 to 99 is reserved for real-time processes. The nice values [−20, +19] are mapped to the range from 100 to 139, as shown in Figure 2-14. Real-time processes thus always have a higher priority than normal processes can ever have. 25setpriority is an alternative system call for setting process priority. It is able to modify not only the priority of an individual thread but also the priorities of all threads in a thread group or of all processes of a specific user, selected by means of the UID. 93 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 94 Chapter 2: Process Management and Scheduling 0 99 100 139 Higher Priority Realtime Normal Nice +19−20 Figure 2-14: Kernel priority scale. The following macros are used to convert between the different forms of representation (MAX_RT_PRIO specifies the maximum priority of real-time processes, and MAX_PRIO is the maximal priority value for regular processes): #define MAX_USER_RT_PRIO 100 #define MAX_RT_PRIO MAX_USER_RT_PRIO #define MAX_PRIO (MAX_RT_PRIO + 40) #define DEFAULT_PRIO (MAX_RT_PRIO + 20) kernel/sched.c #define NICE_TO_PRIO(nice) (MAX_RT_PRIO + (nice) + 20) #define PRIO_TO_NICE(prio) ((prio) - MAX_RT_PRIO - 20) #define TASK_NICE(p) PRIO_TO_NICE((p)->static_prio) Computing Priorities Recall that it is not sufficient to consider just the static priority of a process, but that three priorities must be taken into account: dynamic priority (task_struct->prio), normal priority (task_struct->normal_prio), and static priority (task_struct->static_prio). These priorities are related to each other in interesting ways, and in the following I discuss how. static_prio is the starting point of the calculations. Assume that it has been already set and that the kernel now wants to compute the other priorities. This is done by a one-liner: p->prio = effective_prio(p); The auxiliary function effective_prio performs the following operations: kernel/sched.c static int effective_prio(struct task_struct *p) { p->normal_prio = normal_prio(p); /* * If we are RT tasks or we were boosted to RT priority, * keep the priority unchanged. Otherwise, update priority * to the normal priority: */ if (!rt_prio(p->prio)) return p->normal_prio; return p->prio; } First of all, the normal priority is computed and stored in normal_priority. This side effect allows for setting both prio and normal_prio with a single function invocation. Another helper function, rt_prio, 94 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 95 Chapter 2: Process Management and Scheduling checks if the normal priority is in the real-time range, that is, smaller than RT_RT_PRIO. Notice that the check is not related to any scheduling class, but only to the numerical value of the priority. Assume for now that we are dealing with a regular process that is not subjected to real-time scheduling. In this case, normal_prio just returns the static priority. The effect is then simple: All three priority values have the same value, namely, that of the static priority! Things are different for real-time tasks, however. Observe how the normal priority is computed: kernel/sched.c static inline int normal_prio(struct task_struct *p) { int prio; if (task_has_rt_policy(p)) prio = MAX_RT_PRIO-1 - p->rt_priority; else prio = __normal_prio(p); return prio; } The normal priority needs to be computed differently for regular tasks and real-time tasks. The compu- tation performed in __normal_prio is only valid for a regular task. Real-time tasks, instead, compute the normal priority based on their rt_priority setting. Because higher values of rt_priority denote higher real-time priorities, this runs counter to the kernel-internal representation of priorities, where lower val- ues mean higher priorities. The proper in-kernel priority value is therefore given by MAX_RT_PRIO-1 - p->rt_priority. Notice that this time, the detection of a real-time task is, in contrast to effective_prio, not based on any priority, but on the scheduling policy set in the task_struct. What does __normal_priority do? The function is really simple; it just returns the static priority: kernel/sched.c static inline int __normal_prio(struct task_struct *p) { return p->static_prio; } Now one can certainly wonder why an extra function is used for this purpose. There is a historical reason: Computing the normal priority in the old O(1) scheduler was a much trickier business. Interactive tasks had to be detected and their priority boosted, while non-interactive tasks had to be penalized to obtain good interactive behavior of the system. This required numerous heuristic calculations that either did the job well — or failed at it. The new scheduler, thankfully, does not require such magical calculations anymore. However, one question remains: Why does the kernel base the real-time check in effective_prio on the numerical value of the priority instead of using task_has_rt_policy? This is required for non-real-time tasks that have been temporarily boosted to a real-time priority, which can happen when RT-Mutexes are in use. 26 26Real-time mutexes allow for protection of dangerous parts of the kernel against concurrent access by multiple processors. However, a phenomenon called priority inversion, in which a process with lower priority executes even though a process with higher priority is waiting for the CPU, can occur. This can be solved by temporarily boosting the priority of processes. Refer to the discussion in Section 5.2.8 for more details about this problem. 95 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 96 Chapter 2: Process Management and Scheduling Finally, Table 2-3 summarizes the result of the calculations for different types of tasks. Table 2-3: Computing Priorities for Various Task Types. Task type / priority static_prio normal_prio prio Non-real-time task static_prio static_prio static_prio Priority-boosted non-real-time task static_prio static_prio prio as before Real-time task static_prio MAX_RT_PRIO-1-rt_priority prio as before p->prio is set with the method shown above when a newly created task is woken up with wake_up_new_task, and when the static priority was changed using the nice system call. Notice that when a process forks off a child, the current static priority will be inherited from the parent. The dynamic priority of the child, that is, task_struct->prio, is set to the normal priority of the parent. This ensures that priority boosts caused by RT-Mutexes are not transferred to the child process. Computing Load Weights The importance of a task is not only specified by its priority, but also by the load weight stored in task_struct->se.load. set_load_weight is responsible to compute the load weight depending on the process type and its static priority. The load weight is contained in the data structure load_weight: struct load_weight { unsigned long weight, inv_weight; }; The kernel not only keeps the load itself, but also another quantity that can be used to perform divisions by the weight.27 The general idea is that every process that changes the priority by one nice level down gets 10 percent more CPU power, while changing one nice level up gives 10 percent CPU power less. To enforce this policy, the kernel converts priorities to weight values. Let’s first see the table: kernel/sched.c static const int prio_to_weight[40] = { /* -20 */ 88761, 71755, 56483, 46273, 36291, /* -15 */ 29154, 23254, 18705, 14949, 11916, /* -10 */ 9548, 7620, 6100, 4904, 3906, /* -5 */ 3121, 2501, 1991, 1586, 1277, /* 0 */ 1024, 820, 655, 526, 423, 27Since a normal long is used, the kernel cannot directly store 1/weight, but has to resort to a technique that allows for performing the division with a multiplication and bit shifting. The details are not of interest here, however. 96 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 97 Chapter 2: Process Management and Scheduling /* 5 */ 335, 272, 215, 172, 137, /* 10 */ 110, 87, 70, 56, 45, /* 15 */ 36, 29, 23, 18, 15, }; The array contains one entry for each nice level in the range [0, 39] as used by the kernel. The multiplier between the entries is 1.25. To see why this is required, consider the following example. Two processes A and B run at nice level 0, so each one gets the same share of the CPU, namely, 50 percent. The weight for a nice 0 task is 1,024 as can be deduced from the table. The share for each task is 1024 1024+1024 = 0.5, that is, 50 percent as expected. If task B is re-niced by one priority level, it is supposed to get 10 percent less CPU share. In other words, this means that A will get 55 percent and B will get 45 percent of the total CPU time. Increasing the priority by 1 leads to a decrease of its weight, which is then 1, 024/1.25 ≈ 820. The CPU share A will get now is therefore 1024 1024+820 ≈ 0.55, whereas B will have 820 1024+820 ≈ 0.45 — a 10 percent difference as required. 0 20000 40000 60000 80000 100000 120000 140000 160000 180000 80 90 100 110 120 130 140 Weight Priority Regular task Realtime task 0 200 400 600 800 1000 1200 120 125 130 135 140 10 100 1000 10000 100000 80 90 100 110 120 130 140 Figure 2-15: Relation between static priority and load for regular and real-time processes. The code that performs the conversion also needs to account for real-time tasks. These will get double of the weight of a normal task. SCHED_IDLE tasks, on the other hand, will always receive a very small weight: kernel/sched.c #define WEIGHT_IDLEPRIO 2 #define WMULT_IDLEPRIO (1 << 31) 97 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 98 Chapter 2: Process Management and Scheduling static void set_load_weight(struct task_struct *p) { if (task_has_rt_policy(p)) { p->se.load.weight = prio_to_weight[0] * 2; p->se.load.inv_weight = prio_to_wmult[0] >> 1; return; } /* * SCHED_IDLE tasks get minimal weight: */ if (p->policy == SCHED_IDLE) { p->se.load.weight = WEIGHT_IDLEPRIO; p->se.load.inv_weight = WMULT_IDLEPRIO; return; } p->se.load.weight = prio_to_weight[p->static_prio - MAX_RT_PRIO]; p->se.load.inv_weight = prio_to_wmult[p->static_prio - MAX_RT_PRIO]; } The kernel not only computes the weight itself, but also stores the value required for division. Notice that allowing 10 percent more CPU time per priority change leads to an exponential behavior of the weight (and the related CPU times), which is illustrated in Figure 2-15. The upper inset in the figure shows the graph for a restricted region of regular priorities. The lower inset employs a logarithmic scale on the y axis. Note that the function is discontinuous at the transition point from regular to real-time processes. Recall that not only processes, but also run queues are associated with a load weight. Every time a process is added to a run queue, the kernel calls inc_nr_running. This not only ensures that the run queue keeps track of how many processes are running, but also adds the process weight to the weight of the run queue: kernel/sched.c static inline void update_load_add(struct load_weight *lw, unsigned long inc) { lw->weight += inc; } static inline void inc_load(struct rq *rq, const struct task_struct *p) { update_load_add(&rq->load, p->se.load.weight); } static void inc_nr_running(struct task_struct *p, struct rq *rq) { rq->nr_running++; inc_load(rq, p); } Corresponding functions (dec_nr_running, dec_load,andupdate_load_sub) are called when a process is removed from the run queue. 98 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 99 Chapter 2: Process Management and Scheduling 2.5.4 Core Scheduler As mentioned above, scheduler implementation is based on two functions — the periodic scheduler and the main scheduler function. These distribute CPU time on the basis of the priorities of the available processes; this is why the overall method can also be referred to as priority scheduling — although this is a very general term, naturally. I discuss how priority scheduling is implemented in this section. The Periodic Scheduler The periodic scheduler is implemented in scheduler_tick. The function is automatically called by the kernel with the frequency HZ if system activity is going on. If no processes are waiting to be scheduled, the tick can also be turned off to save power on computers where this is a scarce resource, for instance, lap- tops or small embedded systems. The mechanism underlying periodic actions is discussed in Chapter 15. The function has two principal tasks. 1. To manage the kernel scheduling-specific statistics relating to the whole system and to the individual processes. The main actions performed involve incrementing counters and are of no particular interest to us. 2. To activate the periodic scheduling method of the scheduling class responsible for the cur- rent process. kernel/sched.c void scheduler_tick(void) { int cpu = smp_processor_id(); struct rq *rq = cpu_rq(cpu); struct task_struct *curr = rq->curr; ... __update_rq_clock(rq) update_cpu_load(rq); The first part of the function deals with updating the run queue clock. This is delegated to __update_rq_clock, which essentially advances the clock time stamp of the current instance of struct rq. The function has to deal with some oddities of hardware clocks, but these are not relevant for our purposes. update_cpu_load then deals with updating the cpu_load[] history array of the run queue. This essentially shifts the previously stored load values one array position ahead, and inserts the present run queue load into the first position. Additionally, the function introduces some averaging to ensure that the contents of the load array do not exhibit large discontinuous jumps. Thanks to the modular structure of the scheduler, the main work is really simple, as it can be completely delegated to the scheduler-class-specific method: kernel/sched.c if (curr != rq->idle) curr->sched_class->task_tick(rq, curr); } How task_tick is implemented depends on the underlying scheduler class. The completely fair sched- uler, for instance, will in this method check if a process has been running for too long to avoid large latencies, but I discuss this in detail below. Readers familiar with the old time-slice-based scheduling 99 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 100 Chapter 2: Process Management and Scheduling method should be aware, however, that this is not equivalent to an expiring time slice — they do not exist anymore in the completely fair scheduler. If the current task is supposed to be rescheduled, the scheduler class methods set the TIF_NEED_RESCHED flag in the task structure to express this request, and the kernel fulfills it at the next opportune moment. The Main Scheduler The main scheduler function (schedule) is invoked directly at many points in the kernel to allocate the CPU to a process other than the currently active one. After returning from system calls, the kernel also checks whether the reschedule flag TIF_NEED_RESCHED of the current process is set — for example, the flag is set by scheduler_tick as mentioned above. If it is, the kernel invokes schedule. The function then assumes that the currently active task is definitely to be replaced with another task. Before I discuss schedule in detail, I need to make one remark that concerns the __sched prefix. This is used for functions that can potentially call schedule, including the schedule function itself. The declara- tion looks as follows: void __sched some_function(...) { ... schedule(); ... } The purpose of the prefix is to put the compiled code of the function into a special section of the object file, namely, .sched.text (see Appendix C for more information on ELF sections). This information enables the kernel to ignore all scheduling-related calls when a stack dump or similar information needs to be shown. Since the scheduler function calls are not part of the regular code flow, they are of no interest in such cases. Let’s come back to the implementation of the main scheduler schedule. The function first determines the current run queue and saves a pointer to the task structure of the (still) active process in prev. kernel/sched.c asmlinkage void __sched schedule(void) { struct task_struct *prev, *next; struct rq *rq; int cpu; need_resched: cpu = smp_processor_id(); rq = cpu_rq(cpu); prev = rq->curr; ... As in the periodic scheduler, the kernel takes the opportunity to update the run queue clock and clears the reschedule flag TIF_NEED_RESCHED in the task structure of the currently running task. kernel/sched.c __update_rq_clock(rq); clear_tsk_need_resched(prev); ... 100 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 101 Chapter 2: Process Management and Scheduling Again thanks to the modular structure of the scheduler, most work can be delegated to the scheduling classes. If the current task was in an interruptible sleep but has received a signal now, it must be promoted to a running task again. Otherwise, the task is deactivated with the scheduler-class-specific methods (deactivate_task essentially ends up in calling sched_class->dequeue_task): kernel/sched.c if (unlikely((prev->state & TASK_INTERRUPTIBLE) && unlikely(signal_pending(prev)))) { prev->state = TASK_RUNNING; } else { deactivate_task(rq, prev, 1); } ... put_prev_task first announces to the scheduler class that the currently running task is going to be replaced by another one. Note that this is not equivalent to taking the task off the run queue, but provides the opportunity to perform some accounting and bring statistics up to date. The next task that is sup- posed to be executed must also be selected by the scheduling class, and pick_next_task is responsible to do so: prev->sched_class->put_prev_task(rq, prev); next = pick_next_task(rq, prev); ... It need not necessarily be the case that a new task has been selected. If only one task is currently able to run because all others are sleeping, it will naturally be left on the CPU. If, however, a new task has been selected, then task switching at the hardware level must be prepared and executed. kernel/sched.c if (likely(prev != next)) { rq->curr = next; context_switch(rq, prev, next); } ... context_switch is the interface to the architecture-specific methods that perform a low-level context switch. The following code checks if the reschedule bit of the current task is set, and the scheduler jumps to the label described above and the search for a new process recommences: kernel/sched.c if (unlikely(test_thread_flag(TIF_NEED_RESCHED))) goto need_resched; } Notice that the above piece of code is executed in two different contexts: When no context switch has been performed, it is run directly at the end of the schedule function. If, however, a context switch has been performed, the current process will stop running right before this point — the new task has taken over the CPU. However, when the previous task is reselected to run later on, it will resume its execution directly at this point. Since prev will not point to the proper process in this case, the current thread needs to be found via current by test_thread_flag. 101 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 102 Chapter 2: Process Management and Scheduling Interaction with fork Whenever a new process is created using the fork system call or one of its variants, the scheduler gets a chance to hook into the process with the sched_fork function. On a single-processor system, the function performs essentially three actions: Initialize the scheduling-related fields of the new process, set up data structures (this is rather straightforward), and determine the dynamic priority of the process: kernel/sched.c /* * fork()/clone()-time setup: */ void sched_fork(struct task_struct *p, int clone_flags) { /* Initialize data structures */ ... /* * Make sure we do not leak PI boosting priority to the child: */ p->prio = current->normal_prio; if (!rt_prio(p->prio)) p->sched_class = &fair_sched_class; ... } By using the normal priority of the parent process as the dynamic priority of the child, the kernel ensures that any temporary boosts of the parent’s priority are not inherited by the child. Recall that the dynamic priority of a process can be temporarily modified when RT-Mutexes are used. This effect must not be transferred to the child. If the priority is not in the real-time range, the process will always start out in the completely fair scheduling class. When a new task is woken up using wake_up_new_task, a second opportunity for the scheduler to inter- act with task creation presents itself: The kernel calls the task_new function of the scheduling class. This gives an opportunity to enqueue the new process into the run queue of the respective class. Context Switching Once the kernel has selected a new process, the technical details associated with multitasking must be dealt with; these details are known collectively as context switching. The auxiliary function context_switch is the dispatcher for the required architecture-specific methods. kernel/sched.c static inline void context_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next) { struct mm_struct *mm, *oldmm; prepare_task_switch(rq, prev, next); mm = next->mm; oldmm = prev->active_mm; .. 102 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 103 Chapter 2: Process Management and Scheduling Immediately before a task switch, the prepare_arch_switch hook that must be defined by every archi- tecture is called from prepare_task_switch. This enables the kernel to execute architecture-specific code to prepare for the switch. Most supported architectures (with the exception of Sparc64 and Sparc) do not use this option because it is not needed. The context switch proper is performed by invoking two processor-specific functions: 1. switch_mm changes the memory context described in task_struct->mm. Depending on the processor, this is done by loading the page tables, flushing the translation lookaside buffers (partially or fully), and supplying the MMU with new information. Because these actions go deep into CPU details, I do not intend to discuss their implementation here. 2. switch_to switches the processor register contents and the kernel stack (the virtual user address space is changed in the first step, and as it includes the user mode stack, it is not necessary to change the latter explicitly). This task also varies greatly from architecture to architecture and is usually coded entirely in assembly language. Again, I ignore implemen- tation details. Because the register contents of the userspace process are saved on the kernel stack when kernel mode is entered (see Chapter 14 for details), this need not be done explicitly during the context switch. And because each process first begins to execute in kernel mode (at that point during scheduling at which control is passed to the new process), the register contents are automatically restored using the values on the kernel stack when a return is made to userspace. Remember, however, that kernel threads do not have their own userspace memory context and exe- cute on top of the address space of a random task; their task_struct->mm is NULL. The address space ‘‘borrowed’’ from the current task is noted in active_mm instead: kernel/sched.c if (unlikely(!mm)) { next->active_mm = oldmm; atomic_inc(&oldmm->mm_count); enter_lazy_tlb(oldmm, next); } else switch_mm(oldmm, mm, next); ... enter_lazy_tlb notifies the underlying architecture that exchanging the userspace portion of the virtual address space is not required. This speeds up the context switch and is known as the lazy TLB technique. If the previous task was a kernel thread (i.e., prev->mm is NULL), its active_mm pointer must be reset to NULL to disconnect it from the borrowed address space: kernel/sched.c if (unlikely(!prev->mm)) { prev->active_mm = NULL; rq->prev_mm = oldmm; } ... 103 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 104 Chapter 2: Process Management and Scheduling Finally, the task switch is finished with switch_to, which switches the register state and the stack — the new process will be running after the call: kernel/sched.c /* Here we just switch the register state and the stack. */ switch_to(prev, next, prev); barrier(); /* * this_rq must be evaluated again because prev may have moved * CPUs since it called schedule(), thus the ’rq’ on its stack * frame will be invalid. */ finish_task_switch(this_rq(), prev); } Thecodefollowingafterswitch_to will only be executed when the current process is selected to run next time. finish_task_switch performs some cleanups and allows for correctly releasing locks, which, however, we will not discuss in detail. It also gives individual architectures another possibility to hook into the context switching process, but this is only required on a few machines. The barrier statement is a directive for the compiler that ensures that the order in which the switch_to and finish_task_switch statements are executed is not changed by any unfortunate optimizations (see Chapter 5 for more details). Intricacies of switch_to The interesting thing about finish_task_switch is that the cleanups are performed for the task that has been active before the running task has been selected for execution. Notice that this is not the task that has initiated the context switch, but some random other task in the system! The kernel must find a way to communicate this task to the context_switch routine, and this is achieved with the switch_to macro. It must be implemented by every architecture and has a very unusual calling convention: Two variables are handed over, but in three parameters! This is because not only two, but three processes are involved in a context switch. The situation is illustrated in Figure 2-16. Kernel mode stack next = B prev = A next = C prev = B next = A prev = C next = B prev = A prev = C Process ABCA A = switch_to(A,B) B = switch_to(B,C) C = switch_to(C,A) Before switch_to After switch_to returns Figure 2-16: Behavior of the prev and next variables during context switches. Suppose that three processes A, B, and C are running on the system. At some point in time, the kernel decides to switch from A to B, then from B to C, and then from C back to A again. Before each switch_to call, the pointers next and prev located on the stacks of the individual processes are set such that prev 104 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 105 Chapter 2: Process Management and Scheduling points to the currently running process, while next points to the process that will be running next. To perform the switch from prev to next, the first two arguments are completely sufficient for switch_to. For process A, prev points to A and next points to B. A problem arises when A is selected to execute again. Control will return to the point after switch_to, and if the stack were restored to the exact state it had before the switch, prev and next would still point to the same values as before the switch — namely, next=B and prev=A.Inthissituation,thekernelwould not know that process C has actually run before process A. Therefore, the low-level task switch routine must feed the previously executing task to context_switch when a new task is selected. Since control flow comes back to the middle of the function, this cannot be done with regular function return values, and that is why a three-parameter macro is used. However, the conceptional effect is the same as if switch_to were a function of two arguments that would return a pointer to the previously executing process. What switch_to essentially does is prev = switch_to(prev,next) where the prev value returned is not the prev value used as the argument, but the process that executed last in time. In the above example, process A would feed switch_to with A and B, but would obtain prev=C as result. How the kernel implements this behavior depends on the underlying architecture, but it is obvious that the kernel can reconstruct the desired information by considering the kernel mode stacks of both processes — which are naturally simultaneously available to the kernel, which can access all memory at will. Lazy FPU Mode Because the speed of context switching plays a major role in system performance, the kernel uses a trick to reduce the CPU time needed. Floating-point registers (and other extended registers not used by the kernel; e.g., the SSE2 registers on IA-32 platforms) are not saved unless they are actually used by the application and are not restored unless they are required. This is known as the lazy FPU technique. Its implementation differs from platform to platform because assembly language code is used, but the basic principle is always the same. It should also be noted that, regardless of platform, the contents of the floating-point registers are not saved on the process stack but in its thread data structure. I illustrate this technique by means of an example. For the sake of simplicity, let us assume this time that there are only two processes, A and B, on the system. Process A is running and uses floating-point operations. When the scheduler switches to process B, the contents of the floating-point registers of A are saved in the thread data structure of the process. However, the values in these registers are not immediately replaced with the values for process B. If B does not perform any floating-point operations during its time slice, A sees its former register con- tents when it is next activated. The kernel is therefore spared the effort of explicitly restoring register values, and this represents a time-saving. If, however, B does perform floating-point operations, this fact is reported to the kernel so that it can fill the registers with the appropriate values from the thread data structure. Consequently, the kernel saves and restores floating-point register contents only when needed and wastes no time on superfluous operations. 105 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 106 Chapter 2: Process Management and Scheduling 2.6 The Completely Fair Scheduling Class All information that the core scheduler needs to know about the completely fair scheduler is contained in fair_sched_class: kernel/sched_fair.c static const struct sched_class fair_sched_class = { .next = &idle_sched_class, .enqueue_task = enqueue_task_fair, .dequeue_task = dequeue_task_fair, .yield_task = yield_task_fair, .check_preempt_curr = check_preempt_wakeup, .pick_next_task = pick_next_task_fair, .put_prev_task = put_prev_task_fair, ... .set_curr_task = set_curr_task_fair, .task_tick = task_tick_fair, .task_new = task_new_fair, }; We have seen in the previous discussion when these functions are called by the main scheduler and will examine in the following how they are implemented for CFS. 2.6.1 Data Structures First, I need to introduce how the CFS run queue looks. Recall that an instance is embedded into each per-CPU run queue of the main scheduler: kernel/sched.c struct cfs_rq { struct load_weight load; unsigned long nr_running; u64 min_vruntime; struct rb_root tasks_timeline; struct rb_node *rb_leftmost; struct sched_entity *curr; } The individual elements have the following meaning: ❑ nr_running counts the number of runnable processes on the queue, and load maintains the cumulative load values of them all. Recall that you have already encountered the load calcu- lation in Section 2.5.3. ❑ min_vruntime tracks the minimum virtual run time of all processes on the queue. This value forms the basis to implement the virtual clock associated with a run queue. The name is slightly confusing because min_vruntime can actually be bigger than the vruntime setting of the leftmost 106 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 107 Chapter 2: Process Management and Scheduling tree element as it needs to increase monotonically, but I will come back to this when I discuss how the value is set in detail. ❑ tasks_timeline is the base element to manage all processes in a time-ordered red-black tree. rb_leftmost is always set to the leftmost element of the tree, that is, the element that deserves to be scheduled most. The element could, in principle, be obtained by walking through the red- black tree, but since usually only the leftmost element is of interest, this speeds up the average time spent searching the tree. ❑ curr points to the schedulable entity of the currently executing process. 2.6.2 CFS Operations Let us now turn our attention to how the scheduling methods provided by the CF scheduler are imple- mented. The Virtual Clock I discussed in the Introduction that the completely fair scheduling algorithm depends on a virtual clock that measures the amount of time a waiting process would have been allowed to spend on the CPU on a completely fair system. However, no virtual clock can be found anywhere in the data structures! This is because all required information can be inferred from the existing real-time clocks combined with the load weight associated with every process. All calculations related to the virtual clock are performed in update_curr, which is called from various places in the system including the periodic scheduler. The code flow diagram in Figure 2-17 provides an overview of what the function does. Update physical and virtual run time of the process Update min_vruntime of the CFS queue Set rq->exec_start _ _update_curr update_curr Figure 2-17: Code flow diagram for update_curr. First of all, the function determines the currently executing process of the run queue and also obtains the real clock value of the main scheduler run queue, which is updated at each scheduler tick (rq_of is an auxiliary function to determine the instance of struct rq that is associated with a CFS run queue): static void update_curr(struct cfs_rq *cfs_rq) { struct sched_entity *curr = cfs_rq->curr; u64 now = rq_of(cfs_rq)->clock; unsigned long delta_exec; if (unlikely(!curr)) return; ... 107 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 108 Chapter 2: Process Management and Scheduling If no process is currently executing on the run queue, there is obviously nothing to do. Otherwise, the kernel computes the time difference between the last update of the load statistics and now, and delegates the rest of the work to __update_curr. kernel/sched_fair.c delta_exec = (unsigned long)(now - curr->exec_start); __update_curr(cfs_rq, curr, delta_exec); curr->exec_start = now; } Based on this information, __update_curr has to update the physical and virtual time that the current process has spent executing on the CPU. This is simple for the physical time. The time difference just needs to be added to the previously accounted time: kernel/sched_fair.c static inline void __update_curr(struct cfs_rq *cfs_rq, struct sched_entity *curr, unsigned long delta_exec) { unsigned long delta_exec_weighted; u64 vruntime; curr->sum_exec_runtime += delta_exec; ... The interesting thing is how the non-existing virtual clock is emulated using the given information. Once more, the kernel is clever and saves some time in the common case: For processes that run at nice level 0, virtual and physical time are identical by definition. When a different priority is used, the time must be weighted according to the load weight of the process (recall that Section 2.5.3 discussed how process priority and load weight are connected): kernel/sched_fair.c delta_exec_weighted = delta_exec; if (unlikely(curr->load.weight != NICE_0_LOAD)) { delta_exec_weighted = calc_delta_fair(delta_exec_weighted, &curr->load); } curr->vruntime += delta_exec_weighted; ... Neglecting some rounding and overflow checking, what calc_delta_fair does is to compute the value given by the following formula: delta_exec_weighted = delta_exec × NICE_0_LOAD curr->load.weight The inverse weight values mentioned above can be brought to good use in this calculation. Recall that more important tasks with higher priorities (i.e., lower nice values) will get larger weights, so the virtual run time accounted to them will be smaller. Figure 2-18 illustrates the connection between real and virtual time for various priorities. One can also see from the formula that the virtual and physical time are identical for nice 0 tasks with priority 120, that is, if current->load.weight is NICE_0_LOAD.Noticethat the inset in Figure 2-18 uses a double logarithmic plot to show a wider range of priorities. 108 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 109 Chapter 2: Process Management and Scheduling 0 500 1000 1500 2000 2500 3000 3500 0 100 200 300 400 500 600 700 800 900 1000 Virtual time interval (milliseconds) Real time interval (milliseconds) Nice 0 (prio 120) Nice +5 (prio 125) Nice −5 (prio 115) 1 10 100 1000 10000 100000 1 10 100 1000 Nice 0 (prio 120) Nice +10 (prio 130) Nice +19 (prio 139) Nice −10 (prio 110) Nice −20 (prio 100) Figure 2-18: Relation between real and virtual time for processes depending on their priority/nice level. Finally, the kernel needs to set min_vruntime. Care is taken to ensure that the value is increasing mono- tonically. kernel/sched_fair.c /* * maintain cfs_rq->min_vruntime to be a monotonically increasing * value tracking the leftmost vruntime in the tree. */ if (first_fair(cfs_rq)) { vruntime = min_vruntime(curr->vruntime, __pick_next_entity(cfs_rq)->vruntime); } else vruntime = curr->vruntime; cfs_rq->min_vruntime = max_vruntime(cfs_rq->min_vruntime, vruntime); } first_fair is a helper function that checks if the tree has a leftmost element, that is, if any process is waiting on the tree to be scheduled. If so, the kernel obtains its vruntime, which is the smallest of all elements in the tree. If no leftmost element is in the tree because it is empty, the virtual run time of the current process is used instead. To ensure that the per-queue min_vruntime is monotonic increasing, the kernel sets it to the larger of both values. This means that the per-queue min_vruntime is only updated if it is exceeded by the vruntime of one of the elements on the tree. With this policy, the kernel ensures that min_vrtime can only increase, but never decrease. 109 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 110 Chapter 2: Process Management and Scheduling One really crucial point of the completely fair scheduler is that sorting processes on the red-black tree is based on the following key: kernel/sched_fair.c static inline s64 entity_key(struct cfs_rq *cfs_rq, struct sched_entity *se) { return se->vruntime - cfs_rq->min_vruntime; } Elements with a smaller key will be placed more to the left, and thus be scheduled more quickly. This way, the kernel implements two antagonistic mechanisms: 1. When a process is running, its vruntime will steadily increase, so it will finally move right- ward in the red-black tree. Because vruntime will increase more slowly for more important processes, they will also move rightward more slowly, so their chance to be scheduled is bigger than for a less impor- tant process — just as required. 2. If a process sleeps, its vruntime will remain unchanged. Because the per-queue min_vruntime increases in the meantime (recall that it is monotonic!), the sleeper will be placed more to the left after waking up because the key got smaller.28 In practice, both effects naturally happen simultaneously, but this does not influence the interpretation. Figure 2-19 illustrates the different movement mechanisms on the red-black tree graphically. min_vruntime vruntime vruntime Value increases Value decreases Position in the red- black tree (more to the left is better) Figure 2-19: Influence of the per-entity and per-queue virtual times on the placement of processes in the red-black tree. Latency Tracking The kernel has a built-in notion of what it considers a good scheduling latency, that is, the interval during which every runnable task should run at least once.29 It is given in sysctl_sched_latency,which can be controlled via /proc/sys/kernel/sched_latency_ns and defaults to, respectively, 20,000,000 ns (nanoseconds) and 20 ms (milliseconds). A second control parameter, sched_nr_latency, controls the number of active processes that are at most handled in one latency period. If the number of active pro- cesses grows larger than this bound, the latency period is extended linearly. sched_nr_latency can be indirectly controlled via sysctl_sched_min_granularity, which can be set via /proc/sys/kernel/ sched_min_granularity_ns. The default value is 4,000,000 ns, that is, 4 ms, and sched_nr_latency is 28This is slightly different for short sleepers, but I consider this situation when I discuss the exact mechanism. 29Caution: This has nothing to do with time slices, which were used by the old scheduler! 110 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 111 Chapter 2: Process Management and Scheduling computed as sysctl_sched_latency/sysctl_sched_min_granularity each time one of the values is changed. __sched_period determines the length of the latency period, which is usually just sysctl_sched_latency, but is extended linearly if more processes are running. In this case, the period length is sysctl_sched_latency × nr_running sched_nr_latency. Distribution of the time among active processes in one latency period is performed by considering the rel- ative weights of the respective tasks. The slice length for a given process as represented by a schedulable entity is computed as follows: kernel/sched_fair.c static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) { u64 slice = __sched_period(cfs_rq->nr_running); slice *= se->load.weight; do_div(slice, cfs_rq->load.weight); return slice; } Recall that the run queue load weight accumulates the load weights of all active processes on the queue. The resulting time slice is given in real time, but the kernel sometimes also needs to know the equivalent in virtual time. kernel/sched_fair.c static u64 __sched_vslice(unsigned long rq_weight, unsigned long nr_running) { u64 vslice = __sched_period(nr_running); vslice *= NICE_0_LOAD; do_div(vslice, rq_weight); return vslice; } static u64 sched_vslice(struct cfs_rq *cfs_rq) { return __sched_vslice(cfs_rq->load.weight, cfs_rq->nr_running); } Recall that a real-time interval time for a process with a given weight has the length time × NICE_0_LOAD weight , and this is also used to transfer the latency interval portion. Now we have everything in place to discuss the various methods that must be implemented by CFS to interact with the global scheduler. 111 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 112 Chapter 2: Process Management and Scheduling 2.6.3 Queue Manipulation Two functions are available to move elements to and from the run queue: enqueue_task_fair and dequeue_task_fair. Let us concentrate on placing new tasks on the run queue first. Besides pointers to the generic run queue and the task structure in question, the function accepts one more parameter: wakeup. This allows for specifying if the task that is enqueued has only recently been woken up and changed into the running state (wakeup is 1 in this case), or if it was runnable before (wakeup is 0 then). The code flow diagram for enqueue_task_fair is shown in Figure 2-20. enqueue_task_fair return enqueue_entity update_curr place_entity enqueue_sleeper _ _enqueue_entityse != cfs_rq->curr Task has been woken up? Already on runqueue? Figure 2-20: Code flow diagram for enqueue_task_fair. If the task is already on the run queue as signaled by the on_rq element of struct sched_entity,noth- ing needs to be done. Otherwise, the work is delegated to enqueue_entity, where the kernel takes the opportunity to update the statistics with updater_curr. If the task has recently been running, its virtual run time is still valid, and (unless it is currently executing) it can be directly included into the red-black tree with __enqueue_entity. This function requires some mechanics to handle the red-black tree, but it can rely on standard methods of the kernel (see Appendix C for more information) and is thus not very interesting. The essential point is that the process is placed at the proper position, but this has already been ensured before by setting the vruntime field of the process, and by the constant min_vruntime updates performed by the kernel for the queue. If the process has been sleeping before, the virtual run time of the process is first adjusted in place_entity30: kernel/sched_fair.c static void place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) 30Note that the real kernel sources will execute portions of the code depending on outcomes of sched_feature queries. The CF scheduler supports some ‘‘configurable’’ features, but they can only be turned on and off in debugging mode — otherwise, the set of features is fixed. I will therefore ignore the feature selection mechanism and consider only those that are always compiled in and active. 112 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 113 Chapter 2: Process Management and Scheduling { u64 vruntime; vruntime = cfs_rq->min_vruntime; if (initial) vruntime += sched_vslice_add(cfs_rq, se); if (!initial) { vruntime -= sysctl_sched_latency; vruntime = max_vruntime(se->vruntime, vruntime); } se->vruntime = vruntime; } The function distinguishes between two cases depending on the value of initial. This parameter is only set if a new task is added to the system, but that’s not the case here: initial is zero (I will come back to the other case when I discuss task_new_fair below). Since the kernel has promised to run all active processes at least once within the current latency period, the min_vruntime of the queue is used as the base virtual time, and by subtracting sysctl_sched_latency, it is ensured that the newly awoken process will only run after the current latency period has been finished. However, if the sleeper has accumulated a large unfairness as indicated by a large se_vruntime value, the kernel must honor this. If se->vruntime is larger than the previously computed difference, it is kept as the vruntime of the process, which leads to a leftward placement on the red-black tree — recall that large vruntime values are good to schedule early! Let us go back to enqueue_entity:Afterplace_entity has determined the proper virtual run time for the process, it is placed on the red-black tree with __enqueue_entity. I have already noted before that this is a purely mechanical function that uses standard methods of the kernel to sort the task into the red-black tree. 2.6.4 Selecting the Next Task Selecting the next task to run is performed in pick_next_task_fair. The code flow diagram is shown in Figure 2-21. If no tasks are currently runnable on the queue as indicated by an empty nr_running counter, there is little to do and the function can return immediately. Otherwise, the work is delegated to pick_next_entity. If a leftmost task is available in the tree, it can immediately be determined using the first_fair helper function, and __pick_next_entity extracts the sched_entity instance from the red-black tree. This is done using the container_of mechanism because the red-black tree manages instances of rb_node that are embedded in sched_entitys. Now the task has been selected, but some more work is required to mark it as the running task. This is handled by set_next_entity. 113 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 114 Chapter 2: Process Management and Scheduling kernel/sched_fair.c static void set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) { /* ’current’ is not kept within the tree. */ if (se->on_rq) { __dequeue_entity(cfs_rq, se); } ... No processes running? Leftmost task in red-black tree available? pick_next_task_fair pick_next_entity set_next_entity return __pick_next_entity Figure 2-21: Code flow diagram for pick_next_task_fair. The currently executing process isnotkeptontherunqueue,so__dequeue_entity removes it from the red-black tree, setting the leftmost pointer to the next leftmost task if the current task has been the leftmost one. Notice that in our case, the process has been on the run queue for sure, but this need not be the case when set_next_entity is called from different places. Although the process is not contained in the red-black tree anymore, the connection between process and run queue is not lost, because curr marks it as the running one now: kernel/sched_fair.c cfs_rq->curr = se; se->prev_sum_exec_runtime = se->sum_exec_runtime; } Because the process is now the currently active one, the real time spent on the CPU will be charged to sum_exec_runtime, so the kernel preserves the previous setting in prev_sum_exec_runtime.Note that sum_exec_runtime is not reset in the process. The difference sum_exec_runtime - prev_sum_ exec_runtime does therefore denote the real time spent executing on a CPU. 2.6.5 Handling the Periodic Tick This aforementioned difference is important when the periodic tick is handled. The formally responsible function is task_tick_fair, but the real work is done in entity_tick. Figure 2-22 presents the code flow diagram. 114 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 115 Chapter 2: Process Management and Scheduling More than one process running? Latency limit exceeded? entity_tick update_curr resched_task Check_preempt_tick Figure 2-22: Code flow diagram for entity_tick. First of all, the statistics are — as always — updated using update_curr.Ifthenr_running counter of the queue indicates that fewer than two processes are runnable on the queue, nothing needs to be done. If a process is supposed to be preempted, there needs to be at least another one that could preempt it. Otherwise, the decision is left to check_preempt_tick: kernel/sched_fair.c static void check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr) { unsigned long ideal_runtime, delta_exec; ideal_runtime = sched_slice(cfs_rq, curr); delta_exec = curr->sum_exec_runtime - curr->prev_sum_exec_runtime; if (delta_exec > ideal_runtime) resched_task(rq_of(cfs_rq)->curr); } The purpose of the function is to ensure that no process runs longer than specified by its share of the latency period. This length of this share in real-time is computed in sched_slice,andthereal- time interval during which the process has been running on the CPU is given by sum_exec_runtime - prev_sum_exec_runtime as explained above. The preemption decision is thus easy: If the task has been running for longer than the desired time interval, a reschedule is requested with resched_task.Thissets the TIF_NEED_RESCHED flag in the task structure, and the core scheduler will initiate a rescheduling at the next opportune moment. 2.6.6 Wake-up Preemption When tasks are woken up in try_to_wake_up and wake_up_new_task, the kernel uses check_preempt_curr to see if the new task can preempt the currently running one. Notice that the core scheduler is not involved in this process! For completely fair handled tasks, the function check_ preempt_wakeup performs the desired check. 115 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 116 Chapter 2: Process Management and Scheduling The newly woken task need not necessarily be handled by the completely fair scheduler. If the new task is a real-time task, rescheduling is immediately requested because real-time tasks always preempt CFS tasks: kernel/sched_fair.c static void check_preempt_wakeup(struct rq *rq, struct task_struct *p) { struct task_struct *curr = rq->curr; struct cfs_rq *cfs_rq = task_cfs_rq(curr); struct sched_entity *se = &curr->se, *pse = &p->se; unsigned long gran; if (unlikely(rt_prio(p->prio))) { update_rq_clock(rq); update_curr(cfs_rq); resched_task(curr); return; } ... The most convenient cases are SCHED_BATCH tasks — they do not preempt other tasks by definition. kernel/sched.c if (unlikely(p->policy == SCHED_BATCH)) return; ... When a running task is preempted by a new task, the kernel ensures that the old one has at least been running for a certain minimum amount of time. The minimum is kept in sysctl_sched_ wakeup_granularity, which crossed our path before. Recall that it is per default set to 4 ms. This refers to real time, so the kernel first needs to convert it into virtual time if required: kernel/sched_fair.c gran = sysctl_sched_wakeup_granularity; if (unlikely(se->load.weight != NICE_0_LOAD)) gran = calc_delta_fair(gran, &se->load); ... If the virtual run time of the currently executing task (represented by its scheduling entity se)islarger than the virtual run time of the new task plus the granularity safety, a rescheduling is requested: kernel/sched_fair.c if (pse->vruntime + gran < se->vruntime) resched_task(curr); } The added ‘‘buffer’’ time ensures that tasks are not switched too frequently so that not too much time is spent in context switching instead of doing real work. 2.6.7 Handling New Tasks The last operation of the completely fair scheduler that we need to consider is the hook function that is called when new tasks are created: task_new_fair. The behavior of the function is controllable with 116 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 117 Chapter 2: Process Management and Scheduling the parameter sysctl_sched_child_runs_first. As the name might suggest, it determined if a newly created child process should run before the parent. This is usually beneficial, especially if the child per- forms an exec system call afterward. The default setting is 1, but this can be changed via /proc/sys/ kernel/sched_child_runs_first. Initially, the function performs the usual statistics update with update_curr and then employs the pre- viously discussed place_entity: kernel/sched_fair.c static void task_new_fair(struct rq *rq, struct task_struct *p) { struct cfs_rq *cfs_rq = task_cfs_rq(p); struct sched_entity *se = &p->se, *curr = cfs_rq->curr; int this_cpu = smp_processor_id(); update_curr(cfs_rq); place_entity(cfs_rq, se, 1); ... In this case, place_entity is, however, called with initial set to 1, which amounts to computing the initial vruntime with sched_vslice_add. Recall that this determines the portion of the latency interval that belongs to the process, but converted to virtual time. This is the scheduler’s initial debt to the process. kernel/sched_fair.c if (sysctl_sched_child_runs_first && curr->vruntime < se->vruntime) { swap(curr->vruntime, se->vruntime); } enqueue_task_fair(rq, p, 0); resched_task(rq->curr); } If the virtual run time of the parent (represented by curr) is less than the virtual run time of the child, this would mean that the parent runs before the child — recall that small virtual run times favor left positions in the red-black tree. If the child is supposed to run before the parent, the virtual run times of both need to be swapped. Afterward, the child is enqueued into the run queue as usual, and rescheduling is requested. 2.7 The Real-Time Scheduling Class As mandated by the POSIX standard, Linux supports two real-time scheduling classes in addition to ‘‘normal‘‘ processes. The structure of the scheduler enables real-time processes to be integrated into the kernel without any changes in the core scheduler — this is a definitive advantage of scheduling classes.31 Now is a good place to recall some of the facts discussed a long time ago. Real-time processes can be iden- tified by the fact that they have a higher priority than normal processes — accordingly, their static_prio value is always lower than that of normal processes, as shown in Figure 2-14. The rt_task macro is 31The completely fair scheduler needs to be aware of real-time processes in the wake-up preemption code, but this requires only very little effort. 117 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 118 Chapter 2: Process Management and Scheduling provided to establish whether a given task is a real-time process or not by inspecting its priority, and task_has_rt_policy checks if the process is associated with a real-time scheduling policy. 2.7.1 Properties Real-time processes differ from normal processes in one essential way: If a real-time process exists in the system and is runnable, it will always be selected by the scheduler — unless there is another real-time process with a higher priority. The two available real-time classes differ as follows: ❑ Round robin processes (SCHED_RR) have a time slice whose value is reduced when they run if they are normal processes. Once all time quantums have expired, the value is reset to the initial value, but the process is placed at the end of the queue. This ensures that if there are several SCHED_RR processes with the same priority, they are always executed in turn. ❑ First-in, first-out processes (SCHED_FIFO) do not have a time slice and are permitted to run as long as they want once they have been selected. It is evident that the system can be rendered unusable by badly programmed real-time processes — all that is needed is an endless loop whose loop body never sleeps. Extreme care should therefore be taken when writing real-time applications.32 2.7.2 Data Structures The scheduling class for real-time tasks is defined as follows: kernel/sched-rt.c const struct sched_class rt_sched_class = { .next = &fair_sched_class, .enqueue_task = enqueue_task_rt, .dequeue_task = dequeue_task_rt, .yield_task = yield_task_rt, .check_preempt_curr = check_preempt_curr_rt, .pick_next_task = pick_next_task_rt, .put_prev_task = put_prev_task_rt, .set_curr_task = set_curr_task_rt, .task_tick = task_tick_rt, }; The implementation of the real-time scheduler class is simpler than the completely fair scheduler. Only roughly 250 lines of code compared to 1,100 for CFS are required! The core run queue also contains a sub-run queue for real-time tasks as embedded instance of struct rt_rq: 32Notice that this situation will be eased with the introduction of real-time group scheduling in kernel 2.6.25, which was still under development when this book was written. 118 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 119 Chapter 2: Process Management and Scheduling kernel/sched.c struct rq { ... t_rq rt; ... } The run queue is very straightforward — a linked list is sufficient33: kernel/sched.c struct rt_prio_array { DECLARE_BITMAP(bitmap, MAX_RT_PRIO+1); /* include 1 bit for delimiter */ struct list_head queue[MAX_RT_PRIO]; }; struct rt_rq { struct rt_prio_array active; }; All real-time tasks with the same priority are kept in a linked list headed by active.queue[prio],and the bitmap active.bitmap signals in which list tasks are present by a set bit. If no tasks are on the list, the bit is not set. Figure 2-23 illustrates the situation. Increasing prio Figure 2-23: Run queue of the real-time scheduler. The analog of update_cur for the real-time scheduler class is update_curr_rt: The function keeps track of the time the current process spent executing on the CPU in sum_exec_runtime. All calculations are performed with real times; virtual times are not required. This simplifies things a lot. 2.7.3 Scheduler Operations To enqueue and dequeue tasks is simple: The task is placed or respectively removed from the appropriate list selected by array->queue + p->prio, and the corresponding bit in the bitmap is set if at least one task is present, or removed if no tasks are left on the queue. Notice that new tasks are always queued at the end of each list. The two interesting operations are how the next task is selected and how preemption is handled. Con- sider pick_next_task_rt, which handles selection of the next task first. The code flow diagram is shown in Figure 2-24. 33SMP systems require some more elements for load balancing, but these do not concern us here. 119 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 120 Chapter 2: Process Management and Scheduling Dequeue task Set se.exec_start pick_next_task_rt sched_find_first_bit Figure 2-24: Code flow diagram for pick_next_task_rt. sched_find_first_bit is a standard function that finds the first set bit in active.bitmap — this means that higher real-time priorities (which result in lower in-kernel priorities) are handled before lower real- time priorities. The first task on the selected list is taken out, and se.exec_start is set to the current real-time clock value of the run queue — that’s all that is required. The implementation of the periodic tick is likewise simple. SCHED_FIFO tasks are easiest to handle: They can run as long as they like and must pass control to another task explicitly by using the yield system call: kernel/sched.c static void task_tick_rt(struct rq *rq, struct task_struct *p) { update_curr_rt(rq); /* * RR tasks need a special form of timeslice management. * FIFO tasks have no timeslices. */ if (p->policy != SCHED_RR) return; ... If the current process is a round robin process, its time slice is decremented. When the time quantum is not yet exceeded, nothing more needs to be done — the process can keep running. Once the counter reverts to 0, its value is renewed to DEF_TIMESLICE, which is set to 100 * HZ / 1000, that is, 100 ms. If the task is not the only task in its list, it is requeued to the end. Rescheduling is requested as usual by setting the TIF_NEED_RESCHED flag with set_tsk_need_resched: kernel/sched-rt.c if (--p->time_slice) return; p->time_slice = DEF_TIMESLICE; /* * Requeue to the end of queue if we are not the only element * on the queue: */ if (p->run_list.prev != p->run_list.next) { 120 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 121 Chapter 2: Process Management and Scheduling requeue_task_rt(rq, p); set_tsk_need_resched(p); } } The sched_setscheduler system call must be used to convert a process into a real-time process. This call is not discussed at length because it performs only the following simple tasks: ❑ It removes the process from its current queue using deactivate_task. ❑ It sets the real-time priority and the scheduling class in the task data structure. ❑ It reactivates the task. If the process was not previously on any run queue, only the scheduling class and the new priority value need be set; deactivation and reactivation are unnecessary. Note that changing the scheduler class or priority is only possible without constraints if the sched_setscheduler system call is performed by a process with root rights (or, equivalently, the capability CAP_SYS_NICE). Otherwise, the following conditions apply: ❑ The scheduling class can only be changed from SCHED_NORMAL to SCHED_BATCH or vice versa. A change to SCHED_FIFO is impossible. ❑ Only the priority of processes with the same UID or EUID as the EUID of the caller can be changed. Additionally, the priority may only be decreased, but not increased. 2.8 Scheduler Enhancements So far, we have only considered scheduling on real-time systems — naturally, Linux can do slightly better. Besides support for multiple CPUs, the kernel also provides several other enhancements that relate to scheduling, discussed in the following sections. Notice, however, that these enhancements add much complexity to the scheduler, so I will mostly consider simplified situations that illuminate the essential principle, but do not account for all boundary cases and scheduling oddities. 2.8.1 SMP Scheduling On multiprocessor systems, the kernel must consider a few additional issues in order to ensure good scheduling: ❑ The CPU load must be shared as fairly as possible over the available processors. It makes little sense if one processor is responsible for three concurrent applications while another has only the idle task to deal with. ❑ The affinity of a task to certain processors in the system must be selectable. This makes it possible, for example, to bind a compute-intensive application to the first three CPUs on a 4-CPU system while the remaining (interactive) processes run on the fourth CPU. 121 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 122 Chapter 2: Process Management and Scheduling ❑ The kernel must be able to migrate processes from one CPU to another. However, this option must be used with great care because it can severely impair performance. CPU caches are the biggest problem on smaller SMP systems. For really big systems, a CPU can be located literally some meters away from the memory previously used, so access to it will be very costly. The affinity of a task to particular CPUs is defined in the cpus_allowed element of the task structure specified above. Linux provides the sched_setaffinity system call to change this assignment. Extensions to the Data Structures The scheduling methods that each scheduler class must provide are augmented by two additional func- tions on SMP systems: struct sched_class { ... #ifdef CONFIG_SMP unsigned long (*load_balance) (struct rq *this_rq, int this_cpu, struct rq *busiest, unsigned long max_load_move, struct sched_domain *sd, enum cpu_idle_type idle, int *all_pinned, int *this_best_prio); int (*move_one_task) (struct rq *this_rq, int this_cpu, struct rq *busiest, struct sched_domain *sd, enum cpu_idle_type idle); #endif ... } Despite their names, the functions are, however, not directly responsible to handle load balancing. They are called by the core scheduler code whenever the kernel deems rebalancing necessary. The scheduler class-specific functions then set up an iterator that allows the generic code to walk through all processes that are potential candidates to be moved to another queue, but the internal structures of the individual scheduler classes must not be exposed to the generic code because of the iterator. load_balance employs the generic function load_balance, while move_one_task uses iter_move_one_task. The functions serve different purposes: ❑ iter_move_one_task picks one task off the busy run queue busiest and moves it to the run queue of the current CPU. ❑ load_balance is allowed to distribute multiple tasks from the busiest run queue to the current CPU, but must not move more load than specified by max_load_move. How is load balancing initiated? On SMP systems, the scheduler_tick periodic scheduler function invokes the trigger_load_balance function on completion of the tasks required for all systems as described above. This raises the SCHEDULE_SOFTIRQ softIRQ (the software analog to hardware interrupts; see Chapter 14 for more details), which, in turn, guarantees that run_rebalance_domains will be run in due time. This function finally invokes load balancing for the current CPU by calling rebalance_domains. The time flow is illustrated in Figure 2-25. 122 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 123 Chapter 2: Process Management and Scheduling To perform rebalancing, the kernel needs some more information. Run queues are therefore augmented with additional fields on SMP systems: kernel/sched.c struct rq { ... #ifdef CONFIG_SMP struct sched_domain *sd; /* For active balancing */ int active_balance; int push_cpu; /* cpu of this runqueue: */ int cpu; struct task_struct *migration_thread; struct list_head migration_queue; #endif ... } Timer tick Raise SCHEDULE_SOFTIRQ SoftIRQ scheduler_tick trigger_load_balance run_rebalance_domains rebalance_domains Figure 2-25: Time flow for initiation of load balancing on SMP systems. Run queues are CPU-specific, so cpu denotes the processor to which the run queue belongs. The ker- nel provides one migration thread per run queue to which migration requests can be posted — they are kept on the list migration_queue. Such requests usually originate from the scheduler itself, but can also become necessary when a process is restricted to a certain set of CPUs and must not run on the one it is currently executing on anymore. The kernel tries to balance run queues periodically, but if this fails to be satisfactory for a run queue, then active balancing must be used. active_balance is set to a nonzero value if this is required, and cpu notes the processor from which the request for active balancing initiates. Furthermore, all run queues are organized in scheduling domains. This allows for grouping CPUs that are physically adjacent to each other or share a common cache such that processes should preferably be moved between them. On ‘‘normal’’ SMP systems, however, all processors will be contained in one scheduling domain. I will therefore not discuss this structure in detail, but only mention that it contains numerous parameters that can be set via /proc/sys/kernel/cpuX/domainY. These include the minimal and maximal time interval after which load balancing is initiated, the minimal imbalance for a queue to be re-balanced, and so on. Besides, the structure also manages fields that are set at run time and that 123 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 124 Chapter 2: Process Management and Scheduling allow the kernel to keep track when the last balancing operation has been performed, and when the next will take place. So what does load_balance do? The function checks if enough time has elapsed since the last re-balancing operation, and initiates a new re-balancing cycle if necessary by invoking load_balance. The code flow diagram for this function is shown in Figure 2-26. Notice that I describe a simplified version because the SMP scheduler has to deal with a very large number of corner cases that obstruct the view on the essential actions. More than one process on busiest queue? Balancing failed? Wake up migration task load_balance find_busiest_cpu move_tasks class->load_balance Figure 2-26: Code flow diagram for load_balance. First of all, the function has to identify which queue has most work to do. This task is delegated to find_busiest_queue, which is called for a specific run queue. The function iterates over the queues of all processors (or, to be precise, of all processors in the current scheduling group) and compares their load weights. The busiest queue is the queue with the largest value found in the end. Once find_busiest_queue has identified a very busy queue, and if at least one task is running on this queue (load balancing will otherwise not make too much sense), a suitable number of its tasks are migrated to the current queue using move_tasks. This function, in turn, invokes the scheduler-class- specific load_balance method. When selecting potential migration candidates, the kernel must ensure that the process in question ❑ is not running at the moment or has just finished running because this fact would cancel out the benefits of the CPU caches currently filled with the process data. ❑ may execute on the processor associated with the current queue on the grounds of its CPU affin- ity. If balancing failed (e.g., because all tasks on the remote queue have a higher kernel-internal priority value, i.e., a lower nice priority), the migration thread that is responsible for the busiest run queue is woken up. To ensure that active load balancing is performed that is slightly more aggressive than the method tried now, load_balance sets the active_balance flag of the busiest run queue and also notes the CPU from which the request originates in rq->cpu. The Migration Thread The migration thread serves two purposes: It must fulfill migration requests originating from the sched- uler, and it is used to implement active balancing. This is handled in a kernel thread that executes migration_thread. The code flow diagram for the function is shown in Figure 2-27. 124 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 125 Chapter 2: Process Management and Scheduling class-> move_one_task No migration requests? schedule Get migration request Iterate over all scheduler classes 1 1 migration_thread rq->active_balance set? active_load_balance move_one_task __migrate_task Figure 2-27: Code flow diagram for migration_thread. migration_thread runs an infinite loop and sleeps when there is nothing to do. First of all, the function checks if active balancing is required, and if this is the case, active_load_balance is called to satisfy this request. The function tries to move one task from the current run queue to the run queue of the CPU that initiated the request for active balancing. It uses move_one_task forthispurpose,which,inturn,ends up calling the scheduler-class specific move_one_task functions of all scheduler classes until one of them succeeds. Note that these functions try to move processes more aggressively than load_balance.For instance, they do not perform the previously mentioned priority comparison, so they are more likely to succeed. Once the active load balancing is finished, the migration thread checks if any migration requests from the scheduler are pending in the migrate_req list. If none is available, the thread can reschedule. Otherwise, the request is fulfilled with __migrate_task, which performs the desired process movement directly without further interaction with the scheduler classes. Core Scheduler Changes Besides the additions discussed above, some changes to the existing methods are required in the core scheduler on SMP systems. While numerous small details change all over the place, the most important differences as compared to uniprocessor systems are the following: ❑ When a new process is started with the exec system call, a good opportunity for the sched- uler to move the task across CPUs arises. Naturally, it has not been running yet, so there can- not be any negative effects on the CPU cache by moving the task to another CPU. sched_exec is the hook function invoked by the exec system call, and the code flow diagram is shown in Figure 2-28. sched_balance_self picks the CPU that is currently least loaded (and on which the process is also allowed to run). If this is not the current CPU, then sched_migrate_task forwards an according migration request to the migration thread using sched_migrate_task. ❑ The scheduling granularity of the completely fair scheduler scales with the number of CPUs. The more processors present in the system, the larger the granularities that can be employed. Both sysctl_sched_min_granularity and sysctl_sched_latency for sysctl_sched_min_ 125 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 126 Chapter 2: Process Management and Scheduling granularity are multiplied by the correction factor 1 + log2(nr_cpus), where nr_cpus repre- sents the number of available CPUs. However, they must not exceed 200 ms. sysctl_sched_ wakeup_granularity is also increased by the factor, but is not bounded from above. New CPU selected? sched_exec sched_balance_self sched_migrate_task Figure 2-28: Code flow diagram for sched_exec. 2.8.2 Scheduling Domains and Control Groups In the previous discussion of the scheduler code, we have often come across the situation that the sched- uler does not deal directly with processes, but with schedulable entities. This allows for implementing group scheduling: Processes are placed into different groups, and the scheduler is first fair among these groups and then fair among all processes in the group. This allows, for instance, granting identical shares of the available CPU time to each user. Once the scheduler has decided how much time each user gets, the determined interval is then distributed between the users’ processes in a fair manner. This naturally implies that the more processes a user runs, the less CPU share each process will get. The amount of time for the user in total is not influenced by the number of processes, though. Grouping tasks between users is not the only possibility. The kernel also offers control groups,which allow — via the special filesystem cgroups — creating arbitrary collections of tasks, which may even be sorted into multiple hierarchies. The situation is illustrated in Figure 2-29. Schedulable entity task Figure 2-29: Overview of fair group scheduling: The available CPU time is first distributed fairly among the scheduling groups, and then between the processes in each group. To reflect the hierarchical situation within the kernel, struct sched_entity is augmented with an ele- ment that allows for expressing this hierarchy: struct sched_entity { ... #ifdef CONFIG_FAIR_GROUP_SCHED struct sched_entity *parent; ... #endif ... } 126 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 127 Chapter 2: Process Management and Scheduling This substructure of scheduling entities must be considered by all scheduling-class-related operations. Consider, for instance, how the code to enqueue a task in the completely fair scheduler really looks: kernel/sched_fair.c static void enqueue_task_fair(struct rq *rq, struct task_struct *p, int wakeup) { struct cfs_rq *cfs_rq; struct sched_entity *se = &p->se; for_each_sched_entity(se) { if (se->on_rq) break; cfs_rq = cfs_rq_of(se); enqueue_entity(cfs_rq, se, wakeup); wakeup = 1; } } for_each_sched_entity traverses the scheduling hierarchy defined by the parent elements of sched_entity, and each entity is enqueued on the run queue. Notice that for_each_sched_entity will resolve to a trivial loop that executes the code contained in the loop body exactly once when support for group scheduling is not selected, so the behavior described in the previous discussion is regained. 2.8.3 Kernel Preemption and Low Latency Efforts Let us now turn our attention to kernel preemption, which allows for a smoother experience of the sys- tem, especially in multimedia environments. Closely related are low latency efforts performed by the kernel, which I will discuss afterward. Kernel Preemption As described above, the scheduler is invoked before returning to user mode after system calls or at certain designated points in the kernel. This ensures that the kernel, unlike user processes, cannot be interrupted unless it explicitly wants to be. This behavior can be problematic if the kernel is in the middle of a relatively long operation — this may well be the case with filesystem, or memory-management- related tasks. The kernel is executing on behalf of a specific process for a long amount of time, and other processes do not get to run in the meantime. This may result in deteriorating system latency, which users experience as ‘‘sluggish‘‘ response. Video and audio dropouts may also occur in multimedia applications if they are denied CPU time for too long. These problems can be resolved by compiling the kernel with support for kernel preemption.Thisallows not only userspace applications but also the kernel to be interrupted if a high-priority process has some things to do. Keep in mind that kernel preemption and preemption of userland tasks by other userland tasks are two different concepts! Kernel preemption was added during the development of kernel 2.5. Although astonishingly few changes were required to make the kernel preemptible, the mechanism is not as easy to implement as preemption of tasks running in userspace. If the kernel cannot complete certain actions in a single operation — manipulation of data structures, for instance — race conditions may occur and render the system inconsistent. The same problems arise on multiprocessor systems discussed in Chapter 5. 127 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 128 Chapter 2: Process Management and Scheduling The kernel may not, therefore, be interrupted at all points. Fortunately, most of these points have already been identified by SMP implementation, and this information can be reused to implement kernel pre- emption. Problematic sections of the kernel that may only be accessed by one processor at a time are protected by so-called spinlocks: The first processor to arrive at a dangerous (also called critical) region acquires the lock, and releases the lock once the region is left again. Another processor that wants to access the region in the meantime has to wait until the first user has released the lock. Only then can it acquire the lock and enter the dangerous region. If the kernel can be preempted, even uniprocessor systems will behave like SMP systems. Consider that the kernel is working inside a critical region when it is preempted. The next task also operates in kernel mode, and unfortunately also wants to access the same critical region. This is effectively equivalent to two processors working in the critical region at the same time and must be prevented. Every time the kernel is inside a critical region, kernel preemption must be disabled. How does the kernel keep track of whether it can be preempted or not? Recall that each task in the system is equipped with an architecture-specific instance of struct thread_info. The structure also includes a preemption counter: struct thread_info { ... int preempt_count; /* 0 => preemptable, <0 => BUG */ ... } The value of this element determines whether the kernel is currently at a position where it may be inter- rupted. If preempt_count is zero, the kernel can be interrupted, otherwise not. The value must not be manipulated directly, but only with the auxiliary functions dec_preempt_count and inc_preempt_count, which, respectively, decrement and increment the counter. inc_preempt_count is invoked each time the kernel enters an important area where preemption is forbidden. When this area is exited, dec_ preempt_count decrements the value of the preemption counter by 1. Because the kernel can enter some important areas via different routes — particularly via nested routes — a simple Boolean variable would not be sufficient for preempt_count. When multiple dangerous regions are entered one after another, it must be made sure that all of them have been left before the kernel can be preempted again. The dec_preempt_count and inc_preempt_count calls are integrated in the synchronization opera- tions for SMP systems (see Chapter 5). They are, in any case, already present at all relevant points of the kernel so that the preemption mechanism can make best use of them simply by reusing the existing infrastructure. Some more routines are provided for preemption handling: ❑ preempt_disable disables preemption by calling inc_preempt_count. Additionally, the com- piler is instructed to avoid certain memory optimizations that could lead to problems with the preemption mechanism. ❑ preempt_check_resched checks if scheduling is necessary and does so if required. ❑ preempt_enable enables kernel preemption, and additionally checks afterward if rescheduling is necessary with preempt_check_resched. ❑ preempt_disable_no_resched disables preemption, but does not reschedule. 128 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 129 Chapter 2: Process Management and Scheduling At some points in the kernel, the protection by the normal SMP synchronization methods is not sufficient. This happens, for instance, when per-CPU variables are modified. On a real SMP system, this requires no form of protection because only one processor can by definition operate with the variable — every other CPU in the system has its own instance and does not need to fiddle with the instance of the current processor. However, kernel preemption would allow that two different code paths on the same processor would access the variable quasi-concurrently, which would have the same result as if two independent processors would manipulate the value. Preemption must therefore be explicitly disabled in these situations using manual incovations of preempt_disable and preempt_disable. Note, however, that the get_cpu and put_cpu functions mentioned in the Introduction will automatically disable kernel preemption, so no extra precautions are necessary if per-CPU variables are accessed using this mechanism. How does the kernel know if preemption is required? First of all, the TIF_NEED_RESCHED flag must be set to signalize that a process is waiting to get CPU time. This is honored by preempt_ check_resched: #define preempt_check_resched() \ do { \ if (unlikely(test_thread_flag(TIF_NEED_RESCHED))) \ preempt_schedule(); \ } while (0) Recall that the function is called when preemption is re-enabled after it had been disabled, so this is a good time to check if a process wants to preempt the currently executing kernel code. If this is the case, it should be done as quickly as possible — without waiting for the next routine call of the scheduler. The central function for the preemption mechanism is preempt_schedule. The simple desire that the ker- nel be preempted as indicated by TIF_NEED_RESCHED does not yet guarantee that this is possible —recall that the kernel could currently still be inside a critical region, and must not be disturbed. This is checked by preempt_reschedule: kernel/sched.c asmlinkage void __sched preempt_schedule(void) { struct thread_info *ti = current_thread_info(); /* * If there is a non-zero preempt_count or interrupts are disabled, * we do not want to preempt the current task. Just return.. */ if (unlikely(ti->preempt_count || irqs_disabled())) return; ... If the preemption counter is greater than 0, then preemption is still disabled, and consequently the kernel may not be interrupted — the function terminates immediately. Neither is preemption possible if the kernel has disabled hardware IRQs at important points where processing must be completed in a single operation. irqs_disabled checks whether interrupts are disabled or not, and if they are disabled, the kernel must not be preempted. 129 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 130 Chapter 2: Process Management and Scheduling The following steps are required if preemption is possible: kernel/sched.c do { add_preempt_count(PREEMPT_ACTIVE); schedule(); sub_preempt_count(PREEMPT_ACTIVE); /* * Check again in case we missed a preemption opportunity * between schedule and now. */ } while (unlikely(test_thread_flag(TIF_NEED_RESCHED))); Before the scheduler is invoked, the value of the preemption counter is set to PREEMPT_ACTIVE.Thissets a flag bit in the preemption counter that has such a large value that it is never affected by the regular preemption counter increments as illustrated by Figure 2-30. It indicates to the schedule function that scheduling was not invoked in the normal way but as a result of a kernel preemption. After the kernel has rescheduled, code flow returns to the current task — possibly after some time has elapsed, because the preempting task will have run in between — the flag bit is removed again. Preemption counter PREEMPT_ACTIVE Figure 2-30: The per-process preemption counter. I ignored the implications of this flag for schedule before, so I have to discuss it now. Recall that the scheduler deactivates a task with deactivate_task if it is not in a runnable state at the moment. In fact, this operation is skipped if the scheduling was initiated via the preemption mechanism as can be seen if PREEMPT_ACTIVE is set in the preemption counter: kernel/sched.c asmlinkage void __sched schedule(void) { ... if (prev->state && !(preempt_count() & PREEMPT_ACTIVE)) { if (unlikely((prev->state & TASK_INTERRUPTIBLE) && unlikely(signal_pending(prev)))) { prev->state = TASK_RUNNING; } else { deactivate_task(rq, prev, 1); } } ... } This ensures that the next task is selected as quickly as possible without the hassle of deactivating the current one. If a high-priority task is waiting to be scheduled, it will be picked by the scheduler class and will be allowed to run. 130 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 131 Chapter 2: Process Management and Scheduling This method is only one way of triggering kernel preemption. Another possibility to activate preemption is after a hardware IRQ has been serviced. If the processor returns to kernel mode after handling the IRQ (return to user mode is not affected), the architecture-specific assembler routine checks whether the value of the preemption counter is 0 — that is, if preemption is allowed — and whether the reschedule flag is set — exactly as in preempt_schedule. If both conditions are satisfied, the scheduler is invoked, this time via preempt_schedule_irq to indicate that the preemption request originated from IRQ context. The essential difference between this function and preempt_schedule is that preempt_schedule_irq is called with IRQs disabled to prevent recursive calls for simultaneous IRQs. As a result of the methods described in this section, a kernel with enabled preemption is able to replace processes with more urgent ones faster than a normal kernel could. Low Latency Naturally, the kernel is interested in providing good latency times even if kernel preemption is not enabled. This can, for instance, be important in network servers. While the overhead introduced by kernel preemption is not desired in such an environment, the kernel should nevertheless respond to important events with reasonable speed. If, for example, a network request comes in that needs to be serviced by a daemon, then this should not be overly long delayed by some database doing heavy I/O operations. I have already discussed a number of measures offered by the kernel to reduce this problem: scheduling latency in CFS and kernel preemption. Real-time mutexes as discussed in Chapter 5 also aid in solving the problem, but there is one more scheduling-related action that can help. Basically, long operations in the kernel should not occupy the system completely. Instead, they should check from time to time if another process has become ready to run, and thus call the scheduler to select the process. This mechanism is independent of kernel preemption and will reduce latency also if the kernel is built without explicit preemption support. The function to initiate conditional rescheduling is cond_resched. It is implemented as follows: kernel/sched.c int __sched cond_resched(void) { if (need_resched() && !(preempt_count() & PREEMPT_ACTIVE)) __cond_resched(); return 1; } return 0; } need_resched checks if the TIF_NEED_RESCHED flag is set, and the code additionally ensures that the ker- nel is not currently being preempted already34 and rescheduling is thus allowed. Should both conditions be fulfilled, then __cond_resched takes care of the necessary details to invoke the scheduler. How can cond_resched be used? As an example, consider the case in which the kernel reads in memory pages associated with a given memory mapping. This could be done in an endless loop that terminates after all required data have been read: for (;;) /* Read in data */ if (exit_condition) continue; 34Additionally, the function also makes sure that the system is completely up and running, which is, for instance, not the case if the system has not finished booting yet. Since this is an unimportant corner case, I have omitted the corresponding check. 131 Mauerer runc02.tex V3 - 09/04/2008 4:15pm Page 132 Chapter 2: Process Management and Scheduling If a large number of read operations is required, this can consume a sizeable amount of time. Since the process runs in kernel space, it will not be deselected by the scheduler as in the userspace case, taken that kernel preemption is not enabled. This can be improved by calling cond_resched in every loop iteration: for (;;) cond_resched(); /* Read in data */ if (exit_condition) continue; The kernel has been carefully audited to find the longest-running functions, and calls to cond_resched have been put in the appropriate places. This ensures higher responsiveness even without explicit kernel preemption. Following a long-time tradition for Unix kernels, Linux has supported task states for both interruptible and uninterruptible sleeps. During the 2.6.25 development cycle, however, another state was added: TASK_KILLABLE.35 Tasks in this state are sleeping and do not react to non-fatal signals, but can — in contrast to TASK_UNINTERRUPTIBLE — be killed by fatal signals. At the time of writing, almost all places in the kernel that would provide apt possibilities for killable sleeps are still waiting to be converted to the new form. The scheduler has seen a comparatively large number of cleanups during the development of kernels 2.6.25 and 2.6.26. A new feature added during this period is real-time group scheduling. This means that real-time tasks can now also be handled by the group scheduling framework introduced in this chapter. Additionally, the scheduler documentation was moved into the dedicated directory Documentation/ scheduler/, and obsolete files documenting the old O(1) scheduler have been removed. Documentation on real-time group scheduling can be found in Documentation/scheduler/sched-rt-group.txt. 2.9 Summary Linux is a multiuser and multitasking operating system, and thus has to manage multiple processes from multiple users. In this chapter, you have learned that processes are a very important and fundamental abstraction of Linux. The data structure used to represent individual processes has connections with nearly every subsystem of the kernel. You have seen how Linux implements the traditional fork/exec model inherited from Unix to create new processes that are hierarchically related to their parent, and have also been introduced to Linux- specific extensions to the traditional Unix model in the form of namespaces and the clone system call. Both allow for fine-tuning how a process perceives the system, and which resources are shared between parent and child processes. Explicit methods that enable otherwise separated processes to communicate are discussed in Chapter 5. Additionally, you have seen how the available computational resources are distributed between pro- cesses by the scheduler. Linux supports pluggable scheduling modules, and these are used to implement completely fair and POSIX soft real-time scheduling policies. The scheduler decides when to switch between which tasks, and is augmented by architecture-specific routines to implement the context switch- ing proper. Finally, I have discussed how the scheduler must be augmented to service systems with multiple CPUs, and how kernel preemption and low-latency modifications make Linux handle time-constrained situa- tions better. 35Actually, TASK_KILLABLE is not a completely new task state, but an extension to TASK_UNINTERRUPTIBLE.Theeffectis,how- ever, identical. 132 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 133 Memory Management Memory management is one of the most complex and at the same time most important parts of the kernel. It is characterized by the strong need for cooperation between the processor and the kernel because the tasks to be performed require them to collaborate very closely. Chapter 1 provided a brief overview of the various techniques and abstractions used by the kernel in the implementation of memory management. This chapter examines the technical aspects of implementation in detail. 3.1 Overview Memory management implementation covers many areas: ❑ Management of physical pages in memory. ❑ The buddy system to allocate memory in large chunks. ❑ The slab, slub, and slob allocators to allocate smaller chunks of memory. ❑ The vmalloc mechanism to allocate non-contiguous blocks of memory. ❑ The address space of processes. As we know, the virtual address space of the processor is in general divided into two parts by the Linux kernel. The lower and larger part is available to user processes, and the upper part is reserved for the kernel. Whereas the lower part is modified during a context switch (between two user processes), the kernel part of virtual address space always remains the same. On IA-32 systems, the address space is typically divided between user processes and the kernel in a ratio of 3 : 1; given a virtual address space of 4 GiB, 3 GiB would be available to userspace and 1 GiB for the kernel. This ratio can be changed by modifying the relevant configuration options. However, this has advantages only for very specific configurations and applications. For purposes of our further investigations, I assume a ratio of 3 : 1 for now, but will come back to different ratios later. The available physical memory is mapped into the address space of the kernel. Accesses with virtual addresses whose offset to the start of the kernel area does not exceed the size of the available RAM are therefore automatically associated with physical page frames. This is practical because memory Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 134 Chapter 3: Memory Management allocations in the kernel area always land in physical RAM when this scheme is adopted. However, there is one problem. The virtual address space portion of the kernel is necessarily smaller than the maximum theoretical address space of the CPU. If there is more physical RAM than can be mapped into the kernel address space, the kernel must resort to the highmem method to manage ‘‘super fluous‘‘ memory. On IA-32 systems, up to 896 MiB of RAM can be managed directly; anything above this figure (up to a maximum of 4 GiB) can only be addressed by means of highmem. 4 GiB is the maximum memory size that can be addressed on 32-bit systems (232 = 4 GiB). If a trick is used, modern IA-32 implementations — Pentium PRO and higher — can manage up to 64 GiB of memory if PAE mode is enabled. PAE stands for page address extension and provides additional bits for memory pointers. However, not all 64 GiB can be addressed at the same time, only sections of 4 GiB each. Because most memory management data structures can only be allocated in the range between 0 and 1 GiB, there is a practical limit to the maximum memory size and this is less than 64 GiB. The exact value varies according to kernel configuration. For example, it is possible to allocate third-level page table entries in highmem to reduce the load on the normal zone. Because IA-32 systems with memory in excess of 4 GiB are a rarity and the 64-bit architecture AMD64 that has for all practical purposes replaced IA-32 offers a much cleaner solution to this problem, I won’t bother discussing the second highmem mode here. Highmem mode is not required on 64-bit machines because the available address space is gigantic, even if physical addressing is limited to a smaller number of bits, for example, 48 or 52. Given that exactly the same was thought of the 4-GiB address space on 32-bit systems just a few years ago, one could argue that it would merely seem to be a matter of time before the limits of 64-bit systems are reached, although 16 EiB should suffice for some time. But you never know ... . The use of highmem pages is problematic only for the kernel itself. The kernel must first invoke the kmap and kunmap functions discussed below to map the highmem pages into its virtual address space before it can use them — this is not necessary with normal memory pages. However, for userspace processes, it makes absolutely no difference if the pages are highmem or normal pages because they are always accessed via page tables and never directly. There are two types of machine that manage physical memory in different ways: 1. UMA machines (uniform memory access) organize available memory in a contiguous fashion (possibly with small gaps). Each processor (in a symmetric multiprocessor system) is able to access each memory area equally quickly. 2. NUMA machines (non-uniform memory access) are always multiprocessor machines. Local RAM is available to each CPU of the system to support particularly fast access. The proces- sors are linked via a bus to support access to the local RAM of other CPUs — this is naturally slower than accessing local RAM. Examples of such systems are Alpha-based WildFire servers and NUMA-Q machines from IBM. 134 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 135 Chapter 3: Memory Management Figure 3-1 illustrates the difference between the two approaches. Memory CPU 0 CPU 1 CPU 3 Memory CPU 0 CPU 1 CPU 3 Memory Memory UMA NUMA Figure 3-1: UMA and NUMA systems. A mix of both machine types with discontiguous memory is also possible. Such a mix would then represent a UMA system whose RAM is not contiguous but has large holes. Here it is often helpful to apply the principles of NUMA organization to make memory access simpler for the kernel. In fact, the kernel distinguishes three configuration options — FLATMEM, DISCONTIGMEM,andSPARSEMEM. SPARSEMEM and DISCONTIGMEM serve practically the same purpose, but in the view of developers, differ in the quality of their code — SPARSEMEM is regarded as more experimental and less stable but does feature performance optimizations. Discontiguous memory is presumed to be more stable, but is not prepared for new features like memory hotplugging. In the following sections, we restrict ourselves largely to FLATMEM because this memory organization type is used on most configurations and is also usually the kernel default. The fact that we do not discuss the other options is no great loss because all memory models make use of practically the same data structures. Real NUMA systems will set the configuration option CONFIG_NUMA, and the memory management codes will differ between the two variants. Since the flat memory model will not make sense on NUMA machines, only discontiguous and sparse memory will be available. Notice that the configuration option NUMA_EMU allows AMD64 systems with a flat memory to enjoy the full complexities of NUMA systems by splitting the memory into fake NUMA zones. This can be useful for development when no real NUMA machine is available — for some reason, these tend to be rather costly. This book focuses on the UMA case, and does not consider CONFIG_NUMA.Thisdoesnotmean that the NUMA data structures can be completely neglected. Since UMA systems can choose the configuration option CONFIG_DISCONTIGMEM if their address space contains large holes, then more than one memory node can also be available on systems that do not employ NUMA techniques otherwise. Figure 3-2 summarizes the various possible choices for the configuration options related to memory layout. Notice that we will come across the term allocation order quite often in the following discussion. It denotes the binary logarithm of the number of pages that are contained in a memory region. An order 0 allocation consists of one page, an order two allocation of 21 = 2 pages, an order three allocation of 22 = 4pages, and so on. 135 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 136 Chapter 3: Memory Management Sparse memory NUMA UMA Discontiguous memoryFlat memory NUMA UMA Address space without holes Figure 3-2: Overview of possible memory setups for flat, sparse, and discontiguous memory on UMA and NUMA machines. 3.2 Organization in the (N)UMA Model The various architectures supported differ greatly in terms of how they manage memory. Owing to the intelligent design of the kernel and, in some cases, intervening compatibility layers, these differences are so well concealed that generic code can usually ignore them. As discussed in Chapter 1, a major issue is the varying number of indirection levels for page tables. A second key aspect is the division into NUMA and UMA systems. The kernel uses identical data structures for machines with uniform and non-uniform memory access so that the individual algorithms need make little or no distinction between the various forms of memory arrangement. On UMA systems, a single NUMA node is introduced to help manage the entire system memory. The other parts of memory management are led to believe that they are working with a pseudo- NUMA system. 3.2.1 Overview Before we look at the data structures used to organize memory in the kernel, we need to define a few concepts because the terminology is not always easy to understand. Let’s first consider NUMA systems. This will enable us to show that it is very easy to reduce them to UMA systems. Figure 3-3 is a graphic illustration of the memory partitioning described below (the situation is somewhat simplified, as you will see when we examine the data structures closely). First, RAM memory is divided into nodes. A node is associated with each processor of the system and is represented in the kernel by an instance of pg_data_t (these data structures are defined shortly). Each node is split into zones as further subdivisions of memory. For example, there are restrictions as to the memory area that can be used for DMA operations (with ISA devices); only the first 16 MiB are suitable. There is also a highmem area that cannot be mapped directly. Between these is the ‘‘normal‘‘ memory area for universal use. A node therefore comprises up to three zones. The kernel introduces the following constants to distinguish between them. 136 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 137 Chapter 3: Memory Management pg_data_t ZONELIST Z O N E S pg_data_t ZONELIST Z O N E S pg_data_t ZONELIST Z O N E S struct page Figure 3-3: Memory partitioning in NUMA systems. The kernel introduces the following constants to enumerate all zones in the system: enum zone_type { #ifdef CONFIG_ZONE_DMA ZONE_DMA, #endif #ifdef CONFIG_ZONE_DMA32 ZONE_DMA32, #endif ZONE_NORMAL, #ifdef CONFIG_HIGHMEM ZONE_HIGHMEM, #endif ZONE_MOVABLE, MAX_NR_ZONES }; ❑ ZONE_DMA for DMA-suitable memory. The size of this region depends on the processor type. ON IA-32 machines, the limit is the classical 16 MiB boundary imposed by ancient ISA devices. But also, more modern machines can be affected by this. ❑ ZONE_DMA32 for DMA-suitable memory in a 32-bit addressable area. Obviously, there is only a difference between the two DMA alternatives on 64-bit systems. On 32-bit machines, this zone is empty; that is, its size is 0 MiB. On Alphas and AMD64 systems, for instance, this zone ranges from 0 to 4 GiB. ❑ ZONE_NORMAL for normal memory mapped directly in the kernel segment. This is the only zone guaranteed to be possible present on all architectures. It is, however, not guaranteed that the zone must be equipped with memory. If, for instance, an AMD64 system has 2 GiB of RAM, then all of it will belong to ZONE_DMA32,andZONE_NORMAL will be empty. ❑ ZONE_HIGHMEM for physical memory that extends beyond the kernel segment. 137 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 138 Chapter 3: Memory Management Depending on the compile-time configuration, some zones need not be considered. 64-bit systems, for instance, do not require a high memory zone, and the DMA32 zone is only required on 64-bit systems that also support 32-bit peripheral devices that can only access memory up to 4 GiB. The kernel additionally defines a pseudo-zone ZONE_MOVABLE, which is required when efforts are made to prevent fragmentation of the physical memory. We will look closer into this mechanism in Section 3.5.2. MAX_NR_ZONES acts as an end marker if the kernel wants to iterate over all zones present in the system. Each zone is associated with an array in which the physical memory pages belonging to the zone — known as page frames in the kernel — are organized. An instance of struct page with the required management data is allocated for each page frame. The nodes are kept on a singly linked list so that the kernel can traverse them. For performance reasons, the kernel always attempts to perform the memory allocations of a process on the NUMA node associated with the CPU on which it is currently running. However, this is not always possible — for example, the node may already be full. For such situations, each node provides a fallback list (with the help of struct zonelist). The list contains other nodes (and associated zones) that can be used as alternatives for memory allocation. The further back an entry is on the list, the less suitable it is. What’s the situation on UMA systems? Here, there is just a single node — no others. This node is shown against a gray background in the figure. Everything else is unchanged. 3.2.2 Data Structures Now that I have explained the relationship between the various data structures used in memory man- agement, let’s look at the definition of each. Node Management pg_data_t is the base element used to represent a node and is defined as follows: typedef struct pglist_data { struct zone node_zones[MAX_NR_ZONES]; struct zonelist node_zonelists[MAX_ZONELISTS]; int nr_zones; struct page *node_mem_map; struct bootmem_data *bdata; unsigned long node_start_pfn; unsigned long node_present_pages; /* total number of physical pages */ unsigned long node_spanned_pages; /* total size of physical page range, including holes */ int node_id; struct pglist_data *pgdat_next; wait_queue_head_t kswapd_wait; struct task_struct *kswapd; int kswapd_max_order; } pg_data_t; 138 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 139 Chapter 3: Memory Management ❑ node_zones is an array that holds the data structures of the zones in the node. ❑ node_zonelists specifies alternative nodes and their zones in the order in which they are used for memory allocation if no more space is available in the current zone. ❑ The number of different zones in the node is held in nr_zones. ❑ node_mem_map is a pointer to an array of page instances used to describe all physical pages of the node. It includes the pages of all zones in the node. ❑ During system boot, the kernel needs memory even before memory management has been ini- tialized (memory must also be reserved to initialize memory management). To resolve this prob- lem, the kernel uses the boot memory allocator described in Section 3.4.3. bdata points to the instance of the data structure that characterizes the boot memory allocator. ❑ node_start_pfn is the logical number of the first page frame of the NUMA node. The page frames of all nodes in the system are numbered consecutively, and each frame is given a number that is globally unique (not just unique to the node). node_start_pfn is always 0 in a UMA system because there is only one node whose first page frame is therefore 0. node_present_pages specifies the number of page frames in the zone and node_spanned_pages the size of the zone in page frames. This value need not necessarily be the same as node_present_pages becausetheremaybeholesinthezonethatarenotbackedbya real page frame. ❑ node_id is a global node identifier. All NUMA nodes in the system are numbered starting from 0. ❑ pgdat_next links the nodes in the system on a singly linked list whose end is indicated, as usual, by a null pointer. ❑ kswapd_wait is the wait queue for the swap daemon needed when swapping frames out of the zone (Chapter 18 deals with this at length). kswapd points to the task structure of the swap dae- mon responsible for the zone. kswapd_max_order is used in the implementation of the swapping subsystem to define the size of the area to be freed and is currently of no interest. The association between the node and the zones it contains and the fallback list shown in Figure 3-3 is established by means of arrays at the beginning of the data structure. These are not the usual pointers to arrays. The array data are held in the node struc- ture itself. The zones of the node are held in node_zones[MAX_NR_ZONES]. The array always has three entries, even if the node has fewer zones. If the latter is the case, the remaining entries are filled with null elements. Node State Management If more than one node can be present on the system, the kernel keeps a bitmap that provides state infor- mation for each node. The states are specified with a bitmask, and the following values are possible: enum node_states { N_POSSIBLE, /* The node could become online at some point */ N_ONLINE, /* The node is online */ N_NORMAL_MEMORY, /* The node has regular memory */ #ifdef CONFIG_HIGHMEM 139 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 140 Chapter 3: Memory Management N_HIGH_MEMORY, /* The node has regular or high memory */ #else N_HIGH_MEMORY = N_NORMAL_MEMORY, #endif N_CPU, /* The node has one or more cpus */ NR_NODE_STATES }; The states N_POSSIBLE, N_ONLINE,andN_CPU are required for CPU and memory hotplugging, but these features are not considered in this book. Essential for memory management are the flags N_HIGH_MEMORY and N_NORMAL_MEMORY. While the first one announces that the zone is equipped with memory that may be either regular or high memory, N_NORMAL_MEMORY is only set if non-highmem memory is present on a node. Two auxiliary functions are provided to set or clear, respectively, a bit in the bit-field or a specific node: void node_set_state(int node, enum node_states state) void node_clear_state(int node, enum node_states state) Additionally, the macro for_each_node_state(__node, __state) allows for iterating over all nodes that are in a specific state, and for_each_online_node(node) iterates over all active nodes. If the kernel is compiled to support only a single node, that is, using the flat memory model, the node bitmap is not present, and the functions to manipulate it resolve to empty operations that simply do nothing. Memory Zones The kernel uses the zones structure to describe a zone. It is defined as follows: struct zone { /* Fields commonly accessed by the page allocator */ unsigned long pages_min, pages_low, pages_high; unsigned long lowmem_reserve[MAX_NR_ZONES]; struct per_cpu_pageset pageset[NR_CPUS]; /* * free areas of different sizes */ spinlock_t lock; struct free_area free_area[MAX_ORDER]; ZONE_PADDING(_pad1_) /* Fields commonly accessed by the page reclaim scanner */ spinlock_t lru_lock; struct list_head active_list; struct list_head inactive_list; unsigned long nr_scan_active; unsigned long nr_scan_inactive; unsigned long pages_scanned; /* since last reclaim */ 140 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 141 Chapter 3: Memory Management unsigned long flags; /* zone flags, see below */ /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; int prev_priority; ZONE_PADDING(_pad2_) /* Rarely used or read-mostly fields */ wait_queue_head_t * wait_table; unsigned long wait_table_hash_nr_entries; unsigned long wait_table_bits; /* Discontig memory support fields. */ struct pglist_data *zone_pgdat; unsigned long zone_start_pfn; unsigned long spanned_pages; /* total size, including holes */ unsigned long present_pages; /* amount of memory (excluding holes) */ /* * rarely used fields: */ char *name; } ____cacheline_maxaligned_in_smp; The striking aspect of this structure is that it is divided into several sections separated by ZONE_PADDING. This is because zone structures are very frequently accessed. On multiprocessor systems, it commonly occurs that different CPUs try to access structure elements at the same time. Locks (examined in Chapter 5) are therefore used to prevent them interfering with each, and giving rise to errors and inconsistencies. The two spinlocks of the structure — zone->lock and zone->lru_lock —areoften acquired because the kernel very frequently accesses the structure.1 Data are processed faster they are is held in a cache of the CPU. Caches are divided into lines, and each line is responsible for various memory areas. The kernel invokes the ZONE_PADDING macro to generate ‘‘padding‘‘ that is added to the structure to ensure that each lock is in its own cache line. The compiler keyword __cacheline_maxaligned_in_smp is also used to achieve optimal cache alignment. The last two sections of the structure are also separated from each other by padding. As neither includes a lock, the primary aim is to keep the data in a cache line for quick access and thus to dispense with the need for loading the data from RAM memory, which is a slow process. The increase in size due to the padding structures is negligible, particularly as there are relatively few instances of zone structures in kernel memory. What is the meaning of the structure elements? Since memory management is a complex and comprehen- sive part of the kernel, it is not possible to cover the exact meaning of all elements at this point — a good part of this and of following chapters will be devoted to understanding the associated data structures and mechanisms. What I can provide, however, is an overview that gives a taste of the problems I am about to discuss. A large number of forward references is nevertheless unavoidable. 1Thelocksarethereforeknownashotspots. In Chapter 17, some tricks that are used by the kernel to reduce the pressure on these hotspots are discussed. 141 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 142 Chapter 3: Memory Management ❑ pages_min, pages_high,andpages_low are ‘‘watermarks‘‘used when pages are swapped out. The kernel can write pages to hard disk if insufficient RAM memory is available. These three elements influence the behavior of the swapping daemon. ❑ If more than pages_high pages are free, the state of the zone is ideal. ❑ If the number of free pages falls below pages_low, the kernel begins to swap pages out onto the hard disk. ❑ If the number of free pages falls below pages_min, the pressure to reclaim pages is increased because free pages are urgently needed in the zone. Chapter 18 will discuss various means of the kernel to find relief. The importance of these watermarks will mainly show in Chapter 18, but they also come into play in Section 3.5.5. ❑ The lowmem_reserve array specifies several pages for each memory zone that are reserved for critical allocations that must not fail under any circumstances. Each zone contributes accord- ing to its importance. The algorithm to calculate the individual contributions is discussed in Section 3.2.2. ❑ pageset is an array to implement per-CPU hot-n-cold page lists. The kernel uses these lists to store fresh pages that can be used to satisfy implementations. However, they are distinguished by their cache status: Pages that are most likely still cache-hot and can therefore be quickly accessed are separated from cache-cold pages. The next section discusses the struct per_ cpu_pageset data structure used to realize this behavior. ❑ free_area is an array of data structures of the same name used to implement the buddy system. Each array element stands for contiguous memory areas of a fixed size. Management of free memory pages contained in each area is performed starting from free_area. The employed data structures merit a discussion of their own, and Section 3.5.5 covers the imple- mentation details of the buddy system in depth. ❑ The elements of the second section are responsible for cataloging the pages used in the zone according to activity. A page is regarded as active by the kernel if it is accessed frequently; an inactive page is obviously the opposite. This distinction is important when pages need to be swapped out. If possible, frequently used pages should be left intact, but superfluous inactive pages can be swapped out without impunity. The following elements are involved: ❑ active_list collects the active pages, and inactive_list the inactive pages (page instances). ❑ nr_scan_active and nr_scan_inactive specify how many active and inactive pages are to be scanned when reclaiming memory. ❑ pages_scanned specifies how many pages were unsuccessfully scanned since the last time a page was swapped out. ❑ flags describes the current status of the zone. The following flags are allowed: typedef enum { ZONE_ALL_UNRECLAIMABLE, /* all pages pinned */ ZONE_RECLAIM_LOCKED, /* prevents concurrent reclaim */ 142 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 143 Chapter 3: Memory Management ZONE_OOM_LOCKED, /* zone is in OOM killer zonelist */ } zone_flags_t; It is also possible that none of these flags is set. This is the normal state of the zone. ZONE_ALL_UNRECLAIMABLE is a state that can occur when the kernel tries to reuse some pages of the zone (page reclaim, see Chapter 18), but this is not possible at all because all pages are pinned. For instance, a userspace application could have used the mlock system call to instruct the kernel that pages must not be removed from physical memory, for example, by swapping them out. Such a page is said to be pinned. If all pages in a zone suffer this fate, the zone is unreclaimable, and the flag is set. To waste no time, the swapping daemon scans zones of this kind very briefly when it is looking for pages to reclaim.2 On SMP systems, multiple CPUs could be tempted to reclaim a zone concurrently. The flag ZONE_RECLAIM_LOCKED prevents this: If A CPU is reclaiming a zone, it set the flag. This prevents other CPUs from trying. ZONE_OOM_LOCKED is reserved for an unfortunate situation: If processes use up so much memory that essential operations cannot be completed anymore, then the kernel will try to select the worst memory eater and kill it to obtain more free pages. The flag prevents multiple CPUs from getting into their way in this case. The kernel provides three auxiliary functions to test and set zone flags: void zone_set_flag(struct zone *zone, zone_flags_t flag) int zone_test_and_set_flag(struct zone *zone, zone_flags_t flag) void zone_clear_flag(struct zone *zone, zone_flags_t flag) zone_set_flag and zone_clear_flag set and clear a certain flag, respectively. zone_test_ and_set_flag first tests if a given flag is set and does so if not. The old state of the flag is returned to the caller. ❑ vm_stat keeps a plethora of statistical information about the zone. Since most of the infor- mation kept in there will not make much sense at the moment, a detailed discussion is deferred to Section 17.7.1. For now, it suffices to know that the information is updated from places all over the kernel. The auxiliary function zone_page_state allows for reading the information in vm_stat: static inline unsigned long zone_page_state(struct zone *zone, enum zone_stat_item item) item can, for instance, be NR_ACTIVE or NR_INACTIVE to query the number of active and inactive pages stored on active_list and inactive_list discussed above. The number of free pages in the zone is obtained with NR_FREE_PAGES. ❑ prev_priority stores the priority with which the zone was scanned in the last scan oper- ation until sufficient page frames were freed in try_to_free_pages (see Chapter 17). As you shall also see in Chapter 17, the decision as to whether mapped pages are swapped out depends on this value. 2However, scanning cannot be totally dispensed with because the zone may contain reclaimable pages again at some time in the future. If so, the flag is removed and the kswapd daemon treats the zone again like any other zone. 143 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 144 Chapter 3: Memory Management ❑ wait_table, wait_table_bits,andwait_table_hash_nr_entries implement a wait queue for processes waiting for a page to become available. While the details of this mechanism are shown in Chapter 14, the intuitive notion holds pretty well: Processes queue up in a line to wait for some condition. When this condition becomes true, they are notified by the kernel and can resume their work. ❑ The association between a zone and the parent node is established by zone_pgdat,whichpoints to the corresponding instance of pg_list_data. ❑ zone_start_pfn is the index of the first page frame of the zone. ❑ The remaining three fields are rarely used, so they’ve been placed at the end of the data struc- ture. name is a string that holds a conventional name for the zone. Three options are available at present: Normal, DMA,andHighMem. spanned_pages specifies the total number of pages in the zone. However, not all need be usable since there may be small holes in the zone as already mentioned. A further counter (present_ pages) therefore indicates the number of pages that are actually usable. Generally, the value of this counter is the same as that for spanned_pages. Calculation of Zone Watermarks Before calculating the various watermarks, the kernel first determines the minimum memory space that must remain free for critical allocations. This value scales nonlinearly with the size of the available RAM. It is stored in the global variable min_free_kbytes. Figure 3-4 provides an overview of the scaling behavior, and the inset — which does not use a logarithmic scale for the main memory size in contrast to the main graph — shows a magnification of the region up to 4 GiB. Some exemplary values to provide a feeling for the situation on systems with modest memory that are common in desktop environments are collected in Table 3-1. An invariant is that not less than 128 KiB but not more than 64 MiB may be used. Note, however, that the upper bound is only necessary on machines equipped with a really satisfactory amount of main memory.3 The file /proc/sys/vm/min_free_kbytes allows reading and adapting the value from userland. Filling the watermarks in the data structure is handled by init_per_zone_pages_min, which is invoked during kernel boot and need not be started explicitly.4 setup_per_zone_pages_min sets the pages_min, pages_low,andpages_high elements of struct zone. After the total number of pages outside the highmem zone has been calculated (and stored in lowmem_ pages), the kernel iterates over all zones in the system and performs the following calculation: mm/page_alloc.c void setup_per_zone_pages_min(void) { unsigned long pages_min = min_free_kbytes >> (PAGE_SHIFT - 10); unsigned long lowmem_pages = 0; struct zone *zone; unsigned long flags; 3In practice, it will be unlikely that such an amount of memory is installed on a machine with a single NUMA node, so it will be hard to actually reach the point where the cutoff is required. 4The functions are not only called from here, but are also invoked each time one of the control parameters is modified via the proc filesystem. 144 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 145 Chapter 3: Memory Management ... for_each_zone(zone) { u64 tmp; tmp = (u64)pages_min * zone->present_pages; do_div(tmp,lowmem_pages); if (is_highmem(zone)) { int min_pages; min_pages = zone->present_pages / 1024; if (min_pages < SWAP_CLUSTER_MAX) min_pages = SWAP_CLUSTER_MAX; if (min_pages > 128) min_pages = 128; zone->pages_min = min_pages; } else { zone->pages_min = tmp; } zone->pages_low = zone->pages_min + (tmp >> 2); zone->pages_high = zone->pages_min + (tmp >> 1); } } 0 5000 10000 15000 20000 25000 1 10 100 1000 Pages of 4 KiB Zone memory [GiB] pages_low pages_high pages_min 0 500 1000 1500 2000 2500 3000 3500 0 500 1000 1500 2000 2500 3000 3500 4000 Zone memory [MiB] Figure 3-4: Minimum memory size for critical allocations and zone watermarks depending on the main memory size of a machine (pages_min is nothing other than min_free_kbytes in units of pages). 145 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 146 Chapter 3: Memory Management Table 3-1: Correlation between Main Memory Size and Minimum Memory Available for Critical Allocations. Main memory Reserve 16 MiB 512 KiB 32 MiB 724 KiB 64 MiB 1024 KiB 128 MiB 1448 KiB 256 MiB 2048 KiB 512 MiB 2896 KiB 1024 MiB 4096 KiB 2048 MiB 5792 KiB 4096 MiB 8192 KiB 8192 MiB 11584 KiB 16384 MiB 16384 KiB init_per_zone_pages_min setup_per_zone_pages_min setup_per_zone_lowmem_reserve Figure 3-5: Code flow diagram for init_per_zone_pages_min. The lower bound for highmem zones, SWAP_CLUSTER_MAX, is an important quantity for the whole page reclaim subsystem as discussed in Chapter 17. The code there often operates batchwise on page clusters, and SWAP_CLUSTER_MAX defines the size of such clusters. Figure 3-4 shows the outcome of the calcula- tions for various main memory sizes. Since high memory is not very relevant anymore these days (most machines with large amounts of RAM use 64-bit CPUs), I have restricted the graph to show the outcomes for regular zones. Computing lowmem_reserve is done in setup_per_zone_lowmem_reserve. The kernel iterates over all nodes of the system and calculates the minimum reserve for each zone of the node by dividing the total number of page frames in the zone by sysctl_lowmem_reserve_ratio[zone]. The default settings for the divisor are 256 for low memory and 32 for high memory. Hot-N-Cold Pages The pageset element of struct zone is used to implement a hot-n-cold allocator. The kernel refers to apageinmemoryashot if it is in a CPU cache and its data can therefore be accessed quicker than if 146 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 147 Chapter 3: Memory Management it were in RAM. Conversely, a cold page is not held in cache. As each CPU has one or more caches on multiprocessor systems, management must be separate for each CPU. Even though a zone belongs to a specific NUMA node and is therefore associated with a specific CPU, the caches of other CPUs may include pages from this zone — ultimately, each processor can access all pages in the system, albeit at different speeds. The zone-specific data structure must therefore cater not only for the CPU associated with the NUMA node of the zone but also for all other CPUs in the system. pageset is an array that holds as many entries as the maximum possible number of CPUs that the system can accommodate. struct zone { ... struct per_cpu_pageset pageset[NR_CPUS]; ... }; NR_CPUS is a configurable pre-processor constant defined at compilation time. Its value is always 1 on uniprocessor systems, but on a kernel compiled for SMP systems, it may be between 2 and 32 (or 64 on 64-bit systems). The value does not reflect the number of CPUs actually present in a system but the maximum number of CPUs supported by the kernel. The array elements are of type per_cpu_pageset, which is defined as follows: struct per_cpu_pageset { struct per_cpu_pages pcp[2]; /* 0: hot. 1: cold */ } ____cacheline_aligned_in_smp; The structure consists of an array with two entries, the first to manage hot and the second to manage cold pages. The useful data are held in per_cpu_pages.5 struct per_cpu_pages { int count; /* number of pages in the list */ int high; /* high watermark, emptying needed */ int batch; /* chunk size for buddy add/remove */ struct list_head list; /* the list of pages */ }; 5Kernel 2.6.25, which was still under development when this book was written, will replace the separate lists for hot and cold pages by a single list. Hot pages will be kept at the beginning, while cold pages will be placed at the end. The change was introduced after measurements had shown that having two separate lists would not provide substantial benefits compared to a single list. 147 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 148 Chapter 3: Memory Management Whereas count keeps a record of the number of pages associated with the element, high is a watermark. If the value of count exceeds high, this indicates that there are too many pages in the list. No explicit watermark for low fill states is used: When no elements are left, the list is refilled. list is a doubly linked list that holds the per-CPU pages and is handled using standard methods of the kernel. If possible, the per-CPU caches are not filled with individual pages but with multipage chunks. batch is a guideline to the number of pages to be added in a single pass. Figure 3-6 illustrates graphically how the data structures of the per-CPU cache are filled on a dual- processor system. count = 36 high = 96 batch = 16 count = 16 high = 32 batch = 16 CPU 0 CPU 1 count = 36 high = 96 batch = 16 count = 36 high = 96 batch = 16 Hot pages Cold pages Figure 3-6: Per-CPU cache on a dual-processor system. How watermarks are calculated and how the cache data structures are initialized are discussed in more detail in Section 3.4.2. Page Frames Page frames represent the smallest unit of system memory, and an instance of struct page is created for each page in RAM. Kernel programmers take care to keep this structure as small as possible because the memory of systems even with a moderate RAM configuration is broken down into a very large number of pages. For instance, an IA-32 system working with a standard page size of 4 KiB has around 100,000 pages given a main memory size of 384 MiB. Although this memory size is certainly not excessively large for today’s standards, the number of pages is already considerable. This is why the kernel makes great efforts to keep struct page as small as possible. The sheer number of pages in a typical system causes even small changes in the structure to lead to a large increase in the amount of physical memory required to keep all page instances. Keeping the structure small is not exactly simplified by the ubiquity of pages: They are used in many parts of memory management, and for varying applications. While one part of the kernel absolutely depends on a specific piece of information being available in struct page, this could be useless for another part, which itself depends a different piece of information, which could again be completely useless for the other part, and so on ... . ACunion lends itself naturally as a remedy for this problem, even if clarity of struct page is not increased at first. Consider an example: A physical page can be mapped into the virtual address space via page tables from multiple places, and the kernel wants to keep track of how many places map the page. For this end, a counter in struct page counts the number of mappings. If a page is used by the slub 148 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 149 Chapter 3: Memory Management allocator (a means to subdivide complete pages into into smaller portions, see Section 3.6.1), then it is guaranteed to be only used by the kernel and not from somewhere else, so the map count information is superfluous. Instead, the kernel can reinterpret the field to denote how many small memory objects into which a page is subdivided are in use. The double interpretation looks as follows in the data structure definition: struct page { ... union { atomic_t _mapcount; /* Count of ptes mapped in mms, * to show when page is mapped * & limit reverse map searches. */ unsigned int inuse; /* SLUB: Nr of objects */ }; ... } Note that atomic_t and unsigned int are two different data types — the first allows for changing values atomically, that is, safe against concurrent access, while the second is a classical integer. atomic_t pro- vides 32 bits,6 and an integer also provides this many bits on each architecture supported by Linux. Now it could be tempting to ‘‘simplify’’ the definition as follows: struct page { ... atomic_t counter; ... } This would be bad style, though, and is completely unacceptable to the kernel developers. The slub code does not need atomicity to access its object counter, and this should also be reflected in the data type. And, most importantly, readability of the code will suffer in both subsystems. While _mapcount and inuse provide a clear and concise description of what the element is about, counter could mean almost everything. Definition of page The structure is defined as follows: struct page { unsigned long flags; /* Atomic flags, some possibly * updated asynchronously */ atomic_t _count; /* Usage count, see below. */ union { atomic_t _mapcount; /* Count of ptes mapped in mms, * to show when page is mapped * & limit reverse map searches. */ 6Before kernel 2.6.3, this was not true. The Sparc architecture could only provide 24 bits for atomic manipulation, so the generic code for all architecture needed to stick to this limit. Luckily, this problem has been resolved now by improvements in the Sparc specific code. 149 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 150 Chapter 3: Memory Management unsigned int inuse; /* SLUB: Nr of objects */ }; union { struct { unsigned long private; /* Mapping-private opaque data: * usually used for buffer_heads * if PagePrivate set; used for * swp_entry_t if PageSwapCache; * indicates order in the buddy * system if PG_buddy is set. */ struct address_space *mapping; /* If low bit clear, points to * inode address_space, or NULL. * If page mapped as anonymous * memory, low bit is set, and * it points to anon_vma object: * see PAGE_MAPPING_ANON below. */ }; ... struct kmem_cache *slab; /* SLUB: Pointer to slab */ struct page *first_page; /* Compound tail pages */ }; union { pgoff_t index; /* Our offset within mapping. */ void *freelist; /* SLUB: freelist req. slab lock */ }; struct list_head lru; /* Pageout list, eg. active_list * protected by zone->lru_lock ! */ #if defined(WANT_PAGE_VIRTUAL) void *virtual; /* Kernel virtual address (NULL if not kmapped, ie. highmem) */ #endif /* WANT_PAGE_VIRTUAL */ }; The elements slab, freelist,andinuse are used by the slub allocator. We do not need to be concerned with these special arrangements, and they are not used if support for the slub allocator is not compiled into the kernel, so I omit them in the following discussion to simplify matters. Each page frame is described by this structure in an architecture-independent format that does not depend on the CPU type used. Besides the slub elements, the page structure includes several other ele- ments that can only be explained accurately in the context of kernel subsystems discussed elsewhere. I shall nevertheless provide an overview of the contents of the structure, even though this means referenc- ing later chapters. ❑ flags stores architecture-independent flags to describe page attributes. I discuss the different flag options below. ❑ _count is a usage count indicating the number of references to this page in the kernel. When its value reaches 0, the kernel knows that the page instance is not currently in use and can therefore be removed. If its value is greater than 0, the instance should on no account be removed from memory. If you are not familiar with reference counters, you should consult Appendix C for further information. 150 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 151 Chapter 3: Memory Management ❑ _mapcount indicates how many entries in the page table point to the page. ❑ lru is a list head used to keep the page on various lists that allow grouping the pages into different categories, most importantly active and inactive pages. Especially the discussion in Chapter 18 will come back to these lists. ❑ The kernel allows for combining multiple adjacent pages into a larger compound page.Thefirst page in the cluster is called the head page, while all other pages are named tail page. All tail pages have first_page set to point to the head page. ❑ mapping specifies the address space in which a page frame is located. index is the offset within the mapping. Address spaces are a very general concept used, for example, when reading a file into memory. An address space is used to associate the file contents (data) with the areas in memory into which the contents are read. By means of a small trick,7 mapping is able to hold not only a pointer, but also information on whether a page belongs to an anonymous memory area that is not associated with an address space. If the bit with numeric value 1 is set in mapping,the pointer does not point to an instance of address_space but to another data structure (anon_vma) that is important in the implementation of reverse mapping for anonymous pages; this struc- ture is discussed in Section 4.11.2. Double use of the pointer is possible because address_space instances are always aligned with sizeof(long); the least significant bit of a pointer to this instance is therefore 0 on all machines supported by Linux. The pointer can be used directly if it points normally to an instance of address_space.Ifthetrick involving setting the least significant bit to 1 is used, the kernel can restore the pointer by means of the following operation: anon_vma = (struct anon_vma *) (mapping - PAGE_MAPPING_ANON) ❑ private is a pointer to ‘‘private‘‘ data ignored by virtual memory management. The pointer can be employed in different ways depending on page usage. It is mostly used to associate the page with data buffers as described in the following chapters. ❑ virtual is used for pages in the highmem area, in other words, for pages that cannot be directly mapped into kernel memory. virtual then accepts the virtual address of the page. As the pre-processor statement #ifdef{WANT_PAGE_VIRTUAL} shows, the virtual element is only part of struct page if the corresponding pre-processor constant is defined. Currently, this is only the case for a few architectures, namely, Motorola m68k, FRV, and Extensa. All other architectures adopt a different scheme of addressing virtual pages. At the heart of this is a hash table used to find the address of all highmem pages. Section 3.5.8 deals with the appro- priate techniques in more detail. Handling the hash table requires some mathematical operations that are slow on the aforementioned machines, so they chose the direct approach. Architecture-Independent Page Flags The different attributes of a page are described by a series of page flags stored as bits in the flags element of struct page. The flags are independent of the architecture used and cannot therefore provide CPU- or machine-specific information (this information is held in the page table itself as is shown below). Not only are the individual flags defined with the help of the pre-processor in page-flags.h,butalso macros are generated to set, delete, and query the flags. In doing so, the kernel conforms to a universal 7The trick borders on the unscrupulous but helps save space in one of the most frequently needed kernel structures. 151 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 152 Chapter 3: Memory Management naming scheme; for example, the PG_locked constant defines the bit position in flags to specify whether a page is locked or not. The following macros are available to manipulate the bit: ❑ PageLocked queries whether the bit is set. ❑ SetPageLocked sets the PG_locked bit, regardless of its previous state. ❑ TestSetPageLocked sets the bit, but also returns its old value. ❑ ClearPageLocked deletes the bit regardless of its previous state. ❑ TestClearPageLocked deletes the bit and returns its old value. There is an identical set of macros to perform the operations shown on the appropriate bit for the other page flags. The macros are implemented atomically. Although some of them are made up of several state- ments, special processor commands are used to ensure that they act as if they were a single statement; that is, they cannot be interrupted as this would result in race conditions. (Chapter 14 describes how race conditions arise and how they can be prevented.) Which page flags are available? The following list includes the most important flags (again, their mean- ings become clear in later chapters): ❑ PG_locked specifies whether a page is locked. If the bit is set, other parts of the kernel are not allowed to access the page. This prevents race conditions in memory management, for example, when reading data from hard disk into a page frame. ❑ PG_error is set if an error occurs during an I/O operation involving the page. ❑ PG_referenced and PG_active control how actively a page is used by the system. This infor- mation is important when the swapping subsystem has to select which page to swap out. The interaction of the two flags is explained in Chapter 18. ❑ PG_uptodate indicates that the data of a page have been read without error from a block device. ❑ PG_dirty is set when the contents of the page have changed as compared to the data on hard disk. For reasons of performance, pages are not written back immediately after each change. The kernel therefore uses this flag to note which pages have been changed so that they can be flushed later. Pages for which this flag has been set are referred to as dirty (generally, this means that the data in RAM and the data on a secondary storage medium such as a hard disk have not been synchro- nized). ❑ PG_lru helps implement page reclaim and swapping. The kernel uses two least recently used lists8 to distinguish between active and inactive pages. The bit is set if the page is held on one of these lists. There is also a PG_active flag that is set if the page is on the list of active pages. Chapter 18 discusses this important mechanism in detail. ❑ PG_highmem indicates that a page is in high memory because it cannot be mapped permanently into kernel memory. ❑ PG_private must be set if the value of the private element in the page structure is non-NULL. Pages that are used for I/O use this field to subdivide the page into buffers (see Chapter 16 for more information), but other parts of the kernel find different uses to attach private data to a page. 8Frequently used entries are automatically in the foremost positions on this type of list, whereas inactive entries are always moved toward the end of the list. 152 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 153 Chapter 3: Memory Management ❑ PG_writeback is set for pages whose contents are in the process of being written back to a block device. ❑ PG_slab is set for pages that are part of the slab allocator discussed in Section 3.6. ❑ PG_swapcache is set if the page is in the swap cache; in this case, private contains an entry of type swap_entry_t (further details are provided in Chapter 18). ❑ When the available amount of memory gets smaller, the kernel tries to periodically reclaim pages, that is, get rid of inactive, unused pages. Chapter 18 discusses the details. Once the kernel has decided to reclaim a specific page, this is announced by setting the PG_reclaim flag. ❑ PG_buddy is set if the page is free and contained on the lists of the buddy system, that is, the core of the page allocation mechanism. ❑ PG_compound denotes that the page is part of a larger compound page consisting of multiple adjacent regular pages. A number of standard macros are defined to check if a page has a specific bit is set, or to manipulate a bit. Their names follow a certain pattern: ❑ PageXXX(page) checks if a page has the PG_XXX bit set. For instance, PageDirty checks for the PG_dirty bit, while PageActive checks for PG_active,andsoon. ❑ To set a bit if it is not set and return the previous value, SetPageXXX is provided. ❑ ClearPageXXX unconditionally deletes a specific bit. ❑ TestClearPageXXX clears a bit if it is set, but also returns the previously active value. Notice that these operations are implemented atomically. Chapter 5 discusses what this means in more detail. Often it is necessary to wait until the state of a page changes, and then resume work. Two auxiliary functions provided by the kernel are of particular interest for us: void wait_on_page_locked(struct page *page); void wait_on_page_writeback(struct page *page) Assume that one part of the kernel wants to wait until a locked page has been unlocked. wait_on_page_locked allows for doing this. While how this is technically done is discussed in Chapter 14, it suffices to know here that after calling the function, the kernel will go to sleep if the page is locked. Once the page becomes unlocked, the sleeper is automatically woken up and can continue its work. wait_on_page_writeback works similarly, but waits until any pending writeback operations in which the data contained in the page are synchronized with a block device — a hard disk, for instance — have been finished. 3.3 Page Tables Hierarchically linked page tables are used to support the rapid and efficient management of large address spaces. The principle behind this approach and the benefits it brings as compared to linear addressing are discussed in Chapter 1. Here we take a closer look at the technical aspects of implementation. 153 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 154 Chapter 3: Memory Management Recall that page tables are used to establish an association between the virtual address spaces of user processes and the physical memory of the system (RAM, page frames). The structures discussed so far serve to describe the structure of RAM memory (partitioning into nodes and zones) and to specify the number and state (used or free) of the page frames contained. Page tables are used to make a uniform virtual address space available to each process; the applications see this space as a contiguous memory area. The tables also map the virtual pages used into RAM, thus supporting the implementation of shared memory (memory shared by several processes at the same time) and the swapping-out of pages to a block device to increase the effective size of usable memory without the need for additional physical RAM. Kernel memory management assumes four-level page tables — regardless of whether this is the case for the underlying processor. The best example where this assumption is not true is IA-32 systems. By default, this architecture uses only a two-level paging system — assuming the PAE extensions are not used. Consequently, the third and fourth levels must be emulated by architecture- specific code. Page table management is split into two parts, the first architecture-dependent, the second architecture- independent. Interestingly, all data structures and almost all functions to manipulate them are defined in architecture-specific files. Because there are some big differences between CPU-specific implementations (owing to the various CPU concepts used), I won’t go into the low-level details for the sake of brevity. Extensive knowledge of the individual processors is also required, and the hardware documentation for each processor family is generally spread over several books. Appendix A describes the IA-32 architec- ture in more detail. It also discusses, at least in summary form, the architecture of the other important processors supported by Linux. The descriptions of data structures and functions in the following sections are usually based on the interfaces provided by the architecture-dependent files. The definitions can be found in the header files include/asm-arch/page.h and include/asm-arch/pgtable.h referred to in abbreviated form as page.h and pgtable.h below. Since AMD64 and IA-32 are unified into one architecture but exhibit a good many differences when it comes to handling page tables, the definitions can be found in two different files: include/asm-x86/page_32.h and include/asm-x86/page_64.h, and similar for pgtable_XX.h.When aspects relating to a specific architecture are discussed, I make explicit reference to the architecture. All other information is equally valid for all architectures even if the definitions of the associated structures are architecture-specific. 3.3.1 Data Structures In C, the void* data type is used to specify a pointer to any byte positions in memory. The number of bits required differs according to architecture. All common processors (including all those on which Linux runs) use either 32 or 64 bits. The kernel sources assume that void* and unsigned long have the same number of bits so that they can be mutually converted by means of typecasts without loss of information. This assumption — expressed formally as sizeof(void*) == sizeof(unsigned long) — is, of course, true on all architectures sup- ported by Linux. Memory management prefers to use variables of type unsigned long instead of void pointers because they are easier to handle and manipulate. Technically, they are both equally valid. Breakdown of Addresses in Memory Addresses in virtual memory are split into five parts as required by the structure of the four-level page tables (four table entries to select the page and an index to indicate the position within the page). 154 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 155 Chapter 3: Memory Management Not only the length but also the way in which the address is split are different on the individual architectures. The kernel therefore defines macros to break down the address into its individual components. PAGE_SHIFT PMD_SHIFT PUD_SHIFT PGDIR_SHIFT BITS_PER_LONG PGD PUD PMD PTE Offset Figure 3-7: Breakdown of a virtual address. Figure 3-7 shows how the positions of the address elements are defined by bit shifts. BITS_PER_LONG specifies the number of bits used for an unsigned long variable and therefore also for a generic pointer to virtual address space. At the end of each pointer there are several bits to specify the position within the selected frame page. The number of bits required is held in PAGE_SHIFT. PMD_SHIFT specifies the total number of bits used by a page and by an entry in the last level of the page tables. This number can be subtracted from PAGE_SHIFT to determine the number of bits required by an entry in the last hierarchy level of the page table. More important is the fact that the value indicates the size of the partial address space managed by an entry in the middle page table, namely, 2PMD_SHIFT bytes. PUD_SHIFT adds together the bit lengths of PAGE_OFFSET and PMD_SHIFT,whereasPGDIR_SHIFT combines the bit lengths of PAGE_OFFSET, PUD_SHIFT,andPMD_SHIFT with the bit number of an entry in the page middle directory. The value is the binary logarithm of the size of the partial address space that can be addressed via an entry in the page global directory. The number of pointers that can be stored in the various directories of the page table is also deter- mined by macro definitions. PTRS_PER_PGD specifies the number of entries in the page global directory, PTRS_PER_PMD the number in the page middle directory, PTRS_PER_PUD the number in the page upper directory, and PTRS_PER_PTE the number in the page table entry. Architectures with two-level page tables define PTRS_PER_PMD and PTRS_PER_PUD as 1. This persuades the remaining parts of the kernel that they are working with four-level page translation although only two pages are used — the page middle and page upper directories are effectively eliminated because they have only a single entry. Because only a very few systems use a four-level page table, the kernel uses the header file include/asm-generic/pgtable-nopud.h to hold all the declara- tions needed to simulate the presence of a fourth page table. The header file include/asm-generic/pgtable-nopmd.h is also available to simulate the presence of a third page table level on systems with two-level address translation. 155 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 156 Chapter 3: Memory Management The size of the address area that can be addressed with pointers of n-bit length is 2n bytes. The kernel defines additional macro variables to hold the values calculated so that it is unnecessary to repeat the calculations time and time again. The variables are defined as follows: #define PAGE_SIZE (1UL << PAGE_SHIFT) #define PUD_SIZE (1UL << PUD_SHIFT) #define PMD_SIZE (1UL << PMD_SHIFT) #define PGDIR_SIZE (1UL << PGDIR_SHIFT) The value 2n is easily calculated in the binary system by shifting a bit n positions to the left starting from position 0. The kernel uses this ‘‘trick‘‘ at many places. Those of you unfamiliar with bit arithmetic will find relevant explanations in Appendix C. include/asm-x86/pgtable_64.h #define PGDIR_SHIFT 39 #define PTRS_PER_PGD 512 #define PUD_SHIFT 30 #define PTRS_PER_PUD 512 #define PMD_SHIFT 21 #define PTRS_PER_PMD 512 The macros PTRS_PER_XXX specify how many pointers (i.e., different values) a given directory entry can represent. Since AMD64 employs 9 bits for each directory, 29 = 512 pointers fit into each. The kernel also needs a means of extracting the individual components from a given address. The kernel uses the bitmasks defined below to do this. #define PAGE_MASK (~(PAGE_SIZE-1)) #define PUD_MASK (~(PUD_SIZE-1)) #define PMD_MASK (~(PMD_SIZE-1)) #define PGDIR_MASK (~(PGDIR_SIZE-1)) The masks are applied on a given address by simple bitwise addition. Format of Page Tables The size of the entries in the page tables has been established by the above definitions but not their structure. The kernel provides four data structures (defined in page.h) to represent the entry structures. ❑ pgd_t for entries of the global directory. ❑ pud_t for entries of the page upper directory. ❑ pmd_t for entries of the page middle directory. ❑ pte_t for direct page table entries. The standard functions to analyze page table entries are listed in Table 3-2. (Depending on architecture, some functions are implemented as macros and others as inline functions; I make no distinction between the two below.) 156 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 157 Chapter 3: Memory Management Table 3-2: Functions for Analyzing Page Table Entries. Function Description pgd_val pud_val pmd_val pte_val pgprot_val Convert a variable of type pte_t and so on to an unsigned long number. __pgd __pud __pmd __pte __pgprot Do the reverse of pdg_val and so on: They convert an unsigned long number into avariableoftypepdg_t and so on. pgd_index pud_index pmd_index pte_index Yield the address of the next-level table starting from a memory pointer and a page table entry. pgd_present pud_present pmd_present pte_present Check whether the _PRESENT bit of the corresponding entry is set. This is the case when the page or page table addressed is in RAM memory. pgd_none pud_none pmd_none pte_none Do the logical reverse of the xxx_present functions. If they return a true value, the searched page is not in RAM. pgd_clear pud_clear pmd_clear pte_clear Delete the passed page table entry. This is usually done by setting it to zero. pgd_bad pud_bad pmd_bad Check whether entries of the page middle, upper, and global directories are invalid. They are used for safety purposes in functions that receive input param- eters from the outside where it cannot be assumed that the parameters are valid. pmd_page pud_page pte_page Return the address of the page structure holding the data on the page or the entries of the page middle directories. How do the offset functions work? Let us consider pmd_offset as an example. It requires as parameter an entry from the page global directory (src_pgd)andanaddress in memory. It returns an element from one of the page middle directories. src_pmd = pmd_offset(src_pgd, address); 157 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 158 Chapter 3: Memory Management PAGE_ALIGN is another standard macro that must be defined by each architecture (typically in page.h). It expects an address as parameter and ‘‘rounds‘‘ the address so that it is exactly at the start of the next page. If the page size is 4,096, the macro always returns an integer multiple of this size; PAGE_ALIGN(6000) = 8192 = 2× 4,096, PAGE_ALIGN(0x84590860) = 0x84591000 = 542,097 × 4,096. The alignment of addresses topageboundariesisimportanttoensurethatbestuse is made of the cache resources of the processor. Although C structures are used to represent entries in page tables, most consist of just a single element — typically unsigned long — as an example of AMD64 architecture shows:9 include/asm-x86_64/page.h typedef struct { unsigned long pte; } pte_t; typedef struct { unsigned long pmd; } pmd_t; typedef struct { unsigned long pud; } pud_t; typedef struct { unsigned long pgd; } pgd_t structs are used instead of elementary types to ensure that the contents of page table elements are handled only by the associated helper functions and never directly. The entries may also be constructed of several elementary variables. In this case, the kernel is obliged to use a struct.10 The virtual address is split into several parts that are used as an index into the page table in accordance with the familiar scheme. The individual parts are therefore less than 32 or 64 bits long, depending on the word length of the architecture used. As the excerpt from the kernel sources shows, the kernel (and therefore also the processor) uses 32- or 64-bit types to represent entries in the page tables (regardless of table level). This means that not all bits of a table entry are required to store the useful data — that is, the base address of the next table. The superfluous bits are used to hold additional information. Appendix A describes the structure of the page tables on various architectures in detail. PTE-Specific Entries Each final entry in the page table not only yields a pointer to the memory location of the page, but also holds additional information on the page in the superfluous bits mentioned above. Although these data are CPU-specific, they usually provide at least some information on page access control. The following elements are found in most CPUs supported by the Linux kernel: ❑ _PAGE_PRESENT specifies whether the virtual page is present in RAM memory. This need not necessarily be the case because pages may be swapped out into a swap area as noted briefly in Chapter 1. The structure of the page table entry is usually different if the page is not present in memory because there is no need to describe the position of the page in memory. Instead, information is needed to identify and find the swapped-out page. 9The definitions for IA-32 are similar. However, only pte_t and pgd_t, which are defined as unsigned long, make an effective contribution. I use the code example for AMD64 because it is more regular. 10When IA-32 processors use PAE mode, they define pte_t as, for example, typedef struct { unsigned long pte_low, pte_high; }. 32 bits are then no longer sufficient to address the complete memory because more than 4 GiB can be managed in this mode. In other words, the available amount of memory can be larger than the processor’s address space. Since pointers are, however, still only 32 bits wide, an appropriate subset of the enlarged memory space must be chosen for userspace applications that do still only see 4 GiB each. 158 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 159 Chapter 3: Memory Management ❑ _PAGE_ACCESSED is set automatically by the CPU each time the page is accessed. The kernel reg- ularly checks the field to establish how actively the page is used (infrequently used pages are good swapping candidates). The bit is set after either read or write access. ❑ _PAGE_DIRTY indicates whether the page is ‘‘dirty,’’ that is, whether the page contents have been modified. ❑ _PAGE_FILE has the same numerical value as _PAGE_DIRTY, but is used in a different context, namely, when a page is not present in memory. Obviously, a page that is not present cannot be dirty, so the bit can be reinterpreted: If it is not set, then the entry points to the location of a swapped-out page (see Chapter 18). A set _PAGE_FILE is required for entries that belongs to nonlinear file mappings which are discussed in Section 4.7.3. ❑ If _PAGE_USER is set, userspace code is allowed to access the page. Otherwise, only the kernel is allowed to do this (or when the CPU is in system mode). ❑ _PAGE_READ, _PAGE_WRITE,and_PAGE_EXECUTE specify whether normal user processes are allowed to read the page, write to the page, or execute the machine code in the page. Pages from kernel memory must be protected against writing by user processes. There is, however, no assurance that even pages belonging to user processes can be written to, for example, if the page contains executable code that may not be modified — either intention- ally or unintentionally. Architectures that feature less finely grained access rights define the _PAGE_RW constant to allow or disallow read and write access in combination if no further criterion is available to distinguish between the two. ❑ IA-32andAMD64provide_PAGE_BIT_NX to label the contents of a page as not executable (this protection bit is only available on IA-32 systems if the page address extensions for addressing 64 GiB memory are enabled). It can prevent, for example, execution of code on stack pages that can result in security gaps in programs because of intentionally provoked buffer overflows if malicious code has been introduced. The NX bit cannot prevent buffer overflow but can suppress its effects because the process refuses to run the malicious code. Of course, the same result can also be achieved if the architectures themselves provide a good set of access authorization bits for memory pages, as is the case with some (unfortunately not very common) processors. Each architecture must provide two things to allow memory management to modify the additional bits in pte_t entries — the data type __pgprot in which the additional bits are held, and the pte_modify function to modify the bits. The above pre-processor symbols are used to select the appropriate entry. The kernel also defines various functions to query and set the architecture-dependent state of memory pages. Not all functions can be defined by all processors because of lack of hardware support for a given feature. ❑ pte_present checks if the page to which the page table entry points is present in memory. This function can, for instance, be used to detect if a page has been swapped out. ❑ pte_dirty checks if the page associated with the page table entry is dirty, that is, its contents have been modified since the kernel checked last time. Note that this function may only be called if pte_present has ensured that the page is available. ❑ pte_write checks if the kernel may write to the page. 159 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 160 Chapter 3: Memory Management ❑ pte_file is employed for nonlinear mappings that provide a different view on file contents by manipulating the page table (this mechanism is discussed in more detail in Section 4.7.3). The function checks if a page table entry belongs to such a mapping. pte_file may only be invoked if pte_present returns false; that is, the page asso- ciated with the page table entry is not present in memory. Since the generic code relies on pte_file, it must also be defined if an architecture does not sup- port nonlinear mappings. In this case, the function always returns 0. A summary of all functions provided to manipulate PTE entries can be found in Table 3-3. Table 3-3: Functions for Processing the Architecture-Dependent State of a Memory Page Function Description pte_present Is the page present? pte_read May the page be read from within userspace? pte_write May the page be written to? pte_exec May the data in the page be executed as binary code? pte_dirty Is the page dirty; that is, have its contents been modified? pte_file Does the PTE belong to a nonlinear mapping? pte_young Is the access bit (typically _PAGE_ACCESS)set? pte_rdprotect Removes read permission for the page. pte_wrprotect Deletes write permission for the page. pte_exprotect Removes permission to execute binary data in the page. pte_mkread Sets read permission. pte_mkwrite Sets write permission. pte_mkexec Permits execution of page contents. pte_mkdirty Marks the page as dirty. pte_mkclean ‘‘Cleans‘‘ the page; that is, usually deletes the _PAGE_DIRTY bit. pte_mkyoung Sets the accessed bit — _PAGE_ACCESSED on most architectures. pte_mkold Deletes the accessed bit. 160 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 161 Chapter 3: Memory Management The functions often appear in groups of three to set, delete, and query a specific attribute, for instance, write permission for a page. The kernel assumes that access to page data can be regulated in three dif- ferent ways — by means of write, read, and execution permission. (Execution permission indicates that page binary data may be executed as machine code in the same way as programs are executed.) How- ever, this assumption is a little too optimistic for some CPUs. IA-32 processors support only two control modes to allow reading and writing. In this case, the architecture-dependent code tries to emulate the desired semantics as best it can. 3.3.2 Creating and Manipulating Entries Table 3-4 lists all functions for creating new page table entries. Table 3-4: Functions for Creating New Page Table Entries Function Description mk_pte Createsapteentry;apage instance and the desired page access permissions must be passed as parameters. pte_page Yields the address of the page instance belonging to the page described by the page table entry. pgd_alloc pud_alloc pmd_alloc pte_alloc Reserve and initialize memory to hold a complete page table (not just a single entry). pgd_free pud_free pmd_free pte_free Free the memory occupied by the page table. set_pgd set_pud set_pmd set_pte Set the value of an entry in a page table. The functions in the table must be implemented by all architectures to enable memory management code to create and destroy page tables. 3.4 Initialization of Memory Management In the context of memory management, initialization can have multiple meanings. On many CPUs, it is necessary to explicitly set the memory model suitable for the Linux kernel, for example, by switching to protected mode on IA-32 systems, before it is possible to detect the available memory and register it with the kernel. In the course of initialization, it is also necessary to set up the data structures of memory management, and much more. Because the kernel needs memory before memory management is fully 161 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 162 Chapter 3: Memory Management initialized, a simple additional form of memory management is used during the boot process and is discarded thereafter. As the CPU-specific parts of memory management initialization employ many minor, subtle details of the underlying architecture that reveal little of interest about the structure of the kernel and are simply best practices in assembly language programming, let’s concern ourselves in this section only with ini- tialization work on a higher level. The key aspect is initialization of the pg_data_t data structure (and its subordinate structures) introduced in Section 3.2.2 because this is already machine-independent. The primary purpose of the aforementioned processor-specific operations whose details we will ignore is to investigate how much memory is available in total and how it is shared between the individual nodes and zones of the system. 3.4.1 Data Structure Setup Initialization of the data structures is launched from within the start_kernel global start routine that is executed after kernel loading to render the various subsystems operational. As memory management is a very important kernel component, it is initialized almost immediately after architecture-specific setup, which is responsible for the technical details of detecting memory and establishing how it is distributed in the system (Section 3.4.2 deals briefly with the implementation of system-dependent initialization on IA-32 systems). At this point, an instance of pgdat_t has been generated for each system memory mode to hold information on how much memory there is in the node and how it is distributed over the node zones. The architecture-specific NODE_DATA macro implemented on all platforms is used to query the pgdat_t instance associated with a NUMA node by reference to the number of the instance. Prerequisites Since the majority of systems have just one memory node, only systems of this type are examined below. What is the situation on such systems? To ensure that memory management code is portable (so that it can be used on UMA and NUMA systems alike), the kernel defines a single instance of pg_data_t (called contig_page_data)inmm/page_alloc.c to manage all system memory. As the file pathname suggests, this is not a CPU-specific implementation; in fact, it is adopted by most architectures. The implementation of NODE_DATA is now even simpler. #define NODE_DATA(nid) (&contig_page_data) Although the macro has a formal parameter for selecting a NUMA node, the same data are always returned — there is just one pseudo-node. The kernel can also rely on the fact that the architecture-dependent initialization code has set the numnodes variable to the number of nodes present in the system. This number is 1 on UMA systems because only one (formal) node is present. At compilation time, pre-processor statements select the correct definitions for the particular configuration. System Start Figure 3-8 shows a code flow diagram for start_kernel. It includes only the system initialization func- tions associated with memory management. 162 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 163 Chapter 3: Memory Management start_kernel setup_arch setup_per_cpu_areas build_all_zonelists mem_init setup_per_cpu_pageset Figure 3-8: Kernel initialization in the view of memory management. Let’s take a closer look at the functions invoked in the sections below after first summarizing their tasks as follows: ❑ setup_arch is an architecture-specific set-up function responsible for, among other things, ini- tialization of the boot allocator. ❑ On SMP systems, setup_per_cpu_areas initializes per-CPU variables defined statically inthesourcecode(usingtheper_cpu macro) and of which there is a separate copy for each CPU in the system. Variables of this kind are stored in a separate section of the kernel binaries. The purpose of setup_per_cpu_areas is to create a copy of these data for each system CPU. This function is a null operation on non-SMP systems. ❑ build_all_zonelists sets up the node and zone data structures (see below). ❑ mem_init is another architecture-specific function to disable the bootmem allocator and perform the transition to the actual memory management functions, as discussed shortly. ❑ kmem_cache_init initializes the in-kernel allocator for small memory regions. ❑ setup_per_cpu_pageset allocates memory for the first array element of the pageset arrays from struct zone mentioned above. Allocating the first array element means, in other words, for the first system processor. All memory zones of the system are taken into account. The function is also responsible for setting the limits for the hot-n-cold allocator discussed at length in Section 3.5.3. Notice that the pageset arrays members of other CPUs on SMP systems will be initialized when they are activated. Node and Zone Initialization build_all_zonelists builds the data structures required to manage nodes and their zones. Interest- ingly, it can be implemented by the macros and abstraction mechanisms introduced above regardless of whether it runs on a NUMA or UMA system. This works because the executed functions are available in two flavors: one for NUMA systems and one for UMA systems. Since this little trick is often used by the kernel, I will briefly discuss it. Suppose that a certain task must be performed differently depending on the compile-time configuration. One possibility would 163 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 164 Chapter 3: Memory Management be using two different functions and select the proper one each time it is called with some pre-processor conditionals: void do_something() { ... #ifdef CONFIG_WORK_HARD do_work_fast(); #else do_work_at_your_leisure(); #endif ... } Since this requires using the pre-processor each time the function is called, this approach is consid- ered bad style by the kernel developers. A much more elegant solution is to define the function itself differently depending on the chosen configuration: #ifdef CONFIG_WORK_HARD void do_work() { /* Get going, fast! */ ... } #else void do_work() { /* Relax, take it easy */ ... } #endif Notice that the same name is employed for both implementations because they can never be active at the same time. Calling the proper function is now not more complicated than calling a regular function: void do_something() { ... do_work(); /* Work hard or not, depending on configuration /* ... } Clearly, this variant is much more readable and is always preferred by the kernel developers (in fact, patches using the first style will have a very hard time getting into the mainline kernel, if at all). Let us go back to setting up the zone lists. The portion of build_all_zonelists that is currently of interest to us (there is some more work to do for the page group mobility extensions to the page allocator, but I will discuss this separately below) delegates all work to __build_all_zonelists, which, in turn, invokes build_zonelists for each NUMA node in the system. mm/page_alloc.c static int __build_all_zonelists(void *dummy) { int nid; 164 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 165 Chapter 3: Memory Management for_each_online_node(nid) { pg_data_t *pgdat = NODE_DATA(nid); build_zonelists(pgdat); ... } return 0; } for_each_online_node iterates over all active nodes in the system. As UMA systems have only one node, build_zonelists is invoked just once to create the zone lists for the whole of memory. NUMA systems must invoke the function as many times as there are nodes; each invocation generates the zone data for a different node. build_zonelists expects as parameter a pointer to a pgdat_t instance containing all existing information on the node memory configuration and holding the newly created data structures. On UMA systems, NODE_DATA returns the address of contig_page_data. The task of the function is to establish a ranking order between the zones of the node currently being processed and the other nodes in the system; memory is then allocated according to this order. This is important if no memory is free in the desired node zone. Let us look at an example in which the kernel wants to allocate high memory. It first attempts to find a free segment of suitable size in the highmem area of the current node. If it fails, it looks at the regular memory area of the node. If this also fails, it tries to perform allocation in the DMA zone of the node. If it cannot find a free area in any of the three local zones, it looks at other nodes. In this case, the alternative node should be as close as possible to the primary node to minimize performance loss caused as a result of accessing non-local memory. The kernel defines a memory hierarchy and first tries to allocate ‘‘cheap‘‘ memory. If this fails, it gradually tries to allocate memory that is ‘‘more costly‘‘ in terms of access and capacity. The high memory (highmem) range is cheapest because no part of the kernel depends on memory allo- cated from this area. There is no negative effect on the kernel if the highmem area is full — and this is why it is filled first. The situation in regular memory is different. Many kernel data structures must be held in this area and cannot be kept in highmem. The kernel is therefore faced with a critical situation if regular memory is completely full — as a result, memory is not allocated from this area until there is no free memory in the less critical highmem area. Most costly is the DMA area because it is used for data transfer between peripherals and the system. Memory allocation from this area is therefore a last resort. The kernel also defines a ranking order among the alternative nodes as seen by the current memory nodes. This helps determine an alternative node when all zones of the current node are full. The kernel uses an array of zonelist elements in pg_data_t to represent the described hierarchy as a data structure. 165 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 166 Chapter 3: Memory Management typedef struct pglist_data { ... struct zonelist node_zonelists[MAX_ZONELISTS]; ... } pg_data_t; #define MAX_ZONES_PER_ZONELIST (MAX_NUMNODES * MAX_NR_ZONES) struct zonelist { ... struct zone *zones[MAX_ZONES_PER_ZONELIST + 1]; // NULL delimited }; The node_zonelists array makes a separate entry available for every possible zone type. This entry contains a fallback list of type zonelist whose structure is discussed below. Because the fallback list must include all zones of all nodes, it consists of MAX_NUMNODES * MAX_NZ_ZONES entries, plus a further element for a null pointer to mark the end of the list. The task of creating a fallback hierarchy is delegated to build_zonelists, which creates the data struc- tures for each NUMA node. It requires as parameter a pointer to the relevant pg_data_t instance. Before I discuss the code in detail, let us recall one thing mentioned above. Since we have restricted our dis- cussion to UMA systems, why would it be necessary to consider multiple NUMA nodes? Indeed, the code shown below will be replaced with a different variant by the kernel if CONFIG_NUMA is set. However, it is possible that an architecture selects the discontiguous or sparse memory option on UMA systems. This can be beneficial if the address space contains large holes. The memory ‘‘blocks’’ created by such holes can best be treated using the data structures provided by NUMA. This is why we have to deal with them here. A large external loop first iterates over all node zones. Each loop pass looks for the zone entry for the i-th zone in the zonelist array in which the fallback list is held. mm/page_alloc.c static void __init build_zonelists(pg_data_t *pgdat) { int node, local_node; enum zone_type i,j; local_node = pgdat->node_id; for (i = 0; i < MAX_NR_ZONES; i++) { struct zonelist *zonelist; zonelist = pgdat->node_zonelists + i; j = build_zonelists_node(pgdat, zonelist, 0, j); ... } The array element of node_zonelists is addressed by means of pointer manipulation, a perfectly legal practice in C. The actual work is delegated to build_zonelist_node. When invoked, it first generates the fallback order within the local node. 166 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 167 Chapter 3: Memory Management mm/page_alloc.c static int __init build_zonelists_node(pg_data_t *pgdat, struct zonelist *zonelist, int nr_zones, enum zone_type zone_type) { struct zone *zone; do { zone = pgdat->node_zones + zone_type; if (populated_zone(zone)) { zonelist->zones[nr_zones++] = zone; } zone_type--; } while (zone_type >= 0); return nr_zones; } The fallback list entries are ordered by means of the zone_type parameter that specifies the zone from which memory is best taken and is calculated using highest_zone as shown. Recall that it can have one of the following values: ZONE_HIGHMEM, ZONE_NORMAL, ZONE_DMA,orZONE_DMA32. nr_zone denotes the position in the fallback list at which filling new entries starts. The caller has passed 0 since there is no entry in the list yet. The kernel then iterates over all zones from costly to less costly. In each step, populated_zone ensures that zone->present_pages is greater than 0 for the selected zone; that is, whether there are pages in the zone. If so, a pointer to the zone instance previously determined is added at the current position within thezonelistzonelist->zones. The current position in the zone list is held in nr_zone. At the end of each step, the zone type is decremented by 1; in other words, it is set to a more costly zone type. For example, if the start zone is ZONE_HIGHMEM, decrementing by 1 ensures that the next zone type used is ZONE_NORMAL. Consider a system with the zones ZONE_HIGHMEM, ZONE_NORMAL,andZONE_DMA.Inthefirstrunof build_zonelists_node, the following assignments are made: zonelist->zones[0] = ZONE_HIGHMEM; zonelist->zones[1] = ZONE_NORMAL; zonelist->zones[2] = ZONE_DMA; Figure 3-9 illustrates this for the case in which a fallback list for node 2 of a system is successively filled. There are a total of four nodes in the system (numnodes = 4); k = ZONE_HIGHMEM also applies. C2 C1 C0 C0C1C2 D2 D1 D0 C0C1C2 A2 A1 A0 B2 B1 B0D2 D1 D0 NULL... A = Node 0 B = Node 1 C = Node 2 D = Node 3 0 = DMA 1 = Normal 2 = Highmem Figure 3-9: Successive filling of the fallback list. 167 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 168 Chapter 3: Memory Management After the first step, the allocation targets in the list are highmem, followed by normal memory and finally the DMA zone of the second node. The kernel must then establish the order in which the zones of the other nodes in the system are used as fallback targets. mm/page_alloc.c static void __init build_zonelists(pg_data_t *pgdat) { ... for (node = local_node + 1; node < MAX_NUMNODES; node++) { j = build_zonelists_node(NODE_DATA(node), zonelist, j, i); } for (node = 0; node < local_node; node++) { j = build_zonelists_node(NODE_DATA(node), zonelist, j, i); } zonelist->zones[j] = NULL; } } } The first loop successively iterates over all nodes with a higher number than the node being processed. In our example, there are four nodes numbered 0, 1, 2, and 3 and therefore only node number 3 is left. New entries are added to the fallback list by build_zonelists_node. This is where the meaning of j comes into play. After the fallback targets in the local node had been found, the value of the variable was 3; this is used as the starting position for the new entries. If node number 3 also consists of three zones, the situation after invocation of build_zonelists is as shown in the second step of Figure 3-9. The second for loop then generates the entries for all nodes with lower numbers than the current node. In our example, these nodes have the numbers 0 and 1. If three zones are also present in these nodes, the fallback list situation is as shown in the lower part of Figure 3-9. The number of entries in the fallback list is never known exactly because the zone configurations may be different in the various nodes of the system. The last entry is therefore assigned a null pointer to explicitly mark the end of the list. For any node m of a total number of N nodes, the kernel always selects the order m, m + 1, m + 2, ..., N,0,1,..., m − 1 for the fallback nodes. This ensures that no node is overused (as compared, e.g., to an unchanging fallback list independent of m). Figure 3-10 shows the fallback lists built for the third node in a system with four nodes. Normal DMA HighMem A0D0C0 B2 NULL D1C0C1 D0 A1 B1A0 B0 NULL C0C1C2 A2 A1 A0 B2 B1 B0D2 D1 D0 NULL Figure 3-10: Finished fallback lists. 168 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 169 Chapter 3: Memory Management Section 3.5.5 discusses the implementation of the buddy system that makes use of the fallback lists gen- erated here. 3.4.2 Architecture-Specific Setup The initialization of memory management on IA-32 systems is in some aspects a very subtle undertaking that must overcome a few historical obstacles associated with the processor architecture. These include, for example, switching the processor from normal mode to protected mode to grant the CPU access to the 32-bit world — a legacy from the days when compatibility with 16-bit 8086 processors was important. Similarly, paging is not enabled by default and must be activated manually, which, for instance, involves fiddling with the cr0 register of the processor. However, these subtleties are of no interest to us; you are referred to the appropriate reference manuals. Notice that our focus on the IA-32 architecture does not mean that the things discussed in the following will be completely disconnected from all other architectures supported by the kernel. Quite the opposite is the case: Even if many details will be unique to the IA-32 architecture, many other architectures do things in a similar way. It’s just necessary to choose one particular architecture as an example, and since IA-32 has not only been around for quite some time, but was also the architecture initially supported by Linux, this is reflected in the kernel’s general design. Although there is a clear tendency of the kernel toward 64-bit platforms, many aspects can still be traced to its IA-32 roots. Another reason why we pick the IA-32 architecture as an example is for practical purposes: Since the address space is only 4 GiB large, all addresses can be described with comparatively compact hexadec- imal numbers, which are simply easier to read and work with than the long values required by 64-bit architectures. Interestingly, the IA-32 architecture does not exist as a separate architecture starting with kernel 2.6.24 anymore! It was merged with the AMD64 architecture to form a new, unified x86 architecture. Although both are now constrained to the single architecture-specific directory arch/x86, a good many differences still remain. This is why many files are available in two variants: file_32.c for IA-32, and file_64.c for AMD64. The existence of two different files for each subarchitecture is something that is only temporarily tough. Future development will ensure that finally a single file will contain code for both architectures. Since the unified architecture promotes the AMD64 architecture (even more) to one of the most important architectures supported by the kernel, I will also consider how architecture-specific details differ for AMD64 compared to IA-32. Owing to the large number of architectures supported by the kernel, it is not possible to discuss the specific details for all of them here. Considering one 32- and one 64-bit architecture in the following will, however, provide the taste of how Linux does things in both worlds, and lay the fundamentals to understand the approaches by other architectures. Arrangement of the Kernel in Memory Before discussing the individual memory initialization operations, we need to examine the situation in RAM after the boot loader has copied the kernel into memory and the assembler part of the initialization routines has completed. I concentrate on the default case in which the kernel is loaded to a fixed position in physical RAM that is determined at compile time. It is also possible to configure the initial position of the kernel binary in physical RAM if the crash dump mechanism is enabled. Additionally, some embedded systems will require this ability. The configuration 169 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 170 Chapter 3: Memory Management option PHYSICAL_START determines the position in RAM in this case, subjected to physical alignment specified by the configuration option PHYSICAL_ALIGN. Additionally, the kernel can be built as a relocatable binary, and the physical start address given at compile time is completely ignored in this case. The boot loader can decide where to put the kernel. Since both options are either only required in corner cases or are still considered experimental, I will not discuss them any further. Figure 3-11 shows the lowest megabytes of physical RAM memory in which the various parts of the kernel image reside. First page frame Kernel text Available Kernel dataROM Initialization data 0x9e800 _text _etext _end0x1000 (4 KiB)0x0 640 KiB 0x100000 (1 MiB) _edata Figure 3-11: Arrangement of the Linux kernel in RAM memory. The figure shows the first megabytes of physical memory — how much is exactly required depends on how big the kernel binary is. The first 4,096 KiB — the first page frame — are omitted because they are oftenreservedfortheBIOS.Thenext640KiBwouldbeusableinprinciple,butareagainnotusedfor kernel loading. The reason is that this area is immediately followed by an area reserved for the system into which various ROM ranges are mapped (typically the system BIOS and the graphic card ROM). It is not possible to write to these areas. However, the kernel should always be loaded into a contiguous memory range, and this would be possible only for kernels smaller than 640 KiB if the start address of RAM memory were used as the start position for the kernel image. To resolve these problems, IA-32 kernels use 0x100000 as the start address; this corresponds to the start of the first megabyte in RAM memory. There is sufficient contiguous memory at this point to hold the entire kernel. The memory occupied by the kernel is split into several sections whose bounds are held in variables. ❑ _text and _etext are the start and end address of the text section that contains the compiled kernel code. ❑ The data section in which most kernel variables are kept is located between _etext and _edata. ❑ Initialization data no longer needed after the kernel boot process is finished (among others, e.g., the BSS segment that contains all static global variables initialized to 0) are held in the last section, which extends from _edata to _end. Once kernel initialization has completed, most of the data can be removed from memory leaving more space for applications. The interval is split 170 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 171 Chapter 3: Memory Management into smaller subintervals to control what can be removed and what cannot, but this is not of importance for our purposes now. Although the variables used to define section bounds are defined in the kernel source code (arch/x86/kernel/setup_32.c), no values are assigned to them at this point. This is simply not possible. How can the compiler know at compilation time how large the kernel will be? The exact value is only established when the object files are linked, and it is then patched into the binary file. This action is controlled by arch/arch/vmlinux.ld.S (for IA-32, the file is arch/x86/vmlinux_32.ld.S), where the kernel memory layout is also defined. The exact value varies according to kernel configuration as each configuration has text and data sections of different sizes — depending on which parts of the kernel are enabled and which are not used. Only the start address (_text)isalwaysthesame. Each time the kernel is compiled, a file named System.map is generated and stored in the source base directory. Besides the addresses of all other (global) variables, procedures, and functions defined in the kernel, this file also includes the values of the constants shown in Figure 3-11, wolfgang@meitner> cat System.map ... c0100000 A _text ... c0381ecd A _etext ... c04704e0 A _edata ... c04c3f44 A _end ... All values have the offset 0xC0000000, which is the start address of the kernel segment if the standard 3 : 1 split between user and kernel address space is chosen. The addresses are virtual addresses because RAM memory is mapped into the virtual address space as a linear mapping starting at this address. The correspond- ing physical addresses are obtained by subtraction from 0xC0000000. /proc/iomem also provides information on the sections into which RAM memory is divided. wolfgang@meitner> cat /proc/iomem 00000000-0009e7ff : System RAM 0009e800-0009ffff : reserved 000a0000-000bffff : Video RAM area 000c0000-000c7fff : Video ROM 000f0000-000fffff : System ROM 00100000-17ceffff : System RAM 00100000-00381ecc : Kernel code 00381ecd-004704df : Kernel data ... The kernel image begins above the first megabyte (0x00100000). The size of the code is approximately 2.5 MiB, and the data section accounts for about 0.9 MiB. 171 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 172 Chapter 3: Memory Management The same information is also available for AMD64 systems. Here the kernel starts 2 MiB after the first page frame, and physical memory is mapped into the virtual address space from 0xffffffff80000000 onward. The relevant entries in System.map are as follows: wolfgang@meitner> cat System.map ffffffff80200000 A _text ... ffffffff8041fc6f A _etext ... ffffffff8056c060 A _edata ... ffffffff8077548c A _end This information is also contained in /proc/iomem for the running kernel: root@meitner # cat /proc/iomem ... 00100000-cff7ffff : System RAM 00200000-0041fc6e : Kernel code 0041fc6f-0056c05f : Kernel data 006b6000-0077548b : Kernel bss ... Initialization Steps Which system-specific steps must the kernel perform once it has been loaded into memory and the assem- bler parts of initialization have been completed? Figure 3-12 shows a code flow diagram of the individual actions. setup_arch machine_specific_memory_setup parse_early_param setup_memory paging_init pagetable_init zone_sizes_init add_active_range free_area_init_nodes Figure 3-12: Code flow diagram for memory initialization on IA-32 systems. 172 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 173 Chapter 3: Memory Management The figure includes only those function calls associated with memory management. All others are unimportant in this context and are therefore omitted. Recall that setup_arch is invoked from within start_kernel, as already noted in Section 3.4.1. machine_specific_memory_setup is first invoked to create a list with the memory regions occupied by the system and the free memory regions. Because the way in which this information is obtained differs slightly between the ‘‘subarchitectures‘‘ of the IA-32 family,11 the kernel provides a machine-specific function that is defined in include/asm-x86/mach-type/setup.c. type can then stand for default, voyager,orvisws; I discuss only the default situation. A map provided by the BIOS and showing the individual memory regions is used in this case. These are not the same regions as in the NUMA concept but are areas occupied by system ROM, for example, or by ACPI functions. When the system is booted, the regions found are displayed by the kernel function print_memory_map. wolfgang@meitner> dmesg ... BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 000000000009e800 (usable) BIOS-e820: 000000000009e800 - 00000000000a0000 (reserved) BIOS-e820: 00000000000c0000 - 00000000000cc000 (reserved) BIOS-e820: 00000000000d8000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 0000000017cf0000 (usable) BIOS-e820: 0000000017cf0000 - 0000000017cff000 (ACPI data) BIOS-e820: 0000000017cff000 - 0000000017d00000 (ACPI NVS) BIOS-e820: 0000000017d00000 - 0000000017e80000 (usable) BIOS-e820: 0000000017e80000 - 0000000018000000 (reserved) BIOS-e820: 00000000ff800000 - 00000000ffc00000 (reserved) BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved) ... If this information is not provided by the BIOS (this may be the case on some older machines), the kernel itself generates a table to mark memory in the ranges 0–640 KiB and 1 MiBend as usable. The kernel then analyzes the command line with parse_cmdline_early, concentrating on arguments like mem=XXX[KkmM], highmem=XXX[kKmM],ormemmap=XXX[KkmM]""@XXX[KkmM] arguments. The administrator can overwrite the size of available memory or manually define memory areas if the kernel calculates an incorrect value or is provided with a wrong value by the BIOS. This option is only of relevance on older computers. highmem= permits overwriting of the highmem size value detected. It can be used on machines with a very large RAM configuration to limit available RAM size — as it sometimes yields performance gains. 11There are not only ‘‘normal‘‘ IA-32 computers but also custom products of Silicon Graphics and NCR that, although they consist mainly of standard components, take a different approach to some things — including memory detection. Because these machines are either very old (Voyager from NCR) or not in widespread use (Visual Workstation from SGI), I won’t bother with their oddities. 173 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 174 Chapter 3: Memory Management The next major step is performed in setup_memory of which there are two versions; one for systems with contiguous memory (in arch/x86/kernel/setup_32.c) and one for machines with discontiguous mem- ory (in arch/x86/mm/discontig_32.c). They both have the same effect although their implementations differ. ❑ The number of physical pages available (per node) is determined. ❑ The bootmem allocator is initialized (Section 3.4.3 describes the implementation of the allocator in detail). ❑ Various memory areas are then reserved, for instance, for the initial RAM disk needed when running the first userspace processes. paging_init initializes the kernel page tables and enables paging since it is not active by default on IA-32 machines.12 Execute Disable Protection is also enabled if supported by the processor and if the ker- nel was compiled with PAE support; unfortunately, the feature is otherwise not available. By calling pagetable_init, the function also ensures that the direct mapping of physical memory into the kernel address space is initialized. All page frames in low memory are directly mapped to the virtual memory region above PAGE_OFFSET. This allows the kernel to address a good part of the available memory with- out having to deal with page tables anymore. More details about paging_init and the whole mechanism behind it are discussed below. Calling zone_sizes_init initializes the pgdat_t instances of all nodes of the system. First a com- paratively simple list of the available physical memory is prepared using add_active_range.The architecture-independent function free_are_init_nodes then uses this information to prepare the full-blown kernel data structures. Since this is a very important step that has numerous implications for how the kernel manages page frames at run time, it is discussed in more detail in Section 3.5.3. Notice that the memory-related initialization sequence is quite similar on AMD64 machines, as the code flow diagram in Figure 3-13 shows. The basic memory setup does not require any machine-type-specific handling, but can always be done with setup_memory_region. Information about the available RAM is given by the so-called E820 map supplied from the BIOS. After parsing the command-line options relevant for the early boot process, a simple list of the available memory is created by add_active called from e820_register_active_region, which, in turn, just walks over the information provided by parsing the E820 map above. The kernel then calls init_memory_mapping to directly map the available physical memory into the vir- tual address space portion of the kernel starting from PAGE_OFFSET. contig_initmem_init is responsible to activate the bootmem allocator. The last function in the list, paging_init, is actually a misnomer: It does not initialize paging, but has to deal with some set-up routines for sparse memory systems that are not interesting for our purposes. The important thing, however, is that the function also calls free_area_init_nodes, which is as in the IA-32 case responsible to initialize the data structures required to manage physical page frames by the kernel.Recallthatthisisanarchitecture-independent function and relies on the information provided by add_active_range as mentioned above. A detailed discussion of how free_area_init_nodes sets up memory follows in Section 3.5.3. 12All addresses are interpreted linearly if paging is not explicitly enabled. 174 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 175 Chapter 3: Memory Management setup_arch setup_memory_region parse_early_param e820_register_active_region add_active_range init_memory_mapping contig_initmem_init paging_init free_area_init_nodes Figure 3-13: Code flow diagram for memory initialization on AMD64 systems. Initialization of Paging paging_init is responsible for setting up the page tables that can be used only by the kernel and are not accessible in userspace. This has far-reaching effects on the way in which access to memory is regulated between normal applications and the kernel itself. It is therefore important to explain the purpose of the function before looking closely at its implementation. As noted in Chapter 1, on IA-32 systems the kernel typically divides the total available virtual address space of 4 GiB in a ratio of 3 : 1. The lower 3 GiB are available for user-mode applications, and the upper gigabyte is reserved exclusively for the kernel. Whereas the current system context is irrelevant when assigning the virtual address space of the kernel, each process has its own specific address space. The major reasons for this division are as follows: ❑ When execution of a user application switches to kernel mode (this always happens when, e.g., a system call is used or a periodic timer interrupt is generated), the kernel must be embedded in a reliable environment. It is therefore essential to assign part of the address space exclusively to the kernel. ❑ The physical pages are mapped to the start of the kernel address space so that the kernel can access them directly without the need for complicated page table operations. If all physical pages were mapped into the address space accessible to userspace processes, this would lead to serious security problems if several applications were running on the system. Each application would then be able to read and modify the memory areas of other processes in physical RAM. Obviously this must be prevented at all costs. While the virtual address portion employed for userland processes changes with every task switch, the kernel portion is always the same. The situation is summarized in Figure 3-14. 175 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 176 Chapter 3: Memory Management 1 Page tables Page tables Physical memory Task switch 3 Kernel User Kernel User Figure 3-14: Connection between virtual and physical address space on IA-32 processors. Division of Address Space Division of address space in a ratio of 3 : 1 is only an approximate reflection of the situation in the kernel as the kernel address space itself is split into various sections. Figure 3-15 graphically illustrates the situation. high_memory VMALLOC_START VMALLOC_END _ _PAGE_OFFSET 0xC000000 8 MiB PKMAP_BASE FIXADDR_START 4 GiB Mapping of all physical page frames VMALLOC Persistent Mappings Fixmaps Figure 3-15: Division of the kernel address space on IA-32 systems. The figure shows the structure of the page table entries used to manage the fourth gigabyte of virtual address space. It indicates the purpose of each area of virtual address space, and this has nothing to do with the assignment of physical RAM. The first section of the address space is used to map all physical pages of the system into the virtual address space of the kernel. Because this address space begins at an offset of 0xC0000000 —thefre- quently mentioned 3 GiB — each virtual address x corresponds to the physical address x − 0xC0000000, and is therefore a simple linear shift. As the figure shows, the direct mapping area extends from 0xC0000000 to the high_memory address whose exact value I discuss shortly. As intimated in Chapter 1, there is a problem with this scheme. Because the virtual address space of the kernel comprises only 1 GiB, a maximum of 1 GiB of RAM memory can be mapped. The fact that the maximum memory configuration on IA-32 systems (without PAE) can be up to 4 GiB raises the question of what to do with the remaining memory. Here’s the bad news. The kernel cannot map the whole of physical memory at once if it is larger than 896 MiB.13 This value is even less than the previously stated maximum limit of 1 GiB because 13It would also be possible to get rid of the split completely by introducing two 4 GiB address spaces, one for the kernel and one for each userspace program. However, context switches between kernel and user mode are more costly in this case. 176 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 177 Chapter 3: Memory Management the kernel must reserve the last 128 MiB of its address space for other purposes which I explain shortly. Adding these 128 MiB to the 896 MiB of direct RAM mapping results in a total virtual kernel address space of 1,024 MiB = 1 GiB. The kernel uses the two frequently employed abbreviations ‘‘normal‘‘ and ‘‘highmem‘‘to distinguish between pages that can be mapped directly and those than cannot. The kernel port must provide two macros for each architecture to translate between physical and virtual addresses in the identity-mapped part of virtual kernel memory (ultimately this is a platform-dependent task).14 ❑ __pa(vaddr) returns the physical address associated with the virtual address vaddr. ❑ __va(paddr) yields the virtual address corresponding to the physical address paddr. Both functions operate with void pointers and with unsigned longs because both data types are equally valid for the representation of memory addresses. Caution: The functions are not valid to deal with arbitrary addresses from the virtual address space, but only work for the identity-mapped part! This is why they can usually be implemented with simple linear transformations and do not require a detour over the page tables. IA-32 maps the page frames into the virtual address space starting from PAGE_OFFSET, and correspond- ingly the following simple transformation is sufficient: include/asm-x86/page_32.h #define __pa(x) ((unsigned long)(x)-PAGE_OFFSET) #define __va(x) ((void *)((unsigned long)(x)+PAGE_OFFSET)) For what purpose does the kernel use the last 128 MiB of its address space? As Figure 3-15 shows, it is put to three uses: 1. Virtually contiguous memory areas that are not contiguous in physical memory can be reserved in the vmalloc area. While this mechanism is commonly used with user processes, the kernel itself tries to avoid non-contiguous physical addresses as best it can. It usually succeeds because most of the large memory blocks are allocated for the kernel at boot time when RAM is not yet fragmented. However, on systems that have been running for longer periods, situations can arise in which the kernel requires physical memory but the space available is not contiguous. A prime example of such a situation is when modules are loaded dynamically. 2. Persistent mappings are used to map non-persistent pages from the highmem area into the kernel. Section 3.5.8 takes a close look at this topic. 3. Fixmaps are virtual address space entries associated with a fixed but freely selectable page in physical address space. In contrast to directly mapped pages that are associated with RAM memory by means of a fixed formula, the association between a virtual fixmap address and the position in RAM memory can be freely defined and is then always observed by the kernel. 14The kernel places only two conditions on the functions that must remain as invariants; x1 < x2 ⇒ __va(x1) < __va(x2) must be valid (for any physical addresses xi), and __va(__pa(x)) = x must be valid for any addresses x within the direct mapping. 177 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 178 Chapter 3: Memory Management Two pre-processor symbols are important in this context: __VMALLOC_RESERVE sets the size of the vmalloc area, and MAXMEM denotes the maximum possible amount of physical RAM that can be directly addressed by the kernel. The splitting of memory into the individual areas is controlled by means of the constants shown in Figure 3-15. The constants may have different values depending on the kernel and system configuration. The bound of the direct mappings is specified by high_memory. arch/x86/kernel/setup_32.c static unsigned long __init setup_memory(void) { ... #ifdef CONFIG_HIGHMEM high_memory = (void *) __va(highstart_pfn * PAGE_SIZE - 1) + 1; #else high_memory = (void *) __va(max_low_pfn * PAGE_SIZE - 1) + 1; #endif ... } max_low_pfn specifies the number of memory pages present on systems with less than 896 MiB. The value is also limited upwards to the maximum number of pages that fit in 896 MiB (the exact calculation is given in find_max_low_pfn). If highmem support is enabled, high_memory indicates the bound between two memory areas, which is always at 896 MiB. There is a gap with a minimum size of VMALLOC_OFFSET between the direct mapping of all RAM pages and the area for non-contiguous allocations. include/asm-x86/pgtable_32.h #define VMALLOC_OFFSET (8*1024*1024) This gap acts as a safeguard against any kernel faults. If out of bound addresses are accessed (these are unintentional accesses to memory areas that are no longer physically present), access fails and an excep- tion is generated to report the error. If the vmalloc area were to immediately follow the direct mappings, access would be successful and the error would not be noticed. There should be no need for this addi- tional safeguard in stable operation, but it is useful when developing new kernel features that are not yet mature. VMALLOC_START and VMALLOC_END define the start and end of the vmalloc area used for physically non- contiguous kernel mappings. The values are not defined directly as constants but depend on several parameters. include/asm-x86/pgtable_32.h #define VMALLOC_START (((unsigned long) high_memory + \ 2*VMALLOC_OFFSET-1) & ~(VMALLOC_OFFSET-1)) #ifdef CONFIG_HIGHMEM # define VMALLOC_END (PKMAP_BASE-2*PAGE_SIZE) #else # define VMALLOC_END (FIXADDR_START-2*PAGE_SIZE) #endif 178 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 179 Chapter 3: Memory Management The start address of the vmalloc area depends on how much virtual address space memory is used for the direct mapping of RAM (and therefore on the high_memory variable defined above). The kernel also takes account of the fact that there is a gap of at least VMALLOC_OFFSET between the two areas and that the vmalloc area begins at an address divisible by VMALLOC_OFFSET. This results in the offset values shown in Table 3-5 for different memory configuration levels between 128 and 135 MiB; the offsets start a new cycle at 136 MiB. Table 3-5: VMALLOC_OFFSET Values for Different RAM Sizes Memory (MiB) Offset (MiB) 128 8 129 15 130 14 131 13 132 12 133 11 134 10 135 9 Where the vmalloc area ends depends on whether highmem support is enabled — if it is not, no space is needed for persistent kernel mappings because the whole of RAM memory can be permanently mapped. Depending on configuration, the area therefore ends either at the start of the persistent kernel mappings or at the start of the fixmap area; two pages are always left as a safety gap to the vmalloc area. The start and end of the persistent kernel mappings are defined as follows: include/asm-x86/highmem.h #define LAST_PKMAP 1024 #define PKMAP_BASE ( (FIXADDR_BOOT_START - PAGE_SIZE*(LAST_PKMAP + 1)) & PMD_MASK ) PKMAP_BASE defines the start address (the calculation is made relative to the fixmap area using some constants that are discussed shortly). LAST_PKMAP defines the number of pages used to hold the mappings. The last memory section is occupied by fixed mappings. These are addresses that point to a random location in RAM memory. In contrast to linear mapping at the start of the fourth gigabyte, the correlation between virtual address and position in RAM memory is not preordained with this type of mapping but can be freely defined, even though it cannot be changed later. The fixmap area fills the virtual address space right up to its top end. 179 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 180 Chapter 3: Memory Management include/asm-x86/fixmap_32.h #define __FIXADDR_TOP 0xfffff000 #define FIXADDR_TOP ((unsigned long)__FIXADDR_TOP) #define __FIXADDR_SIZE (__end_of_permanent_fixed_addresses << PAGE_SHIFT) #define FIXADDR_START (FIXADDR_TOP - __FIXADDR_SIZE) The advantage of fixmap addresses is that at compilation time, the address acts like a constant whose physical address is assigned when the kernel is booted. Addresses of this kind can be de-referenced faster than when normal pointers are used. The kernel also ensures that the page table entries of fixmaps are not flushed from the TLB during a context switch so that access is always made via fast cache memory. A constant is created for each fixmap address and must appear in the enum list called fixed_addresses. include/asm-x86/fixmap_32.h enum fixed_addresses { FIX_HOLE, FIX_VDSO, FIX_DBGP_BASE, FIX_EARLYCON_MEM_BASE, #ifdef CONFIG_X86_LOCAL_APIC FIX_APIC_BASE, /* local (CPU) APIC) — required for SMP or not */ #endif ... #ifdef CONFIG_HIGHMEM FIX_KMAP_BEGIN, /* reserved pte’s for temporary kernel mappings */ FIX_KMAP_END = FIX_KMAP_BEGIN+(KM_TYPE_NR*NR_CPUS)-1, #endif ... FIX_WP_TEST, __end_of_fixed_addresses }; The kernel provides the fix_to_virt function to calculate the virtual address of a fixmap constant. include/asm-x86/fixmap_32.h static __always_inline unsigned long fix_to_virt(const unsigned int idx) { if (idx >= __end_of_fixed_addresses) __this_fixmap_does_not_exist(); return __fix_to_virt(idx); } The if query is totally removed by compiler optimization mechanisms — this is possible because the function is defined as an inline function, and only constants are used in the query. Such optimization is necessary because otherwise fixmap addresses would be no better than normal pointers. A formal check is made to ensure that the required fixmap address is in the valid area. __end_of_fixed_adresses is the last element of fixed_addresses and defines the maximum possible number. The pseudo-function __this_fixmap_does_not_exist (for which no definition exists) is invoked if the kernel accesses an invalid address. When the kernel is linked, this leads to an error message indicating that no image can be generated because of undefined symbols. Consequently, kernel faults of this kind are detected at compilation time and not when the kernel is running. 180 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 181 Chapter 3: Memory Management When a valid fixmap address is referenced, the comparison in the if query yields a positive value. Since both comparison objects are constants, the query need not be executed and is therefore removed. __fix_to_virt is defined as a macro. Owing to the inline property of fix_to_virt, it is copied directly to the point in the code where the fixmap address query is executed. This macro is defined as follows: include/asm-x86/fixmap_32.h #define __fix_to_virt(x) (FIXADDR_TOP - ((x) << PAGE_SHIFT)) Starting at the top (and not from the bottom as usual), the kernel goes back n pages to determine the virtual address of the n-th fixmap entry. As, once again, only constants are used in this calculation, the compiler is able to compute the result at compilation time. The address in RAM at which the corresponding virtual address is located has not yet been occupied as a result of the above division of memory. The association between the fixmap address and physical page in memory is established by set_fixmap(fixmap, page_nr) and set_fixmap_nocache (whose implementation is not discussed). They simply associate the corresponding entry in the page tables with a page in RAM. Unlike set_fixmap, set_fixmap_nocache disables hardware caching for the page involved as this is sometimes necessary. Notice that some other architectures also provide fixmaps, including AMD64. Alternative Division Dividing virtual address space in a 3 : 1 ratio is not the only option. Relatively little effort is needed to select a different division because all bounds are defined by constants in the sources. For some purposes it may be better to split the address space symmetrically, 2 GiB for user address space and 2 GiB for kernel address space. __PAGE_OFFSET must then be set to 0x80000000 instead of the typical default of 0xC0000000. This division is useful when the system performs tasks that require a large amount of mem- ory for the kernel but little for the user processes (such tasks are rare). As any change to how memory is divided requires recompilation of all userspace applications, the configuration statements include no option to split memory differently, although this would be easy to do in principle. Basically, it is possible to split memory by manually modifying the kernel sources, but the kernel offers some default splitting ratios. __PAGE_OFFSET is then defined as follows: include/asm-x86/page_32.h #define __PAGE_OFFSET # ((unsigned long)CONFIG_PAGE_OFFSET) Table 3-6 collects all possibilities for splitting the virtual address space and the resulting maximal amount of RAM that can be mapped. Splitting the kernel in ratios other than 3 : 1 can make sense in specific scenarios, for instance, for machines that mainly run code in the kernel — think about network routers. The general case, however, is best served with a 3 : 1 ratio. Splitting the Virtual Address Space paging_init is invoked on IA-32 systems during the boot process to split the virtual address space as described above. The code flow diagram is shown in Figure 3-16. 181 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 182 Chapter 3: Memory Management Table 3-6: Different Splitting Ratios for the IA-32 Virtual Address Space, and the Resulting Maximum Identity-Mapped Physical Memory. Ratio CONFIG_PAGE_OFFSET MAXMEM(MiB) 3:1 0xC0000000 896 ≈ 3:1 0xB0000000 1152 2:2 0x80000000 1920 ≈ 2:2 0x78000000 2048 1:3 0x40000000 2944 PAE enabled? Use PSE, PGE extensions if possible Initialize fixmaps paging_init set_nx pagetable_init kernel_physical_mapping_init permanent_kmaps_init load_cr3 __flush_all_tlb kmap_init Figure 3-16: Code flow diagram for paging_init. pagetable_init first initializes the page tables of the system using swapper_pg_dir as a basic (this variable was previously used to hold the provisional data). Two extensions available on all modern IA-32 variants are then enabled (only a few very old Pentium implementations do not support these). ❑ Support for large memory pages. The size of specially marked pages is 4 MiB instead of the usual 4 KiB. This option is used for kernel pages because they are never swapped out. Increasing the page size means that fewer page table entries are needed, and this has a positive impact on the translation lookaside buffers (TLBs), which are then less burdened with kernel data. ❑ If possible, kernel pages are provided with a further attribute (_PAGE_GLOBAL) that is why the __PAGE_GLOBAL bit is activated in the __PAGE_KERNEL and __PAGE_KERNEL_EXEC variables. These 182 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 183 Chapter 3: Memory Management variables specify the flags set for the kernel itself when pages are allocated; these settings are therefore automatically transferred to the kernel pages. The TLB entries of pages with a set _PAGE_GLOBAL bit are not flushed from the TLBs during con- text switches. Since the kernel is always present at the same location in the virtual address space, this enhances system performance, a welcome effect as kernel data must be made available as quickly as possible. Mapping of the physical pages (or of the first 896 MiB, as discussed above) into virtual address space as of PAGE_OFFSET is done with the help of kernel_physical_mapping_init.Thekernel successively scans all relevant entries of the various page directories and sets the pointers to the correct values. Then the areas for fixmap entries and the persistent kernel mappings are set up. Again, this equates to filling the page tables with appropriate values. Once page table initialization with pagetable_init has been concluded, the cr3 register is supplied with a pointer to the page global directory used (swapper_pg_dir). This is necessary to activate the new page tables. Reassigning the cr3 register has exactly this effect on IA-32 machines. The TLB entries must also be flushed because they still contain boot memory allocation data. __flush_all_tlb does the necessary work. In contrast to TLB flushes during context switches, pages with a _PAGE_GLOBAL bit are also flushed. kmap_init initializes the global variable kmap_pte. The kernel uses this variable to store the page table entry for the area later used to map pages from the highmem zone into kernel address space. Besides, the address of the first fixmap area for highmem kernel mappings is stored in the global variable kmem_vstart. Initialization of the Hot-n-Cold Cache I have already mentioned the per-CPU (or hot-n-cold) cache in Section 3.2.2.. Here we deal with the initialization of the associated data structures and the calculation of the ‘‘watermarks‘‘used to control cache filling behavior. zone_pcp_init is responsible for initializing the cache. The kernel calls the function from free_area_init_nodes, which is, in turn, invoked during boot on both IA-32 and AMD64. mm/page_alloc.c static __devinit void zone_pcp_init(struct zone *zone) { int cpu; unsigned long batch = zone_batchsize(zone); for (cpu = 0; cpu < NR_CPUS; cpu++) { setup_pageset(zone_pcp(zone,cpu), batch); } if (zone->present_pages) printk(KERN_DEBUG " %s zone: %lu pages, LIFO batch:%lu\n", zone->name, zone->present_pages, batch); } 183 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 184 Chapter 3: Memory Management Once the batch size (which is the basis for calculating the minimum and maximum fill level) has been determined with zone_batchsize, the code iterates over all CPUs in the system and invokes setup_pageset to fill the constants of each per_cpu_pageset instance. The zone_pcp macro used when this function is invoked selects the pageset instance of the zone associated with the CPU currently being examined. Let us take a closer look at how the watermark is calculated. mm/page_alloc.c static int __devinit zone_batchsize(struct zone *zone) { int batch; batch = zone->present_pages / 1024; if (batch * PAGE_SIZE > 512 * 1024) batch = (512 * 1024) / PAGE_SIZE; batch /= 4; if (batch < 1) batch = 1; batch = (1 << (fls(batch + batch/2)-1)) - 1; return batch; } The code calculates batch so that it corresponds to roughly 25 percent of a thousandth of the pages present in the zone. The shift operation also ensures that the value calculated has the form 2n − 1because it has been established empirically that this minimizes cache aliasing effects for most system loads. fls is a machine-specific operation to yield the last set bit of a value. Note that this alignment will cause the resulting values to deviate from 25 percent of one-thousandth the zones pages. The maximal deviation arises for that case batch = 22.Since22+ 11 − 1 = 32, fls will find bit 5 as last set bit in the number, and 1<<5-1=31. Because the deviation will usually be smaller, it can be neglected for all practical purposes. The batch size does not increase when the memory in the zone exceeds 512 MiB. For systems with a page size of 4,096 KiB, for instance, this limit is reached when more than 131,072 pages are present. Figure 3-17 shows how the batch size evolves with the number of pages present in a zone. The batch value makes sense when we consider how batch is used to calculate the cache limits in setup_pageset. mm/page_alloc.c inline void setup_pageset(struct per_cpu_pageset *p, unsigned long batch) { struct per_cpu_pages *pcp; memset(p, 0, sizeof(*p)); pcp = &p->pcp[0]; /* hot */ pcp->count = 0; pcp->high = 6 * batch; pcp->batch = max(1UL, 1 * batch); INIT_LIST_HEAD(&pcp->list); 184 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 185 Chapter 3: Memory Management pcp = &p->pcp[1]; /* cold*/ pcp->count = 0; pcp->high = 2 * batch; pcp->batch = max(1UL, batch/2); INIT_LIST_HEAD(&pcp->list); } 0 5 10 15 20 25 30 35 0 100 200 300 400 500 600 700 Batch size Zone memory [MiB] 4 KiB Pages 8 KiB Pages 16 KiB Pages 0 5 10 15 20 25 30 35 0 20000 40000 60000 80000 100000 120000 140000 160000 180000 200000 Batch size Pages in Zone Figure 3-17: Batch sizes dependent on the amount of memory present (left-hand side) on the zone for various page sizes. The graph on the right-hand side shows the dependency against the number of pages present in a zone. As the lower limit used for hot pages is 0 and the upper limit is 6*batch, the average number of pages in the cache will be around 4*batch because the kernel tries to not let the caches drain too much. batch*4, however, corresponds to a thousandth of the total number of zone pages (this is also the reason why zone_batchsize tried to optimize the batch size for 25 percent of one-thousandth of the total pages). The size of the L2 cache on IA-32 processors is in the range between 0.25 and 2 MiB, so it makes no sense to keep much more memory in a hot-n-cold cache than would fit into this space. As a rule of thumb, the cache size is one-thousandth of the main memory size; consider that current systems are equipped with between 1 and 2 GiB of RAM per CPU, so the rule is reasonable. The computed batch size will thus likely allow that the pages on the hot-n-cold cache fit into the CPU’s L2 cache. The watermarks of the cold list are slightly lower because cold pages not held in the cache are used only for actions that are not performance-critical (such actions are, of course, in the minority in the kernel,). Only double of the batch value is used as the upper limit. The pcp->batch size determines how many pages are used at once when the list needs to be refilled. For performance reasons, a whole chunk of pages rather than single pages is added to the list. The number of pages in each zone is output at the end of zone_pcp_init together with the calculated batch sizes as shown in the boot logs (for a system with 4 GiB of RAM in the example below). root@meitner # dmesg | grep LIFO DMA zone: 2530 pages, LIFO batch:0 DMA32 zone: 833464 pages, LIFO batch:31 Normal zone: 193920 pages, LIFO batch:31 185 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 186 Chapter 3: Memory Management Registering Active Memory Regions I noted above that initialization of the zone structures is an extensive task. Luckily this task is identical on all architectures. While kernel versions before 2.6.19 had to set up the required data structures on a per- architecture basis, the approach has become more modular in the meantime: The individual architectures only need to register a very simple map of all active memory regions, and generic code then generates the main data structures from this information. Notice that individual architectures can still decide to set up all data structures on their own without relying on the generic framework provided by the kernel. Since both IA-32 and AMD64 let the kernel do the hard work, I will not discuss this possibility any further. Any architecture that wants to enjoy the pos- sibilities offered by the generic framework must set the configuration option ARCH_POPULATES_NODE_MAP. After all active memory regions are registered, the rest of the work is then performed by the generic kernel code. An active memory region is simply a memory region that does not contain any holes. add_active_range must be used to register a region in the global variable early_node_map. mm/page_alloc.c static struct node_active_region __meminitdata early_node_map[MAX_ACTIVE_REGIONS]; static int __meminitdata nr_nodemap_entries; The number of currently registered regions is denoted by nr_nodemap_entries. The maximal number of distinct regions is given by MAX_ACTIVE_REGIONS. The value can be set by the architecture-specific code using CONFIG_MAX_ACTIVE_REGIONS. If not, the kernel allows for registering 256 active regions per default (or 50 regions per NUMA node if it is running on a system with more than 32 nodes). Each region is described by the following data structure: struct node_active_region { unsigned long start_pfn; unsigned long end_pfn; int nid; }; start_pfn and end_pfn denote the first and last page frame in a continuous region, and nid is the NUMA ID of the node to which the memory belongs. UMA systems naturally set this to 0. An active memory region is registered with add_active_range: mm/page_alloc.c void __init add_active_range(unsigned int nid, unsigned long start_pfn, unsigned long end_pfn) When two adjacent regions are registered, then add_active_regions ensures that they are merged to a single one. Besides, the function does not present any surprises. Recall from Figures 3-12 and 3-13 that the function is called from zone_sizes_init on IA-32 systems, and in e820_register_active_regions onAMD64systems.ThusIwillbrieflydiscussthese functions. 186 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 187 Chapter 3: Memory Management Registering Regions on IA-32 Besides calling add_active_range, the function zone_sizes_init stores the boundaries of the different memory zones in terms of page frames. arch/x86/kernel/setup_32.c void __init zone_sizes_init(void) { unsigned long max_zone_pfns[MAX_NR_ZONES]; memset(max_zone_pfns, 0, sizeof(max_zone_pfns)); max_zone_pfns[ZONE_DMA] = virt_to_phys((char *)MAX_DMA_ADDRESS) >> PAGE_SHIFT; max_zone_pfns[ZONE_NORMAL] = max_low_pfn; #ifdef CONFIG_HIGHMEM max_zone_pfns[ZONE_HIGHMEM] = highend_pfn; add_active_range(0, 0, highend_pfn); #else add_active_range(0, 0, max_low_pfn); #endif free_area_init_nodes(max_zone_pfns); } MAX_DMA_ADDRESS is the highest suitable memory address for DMA operations. The constant is declared as PAGE_OFFSET+0x1000000. Recall that the physical pages are mapped into the virtual starting from PAGE_OFFSET, and the first 16 MiB — hexadecimal 0x1000000 — are suitable for DMA operations. Conversion with virt_to_phys yields the address in physical memory, and shifting right by PAGE_SHIFT bits effectively divides this figure by the page size and produces the number of pages that can be used for DMA. Unsurprisingly, the result is 4,096 since IA-32 uses pages of 4 KiB. max_low_pfn and highend_pfn are global constants to specify the highest page number in the low (usually ≤ 896 MiB if 3 : 1 split of the address space is used) and high memory ranges that were filled before. Notice that free_area_init_nodes will combine the information in early_mem_map and max_zone_pfns: The active ranges for each memory region are selected, and architecture-independent data structures are constructed. Registering Regions on AMD64 Registering the available memory is split between two functions on AMD64. The active memory regions are registered as follows: arch/x86/kernel/e820_64.c e820_register_active_regions(int nid, unsigned long start_pfn, unsigned long end_pfn) { unsigned long ei_startpfn; unsigned long ei_endpfn; int i; 187 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 188 Chapter 3: Memory Management for (i = 0; i < e820.nr_map; i++) if (e820_find_active_region(&e820.map[i], start_pfn, end_pfn, &ei_startpfn, &ei_endpfn)) add_active_range(nid, ei_startpfn, ei_endpfn); } Essentially the code iterates over all regions provided by the BIOS and finds the active region for each entry. This is interesting because add_active_range is potentially called multiple times in contrast to the IA-32 variant. Filling in max_zone_pfns is handled by paging_init: arch/x86/mm/init_64.c void __init paging_init(void) { unsigned long max_zone_pfns[MAX_NR_ZONES]; memset(max_zone_pfns, 0, sizeof(max_zone_pfns)); max_zone_pfns[ZONE_DMA] = MAX_DMA_PFN; max_zone_pfns[ZONE_DMA32] = MAX_DMA32_PFN; max_zone_pfns[ZONE_NORMAL] = end_pfn; ... free_area_init_nodes(max_zone_pfns); } The page frame boundaries for the 16- and 32-bit DMA regions are stored in pre-processor symbols that translate the 16 MiB and 4 GiB ranges into page frames: include/asm-x86/dms_64.h /* 16MB ISA DMA zone */ #define MAX_DMA_PFN ((16*1024*1024) >> PAGE_SHIFT) /* 4GB broken PCI/AGP hardware bus master zone */ #define MAX_DMA32_PFN ((4UL*1024*1024*1024) >> PAGE_SHIFT) end_pfn is the largest page frame number detected. Since AMD64 does not require high memory, the corresponding entry in max_zone_pfns remains NULL. Address Space Setup on AMD64 The address space setup on AMD64 systems is easier than for IA-32 in some respects, but unfortunately also harder in others. While having a 64-bit virtual address space allows for avoiding oddities like high memory, things are complicated by another factor: The address space spanned by 64 bits is so large that there are currently simply no applications that would require this. Current implementations there- fore implement a smaller physical address space that is only 48 bits wide. This allows for simplifying and speeding up address translation without losing flexibility: 248 bits still allows addressing 256 TiB, or 256 × 1,024 GiB — which is plenty even for Firefox! While the physical address space is restricted to 48 bits, addressing the virtual address space is still per- formed with 64-bit pointers, and the space therefore has to span 64 bits formally. This raises a problem, though: Some parts of the virtual address space cannot be addressed because effectively only 48 bits can be handled. 188 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 189 Chapter 3: Memory Management Since future hardware implementations might support larger physical address spaces, it is not possible to simply to remap the subset that is not addressable to a different subset of the address space. Suppose that any program would rely on pointers into the unimplemented address space to be remapped to some part of the regular address space. Next-generation processors that implement more physical address bits would lead to a different behavior and thus break all existing code. Clearly, accessing the unimplemented regions must be hindered by the processor. One possibility to enforce this would be to forbid use of all virtual addresses larger than the physical address space. This, however, is not the approach chosen by the hardware designers. Their solution is based on a sign extension approach, which is illustrated in Figure 3-18. Higher half Lower half Kernelspace o × FFFF Userspace o 264 FFFF FFFF FFFF o × 0000 7FFF FFFF FFFF Bits [0,46] arbitrary [47, 63] not set Bits [0,46] arbitrary [47, 63] not seto × FFFF Non-canonical area 8000 0000 0000 Figure 3-18: Possible virtual versus implemented physical address space on AMD64 machines. The first 47 bits of a virtual address, that is, [0, 46], can be arbitrarily set. Bits in the range [47, 63], however, always need to have the same value: Either all are 0, or all are 1. Such addresses are called canonical.They divide the total address space into three parts: a lower half, a higher half, and a forbidden region in between. Together both portions form an address space that spans exactly 248 bits. The address space for the lower half is [0x0, 0x0000 7FFF FFFF FFFF], while the subset for the top half is [0xFFF 800 0000 0000, 0xFFFF FFFF FFFF FFFF]. Notice that 0x0000 7FFF FFFF FFFF is a binary number with the lower 47 bits set to 1 and all other bits not set, so it is the last address before the non-addressable region. Similarly, 0xFFFF 8000 0000 0000 has the bits [48, 63] set and is thus the first valid address in the higher half. Partitioning the virtual address space into two parts is nothing the kernel is afraid of: It actually relies on a separation of the address space into kernel and user parts on most architectures.15 The separation enforced by the AMD64 therefore lends itself naturally to implement the separation between user and kernel address space. Figure 3-19 shows how the Linux kernel lays out the virtual address space on AMD64 machines.16 The complete lower half of the accessible address space is used as userspace, while the complete upper half is reserved for the kernel. Since both spaces are huge, no fiddling with splitting ratios and the like is required. The kernel address space starts with a guard hole to prevent incidental access on the non-canonical portion of the address space that would result in a general protection exception raised by the processor. Physical pages are identity-mapped into kernel space starting from PAGE_OFFSET.246 bits (as specified by MAXMEM) are reserved for physical page frames. This amounts to 16 TiB of memory. 15There are also machines that allow a different approach. UltraSparc processors provide different virtual address spaces for user and kernel space per default, so a separation of one address space into two components is not required. 16The kernel sources contain some documentation about the address space layout in Documentation/x86_64/mm.txt. 189 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 190 Chapter 3: Memory Management include/asm-x86/pgtable_64.h #define __AC(X,Y) (X##Y) #define _AC(X,Y) __AC(X,Y) #define __PAGE_OFFSET _AC(0xffff810000000000, UL) #define PAGE_OFFSET __PAGE_OFFSET #define MAXMEM _AC(0x3fffffffffff, UL) Modules MODULES_ENDMODULES_VADDR_START_KERNEL_START_KERNEL_MAPKERNEL_TEXT_STARTVMMEMMAP_STARTVMALLIC_ENDVMALLOC_STARTPAGE_OFFSETTASK_SIZE_64 Ox FFFF 8000 0000 0000 Identity maped pages Hole (unused) KERNEL_TEXT_SIZE 212 Bits 246 0 Bits (MAXMEM)Non-canonical area Figure 3-19: Organization of the virtual address space on AMD64 systems. The image is not drawn to scale, naturally. Note that _AC is used to mark a given constant with a suffix. _AC(17,UL) becomes (17UL), for instance, which makes the constant an unsigned long. This can be handy in C code, but is not allowed in assembler code, where the _AC macro directly resolves to the given value without postfix. Another guard hole is placed between the identity-mapped region and the area for vmalloc area, which lies between VMALLOC_START and VMALLOC_END: include/asm-x86/pgtable_64.h #define VMALLOC_START _AC(0xffffc20000000000, UL) #define VMALLOC_END _AC(0xffffe1ffffffffff, UL) The virtual memory map (VMM) area immediately behind the vmalloc area is 1 TiB in size. It is only useful on kernels that use the sparse memory model. Converting between virtual and physical page frame number via pfn_to_page and page_to_pfn can be costly on such machines because all holes in the physical address space must be taken into account. Starting with kernel 2.6.24, a simpler solution is offered by generic code in mm/sparse-memmap.c: The page tables for the VMM area are set up such that all struct page instances located in physical memory are mapped into the area without any holes. This provides a virtually contiguous area in which only the active memory regions are included. The MMU therefore automatically aids the translation between virtual and physical numbers that does not need to be concerned with holes anymore. This accelerates the operation considerably. Besides simplifying the translation between physical and virtual page numbers, the technique also has benefits for the implementation of the auxiliary functions virt_to_page and page_address, because the required calculations are likewise simplified. The kernel text is mapped into the region starting from __START_KERNEL_MAP, with a compile-time con- figurable offset given by CONFIG_PHYSICAL_START. Setting the offset is required for a relocatable kernel, 190 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 191 Chapter 3: Memory Management but it is ensured that the resulting address __START_KERNEL is aligned with __KERNEL_ALIGN.Theregion reserved for the kernel binary is KERNEL_TEXT_SIZE, currently defined to 40 MiB. include/asm-x86/page_64.h #define __PHYSICAL_START CONFIG_PHYSICAL_START #define __KERNEL_ALIGN 0x200000 #define __START_KERNEL (__START_KERNEL_map + __PHYSICAL_START) #define __START_KERNEL_map _AC(0xffffffff80000000, UL) #define KERNEL_TEXT_SIZE (40*1024*1024) #define KERNEL_TEXT_START _AC(0xffffffff80000000, UL) Finally, some space to map modules into must be provided, and this is in the region from MODULES_VADDR to MODULES_END: include/asm-x86/pgtable_64.h #define MODULES_VADDR _AC(0xffffffff88000000, UL) #define MODULES_END _AC(0xfffffffffff00000, UL) #define MODULES_LEN (MODULES_END - MODULES_VADDR) The available amount of memory is computed in MODULES_LEN; currently, this amounts to approximately 1,920 MiB. 3.4.3 Memory Management during the Boot Process Although memory management is not yet initialized, the kernel needs to reserve memory during the boot process so that it can create various data structures. A bootmem allocator that assigns memory in the early boot phase is used to do this. Obviously, what is required is a system that focuses on simplicity rather than on performance and univer- sality. Kernel developers therefore decided to implement a first-fit allocator as the simplest conceivable way of managing memory in the boot phase. A bitmap with (at least) as many bits as there are physical pages present in the system is used to manage pages. Bit value 1 indicates a used page and 0 a free page. When memory needs to be reserved, the allocator scans the bitmap bit by bit until it finds a posi- tion big enough to hold a sufficient number of contiguous pages, literally the first-best or first-fit position. This procedure is not very efficient because the bit chain must be scanned right from the start for each allocation. It cannot therefore be used for memory management once the kernel has been fully initialized. The buddy system (used in conjunction with the slab, slub, or slob allocator) is a far better alternative, as discussed in Section 3.5.5. Data Structures Even the first-fit allocator has to manage some data. The kernel provides an instance of the bootmem_data structure (for each node in the system) for this purpose. Of course, the memory needed for the struc- ture cannot be reserved dynamically but must already be allocated to the kernel at compilation time. 191 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 192 Chapter 3: Memory Management Reservation is implemented in a CPU-independent way on UMA systems (NUMA systems employ architecture-specific solutions). The bootmem_data structure is defined as follows: typedef struct bootmem_data { unsigned long node_boot_start; unsigned long node_low_pfn; void *node_bootmem_map; unsigned long last_offset; unsigned long last_pos; unsigned long last_success; struct list_head list; } bootmem_data_t; When I use the term page below, I always mean a physical page frame. ❑ node_boot_start holds the number of the first page in the system; this is zero for most architec- tures. ❑ node_low_pfn is the number of the last page of the physical address space that can be managed directly; in other words, it is the end of ZONE_NORMAL. ❑ node_bootmem_map is a pointer to the memory area in which the allocation bitmap is stored. On IA-32 systems, the memory area immediately following the kernel image is used for this pur- pose. The corresponding address is held in the _end variable, which is automatically patched into the kernel image during linking. ❑ last_pos is the number of the page last allocated. last_offset is used as an offset within the page if not all of the page memory was requested. This enables the bootmem allocator to assign memory areas that are smaller than a complete page (the buddy system cannot do this). ❑ last_success specifies the point in the bitmap at which allocation was last successful and is used as the starting point for new reservations. Although this makes the first-fit algorithm a little faster, it is still no real substitute for more sophisticated techniques. ❑ Systems with discontinuous memory can require more than one bootmem allocator. This is typ- ically the case on NUMA machines that register one bootmem allocator per node, but it would, for instance, also be possible to register one bootmem allocator for each continuous memory region on systems where the physical address space is interspersed with holes. A new boot allocator is registered with init_bootmem_core, and the list of all registered alloca- tors is headed by the global variable bdata_list. On UMA systems, the single bootmem_t instance required is called contig_bootmem_data. It is associated with contig_page_data by means of the bdata element. mm/page_alloc.c static bootmem_data_t contig_bootmem_data; struct pglist_data contig_page_data = { .bdata = &contig_bootmem_data }; Initialization Initializing the bootmem allocator is an architecture specific process that additionally depends on the memory layout of the machine in question. As discussed above, IA-32 uses setup_memory, 192 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 193 Chapter 3: Memory Management which calls setup_bootmem_allocator to initialize the bootmem allocator, whereas AMD64 uses contig_initmem_init. The code flow diagram in Figure 3-20 illustrates the individual steps involved in the initialization of the bootmem allocator on IA-32 systems, and the corresponding diagram for AMD64 is shown in Figure 3-21. Determine usable low memory page frames Call reserve_bootmem to reserve special regions setup_memory reserve_bootmem(bootmap,bootmap_size) register_bootmem_low_pages init_bootmem setup_bootmem_allocator Figure 3-20: Initialization of the bootmem allocator on IA-32 machines. Find suitable memory area contig_initmem bootmem_bootmap_pages init_bootmem register_bootmem_with_active_regions reserve_bootmem (bootmap,bootmap_size) Figure 3-21: Initialization of the bootmem allocator on AMD64 machines. Initialization for IA-32 setup_memory analyzes the detected memory regions to find the maximal page frame number in the low memory region; high memory is of no use for the bootmem allocator because it is too complicated to handle. The global variable max_low_pfn holds the number of the highest mappable page. The kernel reports the amount of memory found in its boot logs. wolfgang@meitner> dmesg ... 0MB HIGHMEM available. 511MB LOWMEM available. ... 193 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 194 Chapter 3: Memory Management Based on this information, setup_bootmem_allocator is then responsible to initiate all necessary steps to initialize the bootmem allocator. It first invokes the generic function init_bootmem, which itself is a front end for init_bootmem_core. The purpose of init_bootmem_core is to perform the first initialization step of the bootmem allocator. The previously detected range of low-memory page frames is entered in the responsible bootmem_data_t instance, in this case, contig_bootmem_data. All pages are initially marked as used in the bitmap contig_bootmemdata->node_bootmem_map.Becauseinit_bootmem_core is an architecture-independent function, it cannot yet know which pages are available for use and which are not. Some need special handling for architectural reasons, for example, page 0 on IA-32 systems. Some are already used, for instance, by the kernel image. The pages that can actually be used must be explicitly marked by the architecture-dependent code. This marking is done with two architecture-specific functions: register_bootmem_low_pages frees all potentially usable memory pages by setting the corresponding bits in the bitmap to 0 so that the pages are marked as unused. IA-32 systems are supported in this task by the BIOS, which provides the kernel with a list of all usable memory areas — the e820 map — at an earlier point during initialization. Since the bootmem allocator requires some memory pages to manage the reservation bitmap, these must be reserved first by calling reserve_bootmem. However, there are still more regions that are already in use and must be marked accordingly. For this purpose, reserve_bootmem registers the corresponding pages after the event. The exact number of regions that must be registered depends highly on the kernel configuration. It is, for instance, required to reserve the zeroth page because this page is a special BIOS page on many machines that is required for some machine-specific functions to work properly. Other reserve_bootmem calls reserve configuration-dependent memory areas, for ACPI data or SMP boot-time configurations, for instance. Initialization for AMD64 While the technical details of bootmem initialization differ on AMD64, the general structure is rather similar to the IA-32 case. This time contig_initmem is the responsible dispatcher. First of all, bootmem_bootmap_bitmap computes the number of pages required for the bootmem bitmap. Using the information provided by the BIOS in the e820 map, this allows — as on IA- 32 — for finding a continuous memory region of suitable size that is populated with proper RAM pages. This information is then filled into the architecture-independent bootmem data structure using init_bootmem. As before, the function marks all pages as reserved, and the free pages must now be selected. free_bootmem_with_active_regions can once more use the information in the e820 map to free all memory regions that were reported as usable by the BIOS. Finally, a single call to reserve_bootmem is sufficient to register the space required for the bootmem allocation bitmap. In contrast to IA-32, it is not required to reserve space for legacy information on magical places in memory. 194 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 195 Chapter 3: Memory Management Interface to the Kernel Allocating Memory The kernel provides a variety of functions for reserving memory during initialization. The following functions are available on UMA systems: ❑ alloc_bootmem(size) and alloc_bootmem_pages(size) reserve memory whose size is expressed by size in ZONE_NORMAL; data are aligned so that memory begins either at an ideal position for the L1 cache or on a page boundary. Even though the name alloc_bootmem_pages suggests that the required size is specified in page units, _pages refers only to the alignment of the data. ❑ alloc_bootmem_low and alloc_bootmem_low_pages operate in the same ways as the above functions but take the area ZONE_DMA that is suitable for DMA operations as their source. Con- sequently, the functions should only be used when DMA memory is required. Basically the same API applies for NUMA systems, but the suffix _node is appended to the function name. As compared with the UMA functions, an additional parameter is required to specify which node is used for memory reservation. Thesefunctionsareallfrontendsfor__alloc_bootmem, which delegates the real work to __alloc_bootmem_nopanic. Since more than one bootmem allocator can be registered (recall that they are all kept in a global list), __alloc_bootmem_core iterates over all of them until one succeeds. On NUMA systems, __alloc_bootmem_node is used to implement the API functions. First, work is passed on to __alloc_bootmem_core to try the allocation on the specific bootmem allocator of the node. If this fails, the function falls back to __alloc_bootmem, which tries all nodes. mm/bootmem.c void * __init __alloc_bootmem(unsigned long size, unsigned long align, unsigned long goal) __alloc_bootmem requires three parameters to describe a request: size is the size of the desired memory area, align indicates the alignment of the data, and goal specifies the start address at which the search for a suitable free area is to begin. The front ends use the function as follows: #define alloc_bootmem(x) \ __alloc_bootmem((x), SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS)) #define alloc_bootmem_low(x) \ __alloc_bootmem((x), SMP_CACHE_BYTES, 0) #define alloc_bootmem_pages(x) \ __alloc_bootmem((x), PAGE_SIZE, __pa(MAX_DMA_ADDRESS)) #define alloc_bootmem_low_pages(x) \ __alloc_bootmem((x), PAGE_SIZE, 0) 195 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 196 Chapter 3: Memory Management The desired allocation size (x)isforwardedunchangedto__alloc_bootmem, but there are two options for alignment in memory: SMP_CACHE_BYTES aligns the data on most architectures so that they are ideally positioned in the L1 cache (despite its name the constant is, of course, also defined on uniprocessor systems). PAGE_SIZE aligns the data on the page boundaries. The latter alignment is ideal for allocating oneormorecompletepages,buttheformerproduces better results when parts of pages are allocated. The distinction between low and normal memory is made by means of the start address. Searches for DMA-suitable memory begins at the address 0, whereas requests for normal memory with RAM are processed from MAX_DMA_ADDRESS upward (__pa translates the memory address into a page number). __alloc_bootmem_core is a relatively extensive function (efficiency is not required during booting) that I won’t discuss in detail as the main thing it does is to implement the first-fit algorithm already described. However, the allocator has been enhanced to permit reservation not only of complete memory pages but also smaller parts thereof. The function performs the following (outline) actions: 1. Starting at goal, the page bitmap is scanned for a free area to satisfy the allocation request. 2. If the page found immediately follows the last allocated page held in bootmem_data-> last_pos, the kernel checks by reference to bootmem_data->last_offset whether the required memory (including the space needed to align the data) can be allocated in the last page or can at least start there. 3. The bits of the newly allocated pages in the block bitmap are set to 1. The number of the last page allocated is also stored in bootmem_data->last_pos. If the page is not fully allocated, the appropriate offset is held in bootmem_data->last_offset; otherwise, this value is set to 0. Releasing Memory The kernel provides the free_bootmem function to free memory. It requires two parameters — the start address and the size of the area to be freed. The name of the equivalent function on NUMA systems is not surprisingly free_bootmem_node; it expects an additional parameter to define the appropriate node. void free_bootmem(unsigned long addr, unsigned long size); void free_bootmem_node(pg_data_t *pgdat, unsigned long addr, unsigned long size); Both versions delegate their work to __free_bootmem_core. Only whole pages can be freed because the bootmem allocator does not keep any information about page divisions. The kernel uses __free_bootmem_core to first calculate the pages whose contents are fully held in the area to be freed. Pages whose contents are only held in part in this area are ignored. The corresponding entries in the page bitmap are set to 0 to conclude page freeing. This procedure conceals the risk that a page is not freed if parts of its contents are returned in successive requests. If the first half of a page and at some time later the second half of the same page are freed, the allocator has no way of knowing that the entire page is no longer in use and can therefore be freed. The page simply remains ‘‘in use,’’ although this is not the case. Nevertheless, this is not a big problem because free_bootmem is very rarely used. Most memory areas allocated during system initialization are 196 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 197 Chapter 3: Memory Management intended for basic data structures that are needed throughout kernel run time and are therefore never relinquished. Disabling the Bootmem Allocator The bootmem allocator must be disabled once system initialization has progressed so far that the buddy system allocator can assume responsibility for memory management; after all, memory cannot be man- aged by two allocators at the same time. Disabling is done by free_all_bootmem on UMA systems and by free_all_bootmem_node on NUMA systems. Both need to be invoked by the architecture-specific initialization code after the buddy system has been set up. The page bitmap of the bootmem allocator is first scanned, and every unused page is freed. The interface to the buddy system is the __free_pages_bootmem function that is invoked for each freed page. The function relies internally on the standard function __free_page. It enables the pages to be incorporated in the data structures of the buddy system, where they are managed as free pages and are available for allocation. Once the page bitmap has been fully scanned, the memory space it occupies must also be removed. Thereafter, only the buddy system can be used for memory allocation. Releasing Initialization Data Many kernel code chunks and data tables are needed only during the system initialization phase. For example, it is not necessary to keep data structure initialization routines in kernel memory for per- manently linked drivers. They are no longer needed once the structures have been set up. Similarly, hardware databases that drivers need to detect their devices are no longer required once the associated devices have been identified.17 The kernel provides two ‘‘attributes‘‘ (__init and __initcall) to label initialization functions and data. These must be placed before the function or data declarations. For instance, the probing rou- tine of the (fictitious ... ) network card HyperHopper2000 is no longer used once the system has been initialized. int __init hyper_hopper_probe(struct net_device *dev) The __init attribute is inserted between the return type and name in the function declaration. Data sections can likewise be labeled as initialization data. For example, the fictitious network card driver requires a few strings in the system initialization phase only; thereafter the strings can be discarded. static char search_msg[] __initdata = "%s: Desperately looking for HyperHopper at address %x..."; static char stilllooking_msg[] __initdata = "still searching..."; static char found_msg[] __initdata = "found.\n"; static char notfound_msg[] __initdata = "not found (reason = %d)\n"; static char couldnot_msg[] __initdata = "%s: HyperHopper not found\n"; __init and __initdata cannot be implemented using normal C means so that the kernel once again has to resort to special GNU C compiler statements. The general idea behind the implementation of 17At least for compiled-in data and devices that are not hotpluggable. If devices are added to the system dynamically, the data tables cannot, of course, be discarded as they may be required later. 197 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 198 Chapter 3: Memory Management initialization functions is to keep the data in a specific part of the kernel image that can be completely removed from memory when booting has finished. The following macros are defined with this in mind: #define __init __attribute__ ((__section__ (".init.text"))) __cold #define __initdata __attribute__ ((__section__ (".init.data"))) __attribute__ is a special GNU C keyword to permit the use of attributes. The __section__ attribute is used to instruct the compiler to write the subsequent data or function into the respective .init.data and .init.text sections of the binary file (those of you unfamiliar with the structure of ELF files are referred to Appendix E). The prefix __cold also instructs the compiler that paths leading to the function will be unlikely, that is, that the function won’t be called very often, which is usually the case for initialization functions. The readelf tool can be used to display the individual sections of the kernel image. wolfgang@meitner> readelf — sections vmlinux There are 53 section headers, starting at offset 0x2c304c8: Section Headers: [Nr] Name Type Address Offset Size EntSize Flags Link Info Align [ 0] NULL 0000000000000000 00000000 0000000000000000 0000000000000000 0 0 0 [ 1] .text PROGBITS ffffffff80200000 00200000 000000000021fc6f 0000000000000000 AX 0 0 4096 [ 2] __ex_table PROGBITS ffffffff8041fc70 0041fc70 0000000000003e50 0000000000000000 A 0 0 8 [ 3] .notes NOTE ffffffff80423ac0 00423ac0 0000000000000024 0000000000000000 AX 0 0 4 ... [28] .init.text PROGBITS ffffffff8067b000 0087b000 000000000002026e 0000000000000000 AX 0 0 1 [29] .init.data PROGBITS ffffffff8069b270 0089b270 000000000000c02e 0000000000000000 WA 0 0 16 ... To release initialization data from memory, it is not necessary for the kernel to know the nature of the data — which data and functions are held in memory and what purpose they serve is totally irrelevant. The only information of relevance is the addresses in memory at which the data and functions begin and end. Because this information is not available at compilation time, it is patched in when the kernel is linked. I have already mentioned this technique at other places in this chapter. To support it, the kernel defines the variable pair __init_begin and __init_end, whose names reveal their meanings. free_initmem is responsible for freeing the memory area defined for initialization purposes and return- ing the pages to the buddy system. The function is called right at the end of the boot process immediately before init starts the first process in the system. The boot logs include a message indicating how much memory was freed. 198 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 199 Chapter 3: Memory Management wolfgang@meitner> dmesg ... Freeing unused kernel memory: 308k freed ... In comparison with today’s typical main memory sizes, the approximately 300 KiB freed are not gigantic but are a significant contribution. The removal of initialization data is important, particularly on hand- held or embedded systems, which, by their very nature, make do with little memory. 3.5 Management of Physical Memory Responsibility for memory management is assumed by the buddy system once kernel initialization has been completed. The buddy system is based on a relatively simple but nevertheless surprisingly powerful algorithm that has been with us for almost 40 years. It combines two key characteristics of a good memory allocator — speed and efficiency. 3.5.1 Structure of the Buddy System An instance of struct page is available for each physical page of memory (a page frame) in the sys- tem. Each memory zone is associated with an instance of struct zone that holds the central array for managing buddy data. struct zone { ... /* * free areas of different sizes */ struct free_area free_area[MAX_ORDER]; ... }; free_area is an auxiliary data structure we have not yet met. It is defined as follows: struct free_area { struct list_head free_list[MIGRATE_TYPES]; unsigned long nr_free; }; nr_free specifies the number of free page blocks in the current area (counting is page by page for the zeroth area, by two-page pairs for order 1, by sets of four pages for order 2, etc.). free_list is used to link page lists. As discussed in Chapter 1, the page lists contain contiguous memory areas of the same size. While the definition provides more than one page list, I ignore this fact for a moment and come back to why there are different lists below. The order is a very important term in buddy systems. It describes the quantified units in which memory can be allocated. The size of a memory block is 2order,whereorder may extend from 0 to MAX_ORDER. 199 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 200 Chapter 3: Memory Management #ifndef CONFIG_FORCE_MAX_ZONEORDER #define MAX_ORDER 11 #else #define MAX_ORDER CONFIG_FORCE_MAX_ZONEORDER #endif #define MAX_ORDER_NR_PAGES (1 << (MAX_ORDER - 1)) The typical value of this constant is 11, which means that the maximum number of pages that can be requested in a single allocation is 211 = 2, 048. However, this value can be changed manually if the FORCE_MAX_ZONEORDER configuration option is set by the architecture-specific code. For example, the gigantic address spaces on IA-64 systems allow for working with MAX_ORDER = 18, whereas ARM or v850 systems use smaller values such as 8 or 9. This, however, is not necessarily caused by little memory sup- ported by the machine, but can also be because of memory alignment requirements. Or, as the Kconfig configuration file for the V850 architecture puts it: arch/v850/Kconfig # The crappy-ass zone allocator requires that the start of allocatable # memory be aligned to the largest possible allocation. config FORCE_MAX_ZONEORDER int default 8 if V850E2_SIM85E2C || V850E2_FPGA85E2C The indices of the individual elements of the free_area[] array are also interpreted as order parameters and specify how many pages are present in the contiguous areas on a shared list. The zeroth array ele- ment lists sections with one page (20 = 1), the first lists page pairs (21 = 2), the third manages sets of 4 pages, and so on. How are the page areas linked? The list element of the first page in the block is used to keep the blocks in a list. As a result, there is no need to introduce a new data structure to group pages that are physically contiguous — otherwise, they wouldn’t be in a block. Figure 3-22 illustrates the situation graphically. next prev struct page0 1 2 . . MAX_ORDER Figure 3-22: Linking blocks in the buddy system. The buddies need not be linked with each other. If a block is broken down into two blocks of half the size during allocation, the kernel automatically adds the unused half to the list of next smaller blocks. If, at some point in the future, both blocks are not in use after memory has been freed, their addresses can be referenced to automatically determine whether they are buddies. This minimal administrative effort is a major advantage of the buddy system. 200 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 201 Chapter 3: Memory Management Memory management based on the buddy system is concentrated on a single memory zone of a node, for instance, the DMA or high-memory zone. However, the buddy systems of all zones and nodes are linked via the allocation fallback list. Figure 3-23 illustrates this relationship. When a request for memory cannot be satisfied in the preferred zone or node, first another zone in the same node, and then another node is picked to fulfill the request. Zone Zone Zone Zone Zone Zone Fallback list Figure 3-23: Relationship between buddy system and memory zones/nodes. Finally, note that information about the current state of the buddy system is available in /proc/buddyinfo: wolfgang@meitner> cat /proc/buddyinfo Node 0, zone DMA 3 5 7 4 6 3 3 3 1 1 1 Node 0, zone DMA32 130 546 695 271 107 38 2 2 1 4 479 Node 0, zone Normal 23 6 6 81430000 The number of free entries per allocation order is printed for each zone, and the order increases from left to right. The information shown above is taken from an AMD64 system with 4 GiB of RAM. 3.5.2 Avoiding Fragmentation In the simplified explanation given in the Introduction, one doubly linked list was sufficient to satisfy all the needs of the buddy system. This has, indeed, been the situation until kernel 2.6.23. During the devel- opment of the kernel 2.6.24, the buddy system has, however, seen the integration of patches disputed among the kernel developers for an unusually long amount of time. Since the buddy system is one of the most venerable components of the kernel, changes are not accepted lightly. Grouping Pages by Mobility The basic principle of the buddy system has been discussed in the Introduction, and the scheme has, indeed, worked very well during the last couple of years. However, there is one issue that has been a long-standing problem with Linux memory management: After systems have been up and running for longer times, physical memory tends to become fragmented. The situation is depicted in Figure 3-24. 201 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 202 Chapter 3: Memory Management 5948 36 24 12 0 47 35 23 11 Figure 3-24: Fragmentation of physical memory. Assume that the memory consists of 60 pages — clearly, this is not going to be the key component to the next supercomputer, but is fair enough for the sake of example. The free pages are scattered across the address space on the left-hand side. Although roughly 25 percent of the physical mem- ory is still unallocated, the largest continuous free area is only a single page. This is no problem for userspace applications: Since their memory is mapped over page tables, it will always appear continu- ous to them irrespective of how the free pages are distributed in physical memory. The right-hand side shows the situation with the same number of used and free pages, but with all free pages located in a continuous area. Fragmentation is, however, a problem for the kernel: Since (most) RAM is identity-mapped into the kernel’s portion of the address space, it cannot map an area larger than a single page in this scenario. While many kernel allocations are small, there is sometimes the need to allocate more than a single page. Clearly, the situation on the right-hand side, where all reserved and free pages are in continuous regions, would be preferable. Interestingly, problems with fragmentation can already occur when most of the memory is still unallo- cated. Consider the situation in Figure 3-25. Only 4 pages are reserved, but the largest contiguous area that can be allocated is 8 pages because the buddy system can only work that allocation ranges that are powers of 2. 0 7 15 23 31 Figure 3-25: Memory fragmentation where few reserved pages prevent the allocation of larger contiguous blocks. I have mentioned that memory fragmentation only concerns the kernel, but this is only partially true: Most modern CPUs provide the possibility to work with huge pages whose page size is much bigger than for regular pages. This has benefits for memory-intensive applications. When bigger pages are used, the translation lookaside buffer has to handle fewer entries, and the chance of a TLB cache miss is reduced. Allocating huge pages, however, requires free contiguous areas of physical RAM! Fragmentation of physical memory has, indeed, belonged to the weaker points of Linux for an unusually long time span. Although many approaches have been suggested, none could satisfy the demanding needs of the numerous workloads that Linux has to face without having too great an impact on others. During the development of kernel 2.6.24, means to prevent fragmentation finally found their way into the kernel. Before I discuss their strategy, one point calls for clarification: Fragmentation is also known from filesystems, and in this area the problem is typically solved by defragmentation tools: They analyze the filesystem and rearrange the allocated blocks such that larger continuous areas arise. This approach would also be possible for RAM, in principle, but is complicated by the fact that many physical pages 202 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 203 Chapter 3: Memory Management cannot be moved to an arbitrary location. Therefore, the kernel’s approach is anti-fragmentation:Tryto prevent fragmentation as well as possible from the very beginning. How does anti-fragmentation work? To understand the approach, we must be aware that the kernel distinguishes three different types of reserved pages: ❑ Non-movable pages have a fixed position in memory and cannot be moved anywhere else. Most allocations of the core kernel fall into this category. ❑ Reclaimable pages cannot be moved directly, but they can be deleted and their contents regener- ated from some source. Data mapped from files fall into this category, for instance. Reclaimable pages are periodically freed by the kswapd daemon depending on how often they are accessed. This is a complicated process that merits a detailed discussion of its own: Chapter 18 that describes page reclaim in detail. In the meanwhile, it suffices to know that the kernel will take care of removing reclaimable pages when they start to use up too much RAM. It is also possible to initiate page reclaim when there is an acute shortage of memory, that is, when an allocation has failed. You will see further below when the kernel deems it necessary to do so. ❑ Movable pages can be moved around as desired. Pages that belong to userspace applications fall into this category. They are mapped via page tables. If they are copied into a new location, the page table entries can be updated accordingly, and the application won’t notice anything. A page has a certain mobility depending into which of the three categories it falls. The anti-fragmentation technique used by the kernel is based on the idea of grouping pages with identical mobility together. Why does this approach help to reduce fragmentation? Recall from Figure 3-25 that a page that cannot be moved somewhere else can prevent continuous allocations in an otherwise nearly completely empty RAM area. By distributing pages onto different lists depending on their mobility, this situation is pre- vented. For instance, a non-movable page cannot be located in the middle of a block of movable pages and effectively prevent any larger part of the block from being used. Imagine that most of the free pages in Figure 3-25 belong to the reclaimable category, while the reserved pages are non-movable. If the pages had been collected on two different lists, the situation might, how- ever, look as shown in Figure 3-26. It is still hard to find a large continuous free space for non-movable pages, but much easier for reclaimable pages. Reclaimable pages Un-movable pages Figure 3-26: Memory fragmentation is reduced by grouping pages together depending on their mobility. Note, however, that the memory is not partitioned into different mobility regions from the very beginning. They will be populated at run time. A second approach of the kernel does partition the memory into regions for movable and non-movable allocations, and I will discuss how this works below. Such a partitioning, however, is not essential for the approach described here. 203 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 204 Chapter 3: Memory Management Data Structure Although the anti-fragmentation technique used by the kernel is highly effective, it has astonishingly little impact on code and data structures of the buddy allocator. The kernel defines some macros to represent the different migrate types: #define MIGRATE_UNMOVABLE 0 #define MIGRATE_RECLAIMABLE 1 #define MIGRATE_MOVABLE 2 #define MIGRATE_RESERVE 3 #define MIGRATE_ISOLATE 4 /* can’t allocate from here */ #define MIGRATE_TYPES 5 The types MIGRATE_UNMOVABLE, MIGRATE_RECLAIMABLE,andMIGRATE_MOVABLE have already been intro- duced. MIGRATE_RESERVE provides an emergency memory reserve if an allocation request cannot be fulfilled from the mobility-specific lists (it is filled during initialization of the memory subsystem with setup_zone_migrate_reserve, but I will not go into detail about this). MIGRATE_ISOLATE is a special virtual zone that is required to move physical pages across NUMA nodes. On large systems, it can be beneficial to bring physical pages closer to the CPUs that use them most. MIGRATE_TYPES, finally, is also not a zone, but just denotes the number of migrate types. The core adjustment to the buddy system data structures is that the free list is broken into a MIGRATE_TYPE number of lists: struct free_area { struct list_head free_list[MIGRATE_TYPES]; unsigned long nr_free; }; nr_free counts the number of free pages on all lists, but a specific free list is provided for each migrate type. The macro for_each_migratetype_order(order, type) can be used to iterate over the migrate types of all allocation orders. What happens if the kernel cannot fulfill an allocation request for a given migrate type? A similar problem has already occurred before, namely, when we considered what happens when an allocation cannot be fulfilled from a specific NUMA zone. The kernel proceeds similarly as in this case by providing a fallback list regulating which migrate types should be used next if a request cannot be fulfilled from the desired list: mm/page_alloc.c /* * This array describes the order lists are fallen back to when * the free lists for the desirable migrate type are depleted */ static int fallbacks[MIGRATE_TYPES][MIGRATE_TYPES-1] = { [MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_RESERVE }, [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_RESERVE }, [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_RESERVE }, [MIGRATE_RESERVE] = { MIGRATE_RESERVE, MIGRATE_RESERVE, MIGRATE_RESERVE }, /* Never used */ }; 204 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 205 Chapter 3: Memory Management The data structure is mostly self-explanatory: When the kernel wants to allocate un-movable pages, but the corresponding list is empty, then it falls back to reclaimable pages, then to movable pages, and finally to the emergency reserve. Global Variables and Auxiliary Functions While page mobility grouping is always compiled into the kernel, it only makes sense if enough memory that can be distributed across multiple migrate lists is present in a system. Since on each migrate list a suitable amount of memory should be present, the kernel needs a notion of ‘‘suitable.’’Thisisprovided by the two global variables pageblock_order and pageblock_nr_pages. The first denotes an allocation order that is considered to be ‘‘large,’’ and pageblock_nr_pages denotes the corresponding number of pages for this allocation order. Usually the the page order is selected to be the order of huge pages if such are provided by the architecture: #define pageblock_order HUGETLB_PAGE_ORDER On the IA-32 architecture, huge pages are 4 MiB in size, so each huge page consists of 1,024 regular pages and HUGETLB_PAGE_ORDER is defined to be 10. The IA-64 architecture, in contrast, allows varying regular and huge page sizes, so the value of HUGETLB_PAGE_ORDER depends on the kernel configuration. If an architecture does not support huge pages, then the second highest allocation order is taken as a large order: #define pageblock_order (MAX_ORDER-1) Page migration will not provide any benefits if each migrate type cannot at least be equipped with one large page block, so the feature is turned off by the kernel if too little memory is available. This is checked in the function build_all_zonelists, which is used to initialize the zone lists. If not enough memory is available, the global variable page_group_by_mobility is set to 0, otherwise to 1.18 How does the kernel know to which migrate type a given allocation belongs? As you will see in Section 3.5.4, details about each memory allocation are specified by an allocation mask.Thekernel provides two flags that signal that the allocated memory will be movable (__GFP_MOVABLE) or reclaimable (__GFP_RECLAIMABLE). If none of these flags is specified, the allocation is assumed to be non-movable. The following auxiliary function converts between allocation flags and their corresponding migrate types: static inline int allocflags_to_migratetype(gfp_t gfp_flags) { if (unlikely(page_group_by_mobility_disabled)) return MIGRATE_UNMOVABLE; /* Group based on mobility */ return (((gfp_flags & __GFP_MOVABLE) != 0) << 1) | ((gfp_flags & __GFP_RECLAIMABLE) != 0); } 18Note that systems not only with little memory but also with extremely large page sizes can be affected by this since the check is performed on a pages-per-list basis. 205 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 206 Chapter 3: Memory Management If page mobility has been disabled, all pages will be kept in the unmovable zone. Otherwise, the return value of the function can be directly used as an array index in free_area.free_list. Finally, note that each memory zone provides a special field that allows for tracking properties of page blocks with pageblock_nr_pages pages. Since this is currently only used by the page mobility code, I have not introduced this feature before: struct zone { ... unsigned long *pageblock_flags; ... } During initialization, the kernel automatically ensures that for each page block group in the zone, suf- ficient space is available in pageblock_flags to store NR_PAGEBLOCK_BITS bits. Currently, 3 bits are required to denote the migrate type of the page range: /* Macro to aid the definition of ranges of bits */ #define PB_range(name, required_bits) \ name, name ## _end = (name + required_bits) - 1 /* Bit indices that affect a whole block of pages */ enum pageblock_bits { PB_range(PB_migrate, 3), /* 3 bits required for migrate types */ NR_PAGEBLOCK_BITS }; set_pageblock_migratetype is responsible to set the migrate type for a page block headed by page: mm/page_alloc.c void set_pageblock_migratetype(struct page *page, int migratetype) The migratetype argument can be constructed by the auxiliary function allocflags_to_migratetype introduced above. Notice that it is essential that the migrate type of a page is always preserved and not just available when the page is located in the buddy system. When memory is released, the pages must be put back to the proper migrate list, and this is only possible because the required information can be obtained with get_pageblock_migratetype. Finally, notice that the current state of page distribution across the migrate lists can be found in /proc/pagetypeinfo: wolfgang@meitner> cat /proc/pagetypeinfo Page block order: 9 Pages per block: 512 206 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 207 Chapter 3: Memory Management Free pages count per migrate type at order 0 1 2 3 4 5 6 7 8 9 10 Node 0, zone DMA, type Unmovable 0 0 1 1 1 1 1 1 1 1 0 Node 0, zone DMA, type Reclaimable 0 0 0 0 0 0 0 0 0 0 0 Node 0, zone DMA, type Movable 3 5 6 3 5 2 2 2 0 0 0 Node 0, zone DMA, type Reserve 0 0 0 0 0 0 0 0 0 0 1 Node 0, zone DMA, type 0 0 0 0 0 0 0 0 0 0 0 Node 0, zone DMA32, type Unmovable 44 37 29 1 2 0 1 1 0 1 0 Node 0, zone DMA32, type Reclaimable 18 29 3 4 1 0 0 0 1 1 0 Node 0, zone DMA32, type Movable 0 0 191 111 68 26 21 13 7 1 500 Node 0, zone DMA32, type Reserve 0 0 0 0 0 0 0 0 0 1 2 Node 0, zone DMA32, type 0 0 0 0 0 0 0 0 0 0 0 Node 0, zone Normal, type Unmovable 1 5 1 0 0 0 0 0 0 0 0 Node 0, zone Normal, type Reclaimable 0 0 0 0 0 0 0 0 0 0 0 Node 0, zone Normal, type Movable 1 4 0 0 0 0 0 0 0 0 0 Node 0, zone Normal, type Reserve 11 13 7 8 3 4 2 0 0 0 0 Node 0, zone Normal, type 0 0 0 0 0 0 0 0 0 0 0 Number of blocks type Unmovable Reclaimable Movable Reserve Node 0, zone DMA 1 0 6 1 0 Node 0, zone DMA32 13 18 2005 4 0 Node 0, zone Normal 22 10 351 1 0 Initializing Mobility-Based Grouping During the initialization of the memory subsystem, memmap_init_zone is responsible to handle the page instances of a memory zone. The function does some standard initializations that are not too interesting, but one thing is essential: All pages are initially marked to be movable! mm/page_alloc.c void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, unsigned long start_pfn, enum memmap_context context) { struct page *page; unsigned long end_pfn = start_pfn + size; unsigned long pfn; for (pfn = start_pfn; pfn < end_pfn; pfn++) { ... if ((pfn & (pageblock_nr_pages-1))) set_pageblock_migratetype(page, MIGRATE_MOVABLE); ... } As discussed in Section 3.5.4, the kernel favors large page groups when pages must be ‘‘stolen’’ from different migrate zones from those the allocation is intended for. Because all pages initially belong to the movable zone, stealing pages is required when regular, unmovable kernel allocations are performed. 207 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 208 Chapter 3: Memory Management Naturally, not too many movable allocations will have been performed during boot, so chances are good that the allocator can pick maximally sized blocks and transfer them from the movable to the non- movable list. Because the blocks have maximal size, no fragmentation is introduced into the movable zone! All in all, this avoids situations in which kernel allocations that are done during boot (and which often last for the whole system uptime) are spread across the physical RAM such that other allo- cation types are fragmented — one of the most important goals of the page mobility grouping framework. The Virtual Movable Zone Grouping pages by mobility order is one possible method to prevent fragmentation of physical memory, but the kernel additionally provides another means to fight this problem: the virtual zone ZONE_MOVABLE. The mechanism has even found its way into the kernel during the devel- opment of kernel 2.6.23, one release before the mobility grouping framework was merged. In contrast to mobility grouping, the ZONE_MOVABLE feature must be explicitly activated by the administrator. The basic idea is simple: The available physical memory is partitioned into one zone used for movable allocations, and one zone used for non-movable allocations. This will automatically prevent any non- movable pages from introducing fragmentation into the movable zone. This immediately raises the question of how the kernel is supposed to decide how the available memory will be distributed between the two competitors. Clearly, this asks too much of the poor kernel, so the system administrator has to make the decision. After all, a human can predict much better which sce- narios the machine will handle and what the expected distribution of allocations into the various types will be. Data Structures The kernelcore parameter allows for specifying the amount of memory used for non-movable allo- cations, that is, for allocations that can neither be reclaimed nor migrated. The remaining memory is used for movable allocations. After parsing the parameter, the result is stored in the global variable required_kernelcore. It is also possible to use the parameter movablecore to control the amount of memory that is used for movablememory.Thesizeofrequired_kernelcore will be computed accordingly. If wise guys specify both parameters simultaneously, the kernel computes required_kernelcore as before, and takes the larger one of the computed and specified value. Depending on the architecture and the kernel configuration, the new zone ZONE_MOVABLE is located above the high-memory or regular-memory zone: enum zone_type { ... ZONE_NORMAL #ifdef CONFIG_HIGHMEM ZONE_HIGHMEM, 208 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 209 Chapter 3: Memory Management #endif ZONE_MOVABLE, MAX_NR_ZONES }; In contrast to all other zones in the system, ZONE_MOVABLE is not associated with any memory range that is of significance to the hardware. Indeed, the zone is filled with memory taken from either the highmem or the regular zone, and accordingly we call ZONE_MOVABLE a virtual zone in the following. The auxiliary function find_zone_movable_pfns_for_nodes is used to compute the amount of mem- ory that goes into ZONE_MOVABLE. If neither the kernelcore nor movablecore parameter was specified, find_zone_movable_pfns_for_nodes leaves ZONE_MOVABLE empty, and the mechanism is not active. Two things must be considered with respect to how many pages are taken from a physical zone and used for ZONE_MOVABLE: ❑ The memory for non-movable allocations is spread evenly across all memory nodes. ❑ Only memory from the highest zone is used. On 32-bit systems with much memory, this will usually be ZONE_HIGHMEM, but for 64-bit systems, ZONE_NORMAL or ZONE_DMA32 will be used. The actual computation is rather lengthy, but not very interesting, so I do not consider it in detail. What matters are the results: ❑ The physical zone from which pages for the virtual zone ZONE_MOVABLE are taken is stored in the global variable movable_zone. ❑ For each node, the page frame in the movable zone from which onward the memory belongs to ZONE_MOVABLE is in zone_movable_pfn[node_id]. mm/page_alloc.c unsigned long __meminitdata zone_movable_pfn[MAX_NUMNODES]; The kernel ensures that these pages will be used to satisfy allocations that fall into the responsibility of ZONE_MOVABLE. Implementation How are the data structures described so far brought to use? As with the page migration approach, allocation flags play a crucial role. They are discussed below in Section 3.5.4 in more detail. Here, it suffices to say that all movable allocations must specify both __GFP_HIGHMEM and __GFP_MOVABLE. Since the kernel determines the zone from which an allocation is fulfilled by the allocation flags, it can select ZONE_MOVABLE when the said flags are set. This is the only change required to integrate ZONE_MOVABLE into the buddy system! The rest is done by generic routines that work on all zones, discussed below. 3.5.3 Initializing the Zone and Node Data Structures Until now, we have only seen how the kernel detects the available memory in the system in the architecture-specific code. The association with higher-level structures — zones and nodes — needs to 209 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 210 Chapter 3: Memory Management be constructed from this information. Recall that architectures are required to established the following information during boot: ❑ The page frame boundaries of the various zones in the system as stored in the max_zone_pfn array. ❑ The distribution of page frames across nodes as stored in the global variable early_node_map. Managing Data Structure Creation Starting with kernel 2.6.10, a generic framework was provided to transfer this information into the node and zone data structures expected by the buddy system; before this, each architecture had to set up the structures on its own. Today, it suffices to set up the aforementioned simple structures and leave the hard work to free_area_init_nodes. Figure 3-27 shows an overview of the process, and Figure 3-28 shows the code flow diagram for free_area_init_nodes. free_area_init_nodes pg_data_tFill in early_node_map Architecture-specific initialization Set max page frame numbers for zones (max_zone_pfns) pg_data_t Generic representation of nodes and zones Figure 3-27: Overview of the interplay between architecture-specific and generic kernel code for setting up node and zone data memory management data structures. Determine zone boundaries Iterate over all nodes free_area_init_nodes free_area_init_node calculate_node_totalpages alloc_node_mem_map free_area_init_core check_for_regular_memory Figure 3-28: Code flow diagram for free_area_init_nodes. 210 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 211 Chapter 3: Memory Management free_area_init_nodes first has to analyze and rewrite the information provided by the architecture- specific code. Among others, the numbers of the lowest and highest page frames that can be used —in contrast to the principal boundaries specified in zone_max_pfn and zone_min_pfn — need to be obtained for each zone. Two global arrays are used to store the information: mm/page_alloc.c static unsigned long __meminitdata arch_zone_lowest_possible_pfn[MAX_NR_ZONES]; static unsigned long __meminitdata arch_zone_highest_possible_pfn[MAX_NR_ZONES] First of all, however, free_area_init_nodes sortstheentriesinearly_node_map by their first page frame start_pfn. mm/page_alloc.c void __init free_area_init_nodes(unsigned long *max_zone_pfn) { unsigned long nid; enum zone_type i; /* Sort early_node_map as initialisation assumes it is sorted */ sort_node_map(); ... Sorting the entries makes life easier for the following tasks, but is not particularly complicated, so it is not required to inspect sort_node_map further. Just note that the kernel provides a generic heap sort implementation in lib/sort.c that is employed by the function. The information passed to free_area_init_nodes in max_zone_pfn records the maximal page frame numbers that can be contained in each zone. free_area_init_nodes prepares a more convenient representation of this information by providing page frame intervals of the form [low, high] for each zone in the aforementioned global variables (I omit initialization of these variables with zero bytes): mm/page_alloc.c arch_zone_lowest_possible_pfn[0] = find_min_pfn_with_active_regions(); arch_zone_highest_possible_pfn[0] = max_zone_pfn[0]; for (i = 1; i < MAX_NR_ZONES; i++) { if (i == ZONE_MOVABLE) continue; arch_zone_lowest_possible_pfn[i] = arch_zone_highest_possible_pfn[i-1]; arch_zone_highest_possible_pfn[i] = max(max_zone_pfn[i], arch_zone_lowest_possible_pfn[i]); } The auxiliary function find_min_pfn_with_active_regions is used to find the smallest regis- tered usable page frame for the lowest registered zone. This need not necessarily be ZONE_DMA, but can, for instance, also be ZONE_NORMAL if the machine does not require DMA memory. The maximum page frame for the smallest zone can be directly taken from the information provided by max_zone_pfn. 211 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 212 Chapter 3: Memory Management The intervals for the other zones are then constructed in a straightforward manner: The smallest page frame for the n-th zone is the largest page frame of the previous (n − 1) zone. The largest page frames for the current zone are already available in max_zone_pfn. mm/page_alloc.c arch_zone_lowest_possible_pfn[ZONE_MOVABLE] = 0; arch_zone_highest_possible_pfn[ZONE_MOVABLE] = 0; /* Find the PFNs that ZONE_MOVABLE begins at in each node */ ... find_zone_movable_pfns_for_nodes(zone_movable_pfn); Since ZONE_MOVABLE is a virtual zone and not associated with real hardware zones, the zone bound- aries are always set to zero. Recall from above that it only exists if any of the kernel command-line parameter kernelcore or movablecore was specified. The movable zone for each node starts above a certain page frame number of a specific zone for each node. The corresponding numbers are computed in find_zone_movable_pfns_for_nodes. Some information about the determined page frame intervals is proudly presented to the user. This includes, for instance, the following (the output is taken on an AMD64 system with 4 GiB of RAM): root@meitner # dmesg ... Zone PFN ranges: DMA 0 -> 4096 DMA32 4096 -> 1048576 Normal 1048576 -> 1245184 ... The remaining portion of free_area_init_nodes iterates over all nodes to set up the data structures for each. mm/page_alloc.c /* Print information about zones */ ... /* Initialise every node */ for_each_online_node(nid) { pg_data_t *pgdat = NODE_DATA(nid); free_area_init_node(nid, pgdat, NULL, find_min_pfn_for_node(nid), NULL); /* Any memory on that node */ if (pgdat->node_present_pages) node_set_state(nid, N_HIGH_MEMORY); check_for_regular_memory(pgdat); } } The code iterates over all active nodes and delegates setting up the data structures for each to free_area_init_node. The function requires the first available page frame as a parameter, and find_min_pfn_for_node extracts this information from the early_node_map array. 212 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 213 Chapter 3: Memory Management If the node is equipped with memory as indicated by the node_present_pages field, this is reflected in the node bitmap by setting the N_HIGH_MEMORY flag. Recall from Section 3.2.2 that the flag — despite its name — only signals that either regular or high memory is present on the node, so check_for_regular_memory checks if pages in any zone below ZONE_HIGHMEM are present and sets the flag N_NORMAL_MEMORY in the node bitmap accordingly. Creating Data Structures for Each Node Once the zone boundaries have been determined, free_area_init_nodes creates the data structures for the individual zones iteratively by calling free_area_init_node. Several helper functions are required for this purpose. calculate_node_totalpages first calculates the total number of pages in the node by summing up the pages in the individual zones. In the case of contiguous memory, this could be done in zones_size_init, but calculate_zone_totalpages also takes holes in the zone into account. The number of pages found for each node is output in a short message when the system is booted. The example below is taken from a UMA system with 512 MiB of RAM. wolfgang@meitner> dmesg ... On node 0 totalpages: 131056 ... alloc_node_mem_map is responsible for initializing a simple but nevertheless very important data struc- ture. As noted above, there is an instance of struct page for every physical memory page in the system. Initialization of this structure is performed by alloc_node_mem_map. mm/page_alloc.c static void __init_refok alloc_node_mem_map(struct pglist_data *pgdat) { /* Skip empty nodes */ if (!pgdat->node_spanned_pages) return; if (!pgdat->node_mem_map) { unsigned long size, start, end; struct page *map; start = pgdat->node_start_pfn & ~(MAX_ORDER_NR_PAGES - 1); end = pgdat->node_start_pfn + pgdat->node_spanned_pages; end = ALIGN(end, MAX_ORDER_NR_PAGES); size = (end - start) * sizeof(struct page); map = alloc_remap(pgdat->node_id, size); if (!map) map = alloc_bootmem_node(pgdat, size); pgdat->node_mem_map = map + (pgdat->node_start_pfn - start); } if (pgdat == NODE_DATA(0)) mem_map = NODE_DATA(0)->node_mem_map; } 213 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 214 Chapter 3: Memory Management Empty nodes with no pages can obviously be skipped. If the memory map has not already been set up by architecture-specific code (this can happen, e.g., on IA-64 systems), then the memory required for all instances of struct page associated with the node must be allocated. Individual architectures can provide a specific function for this purpose. This is, however, currently only the case for IA-32 with a discontiguous memory configuration. On all other configurations, the regular boot memory allocator is used to perform the allocation. Notice that the code aligns the memory map with the maximal allocation order of the buddy system because this is required for all calculations to work properly. A pointer to this space is held not only in the pglist_data instance but also in the global variable mem_map — providing the node just examined is the zeroth node of the system (always the case on a system with just one memory node). mem_map is a global array that we will come across frequently in our description of memory management. mm/memory.c struct page *mem_map; The heavy work involved in the initialization of zone data structures is carried out by free_area_init_core, which iterates over all zones of the node one after the other. mm/page_alloc.c static void __init free_area_init_core(struct pglist_data *pgdat, unsigned long *zones_size, unsigned long *zholes_size) { enum zone_type j; int nid = pgdat->node_id; unsigned long zone_start_pfn = pgdat->node_start_pfn; ... for (j = 0; j < MAX_NR_ZONES; j++) { struct zone *zone = pgdat->node_zones + j; unsigned long size, realsize, memmap_pages; size = zone_spanned_pages_in_node(nid, j, zones_size); realsize = size - zone_absent_pages_in_node(nid, j, zholes_size); ... The true size of the zone is obtained by correcting the number of spanned pages with the number of holes. Both quantities are computed by two helper functions, which I will not bother to discuss in more detail. Their complexity naturally depends on the memory model and configuration options chosen, but ultimately all variants do not provide any unexpected surprises. mm/page_alloc.c ... if (!is_highmem_idx(j)) nr_kernel_pages += realsize; nr_all_pages += realsize; zone->spanned_pages = size; zone->present_pages = realsize; ... zone->name = zone_names[j]; ... zone->zone_pgdat = pgdat; 214 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 215 Chapter 3: Memory Management /* Initialize zone fields to default values, * and call helper functions */ ... } The kernel uses two global variables to keep track of how many pages are present in the system: nr_kernel_pages counts all identity mapped pages, while nr_all_pages also includes high-memory pages. The task of the remaining part of free_area_init_core is to initialize the list heads of the zone struc- ture and to initialize the various structure members to 0. Of particular interest are two helper functions invoked: ❑ zone_pcp_init initializes the per-CPU caches for the zone as discussed extensively in the next section. ❑ init_currently_empty_zone initializes the free_area lists and sets all page instances of pages belonging to the zone to their initial defaults. memmap_init_zone as discussed above is invoked to initialize the pages of the zone. Also recall that all pages are attributed to MIGRATE_MOVABLE in the beginning. Additionally, the free lists are initialized in zone_init_free_lists: mm/page_alloc.c static void __meminit zone_init_free_lists(struct pglist_data *pgdat, struct zone *zone, unsigned long size) { int order, t; for_each_migratetype_order(order, t) { INIT_LIST_HEAD(&zone->free_area[order].free_list[t]); zone->free_area[order].nr_free = 0; } } Thenumberoffreepages(nr_free) is still currently defined as 0, and this obviously does not reflect the true situation. The correct value is not set until the bootmem allocator is disabled and normal buddy allocation comes into effect. 3.5.4 Allocator API As far as the interface to the buddy system is concerned, it makes no difference whether a NUMA or a UMA architecture is used as the call syntax is the same for both. Common to all functions is the fact that pages can only be allocated in integer powers of 2. For this reason, the desired memory size is not specified as parameter as it would be in the malloc function of the C standard library or in the bootmem allocator. Instead, the order of the allocation must be specified, and this causes the buddy system to reserve 2order pages in memory. Finer-grained allocation in the kernel is only possible with the help of the slab allocator (or alternatively, the slub or slob allocators), which builds on the buddy system (Section 3.6 gives further details). ❑ alloc_pages(mask, order) allocates 2order pages and returns an instance of struct page to rep- resent the start of the reserved block. alloc_page(mask) is a shorter notation for order = 0 if only one page is requested. 215 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 216 Chapter 3: Memory Management ❑ get_zeroed_page(mask) allocates a page and returns a page instance but fills the page with zeros (with all other functions, page contents are undefined after allocation). ❑ __get_free_pages(mask, order) and __get_free_page(mask) work in the same way as the above functions but return the virtual address of the reserved memory chunk instead of a page instance. ❑ get_dma_pages(gfp_mask,order) allows for obtaining pages suitable for DMA. If allocation fails because insufficient memory is free to satisfy the request, all the above functions return either a null pointer (alloc_pages and alloc_page)orthevalue0(get_zeroed_page, __get_free_pages,and__get_free_page). The kernel must therefore check the result returned after every allocation attempt. This practice is not different from any well-designed userland applications, but neglecting the check in the kernel will lead to much more severe failures. The kernel provides other memory management functions in addition to the buddy system functions. They build on layers that are used as a basis by the buddy system but do not belong to the buddy allocator itself. These functions are vmalloc and vmalloc_32, which use page tables to map discontiguous memory into kernel address space so that it appears to be contiguous. There is also a set of functions of the kmalloc type to reserve memory areas smaller than a complete page. Their implementation is discussed separately in later sections of this chapter. Four slightly different functions are defined to return pages no longer needed in memory to the kernel. ❑ free_page(struct page*) and free_pages(struct page*, order) return one or 2order pages to memory management. The start of the memory area is indicated by means of a pointer to the first page instance of the area. ❑ __free_page(addr) and __free_pages(addr, order) operate in the same way as the functions just mentioned but use a virtual memory address instead of a page instance to select the memory area to be returned. Allocation Masks What is the meaning of the mask parameter that is mandatory for all functions? As we know from Section 3.2.1, Linux divides memory into zones. The kernel provides what is known as zone modifiers (defined in the least significant 4 bits of a mask) to specify the zone from which the pages are to be taken for purposes of memory allocation. /* Zone modifiers in GFP_ZONEMASK (see linux/mmzone.h - low three bits) */ #define __GFP_DMA ((__force gfp_t)0x01u) #define __GFP_HIGHMEM ((__force gfp_t)0x02u) #define __GFP_DMA32 ((__force gfp_t)0x04u) ... #define __GFP_MOVABLE ((__force gfp_t)0x100000u) /* Page is movable */ These constants are familiar from Section 3.4.1 in which the creation of fallback lists is discussed. The abbreviation GFP stands for get free pages. __GFP_MOVABLE does not represent a physical memory zone, but instructs the kernel that an allocation should be fulfilled from the special virtual zone ZONE_MOVABLE. 216 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 217 Chapter 3: Memory Management Interestingly, there is no __GFP_NORMAL constant, although the main burden of allocation falls on this zone. The kernel takes account of this fact by providing a function that calculates the highest memory zone compatible with the given allocation flags. Allocations can then be made from this zone and from those below it. mm/page_alloc.c static inline enum zone_type gfp_zone(gfp_t flags) { #ifdef CONFIG_ZONE_DMA if (flags & __GFP_DMA) return ZONE_DMA; #endif #ifdef CONFIG_ZONE_DMA32 if (flags & __GFP_DMA32) return ZONE_DMA32; #endif if ((flags & (__GFP_HIGHMEM | __GFP_MOVABLE)) == (__GFP_HIGHMEM | __GFP_MOVABLE)) return ZONE_MOVABLE; #ifdef CONFIG_HIGHMEM if (flags & __GFP_HIGHMEM) return ZONE_HIGHMEM; #endif return ZONE_NORMAL; } Because the way in which the zone modifiers are interpreted may not immediately appear to be intuitive, Table 3-7 shows an example of the function results when the zones for DMA and DMA32 are identical. Assume that the __GFP_MOVABLE modifier is not set in the following: If both __GFP_DMA and __GFP_HIGHMEM are not set, ZONE_NORMAL is first scanned, followed by ZONE_DMA.If __GFP_HIGHMEM is set and __GFP_DMA is not set, the result is that all three zones are scanned starting with ZONE_HIGHMEM.If__GFP_DMA is set, it is irrelevant to the kernel whether __GFP_HIGHMEM is set or not. Only ZONE_DMA is used in both cases. This is reasonable because the simultaneous use of __GFP_HIGHMEM and __GFP_DMA makes no sense. Highmem is never DMA-suitable. Table 3-7: Correlation between Zone Modifiers and Zones Scanned Modifier Zones scanned Empty ZONE_NORMAL, ZONE_DMA __GFP_DMA ZONE_DMA __GFP_DMA & __GFP_HIGHMEM ZONE_DMA __GFP_HIGHMEM ZONE_HIGHMEM, ZONE_NORMAL, ZONE_DMA 217 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 218 Chapter 3: Memory Management Setting __GFP_MOVABLE will not influence the kernel’s decision unless it is specified together with __GFP_HIGHMEM. In this case, the special virtual zone ZONE_MOVABLE will be used to satisfy a memory request. This behavior is essential for the anti-fragmentation strategy of the kernel as outlined. A few flags can be set in the mask in addition to the zone modifiers. Figure 3-29 shows the layout of the mask and the constants associated with the bit positions. __GFP_DMA32 appears several times because it may be located at different places. Flags Zone modifiers _ _GFP_MOVABLE _ _GFP_RECLAIMABLE _ _GFP_THISNODE _ _GFP_HADRWALL _ _GFP_NOMEMMALLOC _ _GFP_ZERO _ _GFP_COMP _ _GFP_NO_GROW _ _GFP_NORETRY _ _GFP_NOFAIL _ _GFP_REPEAT _ _GFP_NOWARN _ _GFP_COLD _ _GFP_FS _ _GFP_IO _ _GFP_HIGH _ _GFP_WAIT _ _GFP_DMA _ _GFP_HIGHMEM _ _GFP_DMA (32) Figure 3-29: Layout of a GFP mask. In contrast to the zone modifiers, the additional flags do not limit the RAM segments from which memory can be allocated, but they do alter the behavior of the allocator. For example, they modify how aggressively a search is made for free memory. The following flags are defined in the kernel sources: /* Action modifiers - doesn’t change the zoning */ #define __GFP_WAIT ((__force gfp_t)0x10u) /* Can wait and reschedule? */ #define __GFP_HIGH ((__force gfp_t)0x20u) /* Should access emergency pools? */ #define __GFP_IO ((__force gfp_t)0x40u) /* Can start physical IO? */ #define __GFP_FS ((__force gfp_t)0x80u) /* Can call down to low-level FS? */ #define __GFP_COLD ((__force gfp_t)0x100u) /* Cache-cold page required */ #define __GFP_NOWARN ((__force gfp_t)0x200u) /* Suppress page allocation failure warning */ #define __GFP_REPEAT ((__force gfp_t)0x400u) /* Retry the allocation. Might fail */ #define __GFP_NOFAIL ((__force gfp_t)0x800u) /* Retry for ever. Cannot fail */ #define __GFP_NORETRY ((__force gfp_t)0x1000u)/* Do not retry. Might fail */ #define __GFP_NO_GROW ((__force gfp_t)0x2000u)/* Slab internal usage */ #define __GFP_COMP ((__force gfp_t)0x4000u)/* Add compound page metadata */ #define __GFP_ZERO ((__force gfp_t)0x8000u)/* Return zeroed page on success */ #define __GFP_NOMEMALLOC ((__force gfp_t)0x10000u) /* Don’t use emergency reserves */ #define __GFP_HARDWALL ((__force gfp_t)0x20000u) /* Enforce hardwall cpuset memory allocs */ #define __GFP_THISNODE ((__force gfp_t)0x40000u)/* No fallback, no policies */ #define __GFP_RECLAIMABLE ((__force gfp_t)0x80000u) /* Page is reclaimable */ #define __GFP_MOVABLE ((__force gfp_t)0x100000u) /* Page is movable */ 218 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 219 Chapter 3: Memory Management Some of the constants shown are used only in rare situations, so I won’t discuss them. The meanings of the most important constants are as follows: ❑ __GFP_WAIT indicates that the memory request may be interrupted; that is, the scheduler is free to select another process during the request, or the request can be interrupted by a more impor- tant event. The allocator is also permitted to wait for an event on a queue (and to put the process to sleep) before memory is returned. ❑ __GFP_HIGH is set if the request is very important, that is, when the kernel urgently needs mem- ory. This flag is always used when failure to allocate memory would have massive consequences for the kernel resulting in a threat to system stability or even a system crash. Despite the similarity in name, __GFP_HIGH has nothing to do with __GFP_HIGHMEM and must not be confused with it. ❑ __GFP_IO specifies that the kernel can perform I/O operations during an attempt to find fresh memory. In real terms, this means that if the kernel begins to swap out pages during memory allocation, the selected pages may be written to hard disk only if this flag is set. ❑ __GFP_FS allows the kernel to perform VFS operations. This must be prevented in kernel layers linked with the VFS layer because interactions of this kind could cause endless recursive calls. ❑ __GFP_COLD is set if allocation of a ‘‘cold‘‘ page that is not resident in the CPU cache is required. ❑ __GFP_NOWARN suppresses a kernel failure warning if allocation fails. There are very few occa- sions when this flag is useful. ❑ __GFP_REPEAT automatically retries a failed allocation but stops after a few attempts. __GFP_NOFAIL retries the failed allocation until it succeeds. ❑ __GFP_ZERO returns a page filled with zero bytes if allocation succeeds. ❑ __GFP_HARDWALL is meaningful on NUMA systems only. It limits memory allocation to the nodes associated with the CPUs assigned to a process. The flag is meaningless if a process is allowed to run on all CPUs (this is the default). It only has an explicit effect if the CPUs on which a process may run are limited. ❑ __GFP_THISNODE also only makes sense on NUMA systems. If the bit is set, then fallback to other nodes is not permitted, and memory is guaranteed to be allocated on either the current node or on an explicitly specified node. ❑ __GFP_RECLAIMABLE and __GFP_MOVABLE are required by the page mobility mechanism. As their names indicate, they mark that the allocated memory will be reclaimable or mov- able, respectively. This influences from which sublist of the freelist the page or pages will be taken. As the flags are used in combination and hardly ever on their own, the kernel classifies them into groups containing appropriate flags for a variety of standard situations. If at all possible, one of the follow- ing groups should always be used for memory allocation outside of memory management itself. (This requirement is reinforced by the fact that the names of the predefined groups do not begin with a double underscore — the usual convention for internal data and definitions in the kernel sources.) 219 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 220 Chapter 3: Memory Management #define GFP_ATOMIC (__GFP_HIGH) #define GFP_NOIO (__GFP_WAIT) #define GFP_NOFS (__GFP_WAIT | __GFP_IO) #define GFP_KERNEL (__GFP_WAIT | __GFP_IO | __GFP_FS) #define GFP_USER (__GFP_WAIT | __GFP_IO | __GFP_FS | __GFP_HARDWALL) #define GFP_HIGHUSER (__GFP_WAIT | __GFP_IO | __GFP_FS | __GFP_HARDWALL | \ __GFP_HIGHMEM) #define GFP_HIGHUSER_MOVABLE (__GFP_WAIT | __GFP_IO | __GFP_FS | \ __GFP_HARDWALL | __GFP_HIGHMEM | \ __GFP_MOVABLE) #define GFP_DMA __GFP_DMA #define GFP_DMA32 __GFP_DMA32 ❑ The meaning of the first three combinations is clear. GFP_ATOMIC is used for atomic allocations that may not be interrupted on any account and may also draw on the ‘‘emergency reserves‘‘ of memory. GFP_NOIO and GFP_NOFS explicitly exclude I/O operations and access to the VFS layer, respectively, but may be interrupted because __GFP_WAIT is set. ❑ GFP_KERNEL and GFP_USER are the default settings for kernel and user allocations, respectively. Their failure is not an immediate threat to system stability. GFP_KERNEL is far and away the most frequently used flag in the kernel sources. ❑ GFP_HIGHUSER is an extension of GFP_USER that is also used on behalf of userspace. It also permits the use of high-memory areas that can no longer be mapped directly. There is no disadvantage to using highmem pages because the address space of user processes is always organized by means of nonlinear page table assignments. GFP_HIGHUSER_MOVABLE is similar to GFP_HIGHUSER in purpose, but allocations will be satisfied from the virtual zone ZONE_MOVABLE. ❑ GFP_DMA is used for DMA allocations and is currently a simple synonym for __GFP_DMA; GFP_DMA32 is likewise a synonym for __GFP_GMA32. Allocation Macros Through the use of flags, zone modifiers, and the various allocation functions, the kernel offers a very flexible system of memory reservation. Nevertheless, all interface functions can be traced back to a single base function (alloc_pages_node). alloc_page and __get_free_page that reserve a single page are defined with the help of macros, as is alloc_pages. #define alloc_page(gfp_mask) alloc_pages(gfp_mask, 0) ... #define __get_free_page(gfp_mask) \ __get_free_pages((gfp_mask),0) #define __get_dma_pages(gfp_mask, order) \ __get_free_pages((gfp_mask) | GFP_DMA,(order)) Neither is the implementation of get_zeroed_page particularly difficult. alloc_pages used with the __GFP_ZERO flag reserves a page already filled with null bytes — only the address of the memory area associated with the page need be returned. 220 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 221 Chapter 3: Memory Management The clear_page standard function that must be implemented by all architectures helps alloc_pages fill pages with null bytes.19 __get_free_pages accesses alloc_pages, while alloc_pages, in turn, resorts to alloc_pages_node: #define alloc_pages(gfp_mask, order) \ alloc_pages_node(numa_node_id(), gfp_mask, order) mm/page_alloc.c fastcall unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order) { struct page * page; page = alloc_pages(gfp_mask, order); if (!page) return 0; return (unsigned long) page_address(page); } In this case, a proper function is used instead of a macro because the page instance returned by alloc_pages stillremainstobetranslatedintoamemoryaddressusingthehelperfunction page_address. At this point it is enough for us to know that the function yields the linear memory address of a page associated with the passed page instance. This is problematic with highmem pages, so I discuss the details of the function in Section 3.5.7. The unification of all API functions to a common base function — alloc_pages —isthuscomplete. Figure 3-30 shows the relationships among the various functions in a graphical overview. alloc_page get_zeroed_page __get_free_page __get_dma_pages get_free_pages alloc_pages alloc_pages_node Figure 3-30: Relationships among the allocation functions of the buddy system. page_cache_alloc and page_cache_alloc_cold are also convenience functions to yield cache-warm and cache-cold pages, respectively, by setting the __GFP_COLD modifier accordingly. Similarly, the memory-freeing functions can be reduced to a central function (__free_pages)invoked with different parameters: #define __free_page(page) __free_pages((page), 0) #define free_page(addr) free_pages((addr),0) 19Of course, pages could be filled with zeros by generic processor-independent code, but most CPUs feature special commands that do this much faster. 221 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 222 Chapter 3: Memory Management The relationship between free_pages and __free_pages is established by means of a function instead of a macro because the virtual address must first be converted to a pointer to struct page. mm/page_alloc.c void free_pages(unsigned long addr, unsigned int order) { if (addr != 0) { __free_pages(virt_to_page(addr), order); } } virt_to_page converts virtual memory addresses to pointers to page instances. Basically, this is the reverse of the page_address helper function introduced above. Figure 3-31 summarizes the relationships among the various memory-freeing functions in a graphical overview. free_page free_pages __free_pages __free_page Figure 3-31: Relationships among the memory-freeing functions of the buddy system. 3.5.5 Reserving Pages All API functions lead back to alloc_pages_node, which is a kind of ‘‘launch pad‘‘ for central implemen- tation of the buddy system. static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask, unsigned int order) { if (unlikely(order >= MAX_ORDER)) return NULL; /* Unknown node is current node */ if (nid < 0) nid = numa_node_id(); return __alloc_pages(gfp_mask, order, NODE_DATA(nid)->node_zonelists + gfp_zone(gfp_mask)); } Just a simple check is carried out to ensure that no overly large memory chunk is allocated. If a neg- ative node ID (which does not exist) is specified, the kernel automatically uses the ID that belongs to 222 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 223 Chapter 3: Memory Management the currently executing CPU. Work is then delegated to __alloc_pages to which an appropriate set of parameters is passed. Notice that gfp_zone is used to select the zone from which the allocation is supposed to be fulfilled. This is an important detail that can easily be missed! The kernel sources refer to this __alloc_pages as the ‘‘heart of the buddy system‘‘ because it deals with the essential aspects of allocation. Since a heart is an important thing to have, I shall introduce the function in detail below. Selecting Pages Let us therefore turn our attention to how page selection works. Helper Functions First, we need to define some flags used by the functions to control behavior when various watermarks are reached. mm/page_alloc.c #define ALLOC_NO_WATERMARKS 0x01 /* don’t check watermarks at all */ #define ALLOC_WMARK_MIN 0x02 /* use pages_min watermark */ #define ALLOC_WMARK_LOW 0x04 /* use pages_low watermark */ #define ALLOC_WMARK_HIGH 0x08 /* use pages_high watermark */ #define ALLOC_HARDER 0x10 /* try to alloc harder */ #define ALLOC_HIGH 0x20 /* __GFP_HIGH set */ #define ALLOC_CPUSET 0x40 /* check for correct cpuset */ The first flags indicate which watermark applies when the decision is made as to whether pages can be taken or not. By default (that is, there is no absolute need for more memory because of pressure exerted by other factors), pages are taken only when the zone still contains at least zone->pages_high pages. This corresponds to the ALLOC_WMARK_HIGH flag. ALLOC_WMARK_MIN or _LOW must be set accordingly in order to use the low (zone->pages_low) or minimum (zone->pages_min) setting instead. ALLOC_HARDER instructs the buddy system to apply the allocation rules more generously when memory is urgently needed; ALLOC_HIGH relaxes these rules even more when highmem is allocated. Finally, ALLOC_CPUSET tells the kernel to note that memory must be taken only from the areas associated with the CPUs that the current process is allowed to use — of course, this option only makes sense on NUMA systems. The flag settings are applied in the zone_watermark_ok function, which checks whether memory can still be taken from a given zone depending on the allocation flags set. mm/page_alloc.c int zone_watermark_ok(struct zone *z, int order, unsigned long mark, int classzone_idx, int alloc_flags) { /* free_pages my go negative - that’s OK */ long min = mark long free_pages = zone_page_state(z, NR_FREE_PAGES) - (1 << order) + 1; int o; if (alloc_flags & ALLOC_HIGH) min -= min / 2; if (alloc_flags & ALLOC_HARDER) min -= min / 4; 223 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 224 Chapter 3: Memory Management if (free_pages <= min + z->lowmem_reserve[classzone_idx]) return 0; for (o = 0; o < order; o++) { /* At the next order, this order’s pages become unavailable */ free_pages -= z->free_area[o].nr_free << o; /* Require fewer higher order pages to be free */ min >>= 1; if (free_pages <= min) return 0; } return 1; } Recall that zone_per_state allows for accessing the per-zone statistics. In this case, the number of free pages is obtained. Once the ALLOC_HIGH and ALLOC_HARDER flags have been interpreted (they reduce the minimum mark by a half or quarter of the current value, which makes the allocation effectively try hard or even harder), the function checks whether the number of free pages is less than the desired minimum plus the emergency reserve specified in lowmem_reserve. If not, the code iterates over all orders less than the current order and subtracts all pages in the current zone from free_pages (the o-fold left shift is necessary because nr_free stores the free page blocks). At the same time, the required number of free pages is halved for each zone. The allocation is freed if the kernel establishes that not enough pages are present after iterating over all low-memory zones. get_page_from_freelist is another important helper function used by the buddy system. It refers to the flags set and the allocation order to decide whether allocation can be made; if so, it initiates actual page allocation.20 mm/page_alloc.c static struct page * get_page_from_freelist(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, int alloc_flags) { struct zone **z; struct page *page = NULL; int classzone_idx = zone_idx(zonelist->zones[0]); struct zone *zone; ... /* * Scan zonelist, looking for a zone with enough free. * See also cpuset_zone_allowed() comment in kernel/cpuset.c. */ z = zonelist->zones; do { ... zone = *z; 20Notice that NUMA systems use a zone list cache that accelerates scanning through the zones. Although the cache is not active on UMA systems, it has some influence on the code below that I have removed for the sake of simplicity. 224 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 225 Chapter 3: Memory Management if ((alloc_flags & ALLOC_CPUSET) && !cpuset_zone_allowed_softwall(zone, gfp_mask)) continue; if (!(alloc_flags & ALLOC_NO_WATERMARKS)) { unsigned long mark; if (alloc_flags & ALLOC_WMARK_MIN) mark = zone->pages_min; else if (alloc_flags & ALLOC_WMARK_LOW) mark = zone->pages_low; else mark = zone->pages_high; if (!zone_watermark_ok(zone, order, mark, classzone_idx, alloc_flags)) continue; } ... A pointer to the fallback list is passed as parameter to the function. This list determines the order in which the other zones (and nodes) of the system are scanned if no pages are free in the desired zone. The layout and meaning of this data structure are discussed extensively in Section 3.4.1. The subsequent do loop does more or less exactly what would intuitively be expected as the simplest way of finding a suitable free memory block — it iterates over all zones of the fallback list. First of all, the ALLOC_* flags are interpreted (cpuset_zone_allowed_softwall is another helper function to check whether the given zone belongs to the allowed CPUs for the process). zone_watermark_ok then checks each zone to find out if enough pages are present and attempts to allocate a contiguous memory block. If one of these two conditions is not met — either there are not enough free pages or the request cannot be satisfied with contiguous pages — the next zone in the fallback list is checked in the same way. If the zone is suitable for the current request, buffered_rmqueue tries to remove the desired number of pages from it: mm/page_alloc.c ... page = buffered_rmqueue(*z, order, gfp_mask); if (page) { zone_statistics(zonelist, *z); break; } } while (*(++z) != NULL); return page; } We take a closer look at buffered_rmqueue in Section 3.5.4. If page removal was successful, the page(s) can be returned to the caller. Otherwise, the loop starts anew, and the next best zone is selected. Allocation Control As mentioned above, __alloc_pages is the main function of the buddy system. Now that we have dealt with all preparatory work and described all possible flags, we turn our attention to the relatively complex implementation of the function that is one of the lengthier parts of the kernel. Complexity arises above 225 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 226 Chapter 3: Memory Management all when too little memory is available to satisfy a request or when available memory is slowly running out. If sufficient memory is available, the necessary work is quickly done as the start of the code shows. mm/page_alloc.c struct page * fastcall __alloc_pages(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist) { const gfp_t wait = gfp_mask & __GFP_WAIT; struct zone **z; struct page *page; struct reclaim_state reclaim_state; struct task_struct *p = current; int do_retry; int alloc_flags; int did_some_progress; might_sleep_if(wait); restart: z = zonelist->zones; /* the list of zones suitable for gfp_mask */ if (unlikely(*z == NULL)) { /* * Happens if we have an empty zonelist as a result of * GFP_THISNODE being used on a memoryless node */ return NULL; } page = get_page_from_freelist(gfp_mask|__GFP_HARDWALL, order, zonelist, ALLOC_WMARK_LOW|ALLOC_CPUSET); if (page) goto got_pg; ... In the simplest scenario, allocation of a fresh memory area involves a single invocation of get_page_from_freelist to return the required number of pages (which is handled by the code at the label got_pg). The first memory allocation attempt is not particularly aggressive. A failure to find memory in any of the zones means that there isn’t much memory left but requires more than a moderate increase in effort from the kernel to find more memory (the big guns are brought out later). mm/page_alloc.c ... for (z = zonelist->zones; *z; z++) wakeup_kswapd(*z, order); alloc_flags = ALLOC_WMARK_MIN; if ((unlikely(rt_task(p)) && !in_interrupt()) || !wait) alloc_flags |= ALLOC_HARDER; if (gfp_mask & __GFP_HIGH) 226 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 227 Chapter 3: Memory Management alloc_flags |= ALLOC_HIGH; if (wait) alloc_flags |= ALLOC_CPUSET; page = get_page_from_freelist(gfp_mask, order, zonelist, alloc_flags); if (page) goto got_pg; ... } The kernel again iterates over all zones in the fallback list and invokes the wakeup_kswapd each time. As its name suggests, this function wakes the kswapd daemon responsible for swapping out pages. The task of the swapping daemons is complex and is therefore described in a separate chapter (Chapter 18). All you need note here is that fresh memory can be obtained by, for example, shrinking kernel caches and page reclaim, that is, writing back or swapping out rarely used pages. Both measures are initiated by the daemon. Once the swapping daemon has been woken, the kernel starts a new attempt to find a suitable memory chunk in one of the zones. This time it goes about its search more aggressively by adjusting the allocation flags to more promising values for the particular situation. In doing so, it reduces the watermark to its minimum value. ALLOC_HARDER is set for real-time processes and for calls with __GFP_WAIT that may not go to sleep. A further call of get_page_from_freelist with a changed set of flags tries to obtain the desired pages. If this also fails, the kernel resorts to more drastic measures: mm/page_alloc.c rebalance: if (((p->flags & PF_MEMALLOC) || unlikely(test_thread_flag(TIF_MEMDIE))) && !in_interrupt()) { if (!(gfp_mask & __GFP_NOMEMALLOC)) { nofail_alloc: /* go through the zonelist yet again, ignoring mins */ page = get_page_from_freelist(gfp_mask, order, zonelist, ALLOC_NO_WATERMARKS); if (page) goto got_pg; if (gfp_mask & __GFP_NOFAIL) { congestion_wait(WRITE, HZ/50); goto nofail_alloc; } } goto nopage; } ... If PF_MEMALLOC is set or if the TIF_MEMDIE flag is set for the task (in both cases, the kernel must not be in the interrupt context). get_page_from_freelist tries once more to obtain the desired pages, but this time, watermarks are completely ignored because ALLOC_NO_WATERMARKS is set. Whereas the PF_MEMALLOC condition usually only applies when the call for more memory originates from the allocator itself, TIF_MEMDIE is set when a thread has just been hit by the OOM killer. 227 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 228 Chapter 3: Memory Management The search can come to an end here for two reasons: 1. __GFP_NOMEMALLOC is set. This flag prohibits using the emergency reserve (which can well be if the watermarks are ignored), so calling get_page_from_freelist without obeying the watermarks is forbidden. The kernel can do nothing more than fail ultimately in this case by jumping to the noopage label, where the failure is reported to the user with a kernel message, and a NULL pointer is returned to the caller. 2. get_page_from_freelist fails despite watermarks being ignored. In this case, the search is also aborted and terminates with an error message. However, if __GFP_NOFAIL is set, the kernel goes into an endless loop (implemented by branching back to the nofail_alloc label) to first wait (by means of congestion_wait) for the end of ‘‘congestion‘‘ in the block layer, which can arise when pages are reclaimed (see Chapter 18). Allocation is then attempted again until it succeeds. If PF_MEMALLOC is not set, the kernel still has some more options to try, but these require going sleep. This is necessary to allow kswapd to make some progress. The kernel now enters on a slow path where time-consuming operations begin. A prerequisite is that the __GFP_WAIT flag is set in the allocation mask because the subsequent actions can put the process to sleep. mm/page_alloc.c /* Atomic allocations - we can’t balance anything */ if (!wait) goto nopage; cond_schedule(); ... Recall that wait is 1 if the bit is set, and 0 otherwise. If this flag is not set, allocation is aborted at this point. Before further attempts are made, the kernel provides the opportunity of rescheduling by means of cond_resched. This prevents too much time being spent searching for memory so that other tasks are left unfulfilled. The paging mechanism provides an as-yet-unused option for swapping rarely used pages out to a block medium to create more space in RAM. However, this option is very time-consuming and can sleep. try_to_free_pages is the respective helper function that attempts to find pages that are currently not urgently needed and can therefore be swapped out. It is invoked after the PF_MEMALLOC flag has been set for the task to indicate to the remaining kernel code that all subsequent memory allocations are needed in the search for memory. mm/page_alloc.c /* We now go into synchronous reclaim */ p->flags |= PF_MEMALLOC; ... did_some_progress = try_to_free_pages(zonelist->zones, order, gfp_mask); ... p->flags &= ~PF_MEMALLOC; cond_resched(); ... 228 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 229 Chapter 3: Memory Management ThecallisframedbycodethatsetstheabovePF_MEMALLOC flag. It may be necessary for try_to_free_pages to allocate new memory for its own work. As this additional memory is needed to obtain fresh memory (a rather paradoxical situation), the process should, of course, enjoy maximum priority in terms of memory management from this point on — this is achieved by setting the above flag. Recall that only a few lines ago, a very aggressive attempt at memory allocation was tried conditioned on PF_MEMALLOC being set. Besides, setting the flag ensures that try_to_free_pages is not called recursively because __alloc_pages will already have aborted before if PF_MEMALLOC is set. try_to_free_pages is itself a lengthy and complex function whose implementation I won’t discuss here. Instead, see Chapter 18, which includes a detailed description of the underlying mechanism. At the moment, it is sufficient to know that the function selects pages not recently in very active use and writes them to the swap area to free space in RAM memory. The number of freed pages by try_to_free_pages is returned as the result. try_to_free_pages acts only on the node containing the desired zone. All other nodes are ignored. If more than one page is to be allocated, pages from the per-CPU cache are brought back into the buddy system: mm/page_alloc.c if (order != 0) drain_all_local_pages(); How this is technically done is not of relevance here, so it is not necessary to discuss drain_all_ local_pages in detail. The next kernel action — could it be any different — is to invoke get_page_from_freelist to attempt allocation again if some pages could be freed by try_to_free_pages: mm/page_alloc.c if (likely(did_some_progress)) { page = get_page_from_freelist(gfp_mask, order, zonelist, alloc_flags); if (page) goto got_pg; } else if ((gfp_mask & __GFP_FS) && !(gfp_mask & __GFP_NORETRY)) { ... If the kernel may perform calls that affect the VFS layer and is not hindered by GFP_NORETRY, the out-of- memory (OOM) killer is invoked: mm/page_alloc.c /* The OOM killer will not help higher order allocs so fail */ if (order > PAGE_ALLOC_COSTLY_ORDER) { clear_zonelist_oom(zonelist); 229 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 230 Chapter 3: Memory Management goto nopage; } out_of_memory(zonelist, gfp_mask, order); goto restart; } Without going into the details of implementation, note that out_of_memory picks one task that the kernel deems particularly guilty of reserving all the memory — and kills it. This, hopefully, will lead to a good number of free pages, and the allocation is retried by jumping to the label restart. However, it is unlikely that killing a process will immediately lead to a continuous range of more than 2PAGE_COSTLY_ORDER pages (where PAGE_COSTLY_ORDER_PAGES is usually set to 3), so the kernel spares one innocent task’s life if such a big allocation was requested, does not perform out-of-memory killing, and admits failure by jumping to nopage. What happens if __GFP_NORETRY is set or the kernel is not allowed to use operations that might affect the VFS layer? In this case, the size of the desired allocation comes in: mm/page_alloc.c ... do_retry = 0; if (!(gfp_mask & __GFP_NORETRY)) { if ((order <= PAGE_ALLOC_COSTLY_ORDER) || (gfp_mask & __GFP_REPEAT)) do_retry = 1; if (gfp_mask & __GFP_NOFAIL) do_retry = 1; } if (do_retry) { congestion_wait(WRITE, HZ/50); goto rebalance; } nopage: if (!(gfp_mask & __GFP_NOWARN) && printk_ratelimit()) { printk(KERN_WARNING "%s: page allocation failure." " order:%d, mode:0x%x\n", p->comm, order, gfp_mask); dump_stack(); show_mem(); } got_pg: return page; } The kernel goes into an endless loop if the allocation size is less than 2PAGE_ALLOC_COSTLY_ORDER = 8 pages, or the __GFP_REPEAT flag is set. GFP_NORETRY must naturally not be set in both cases since the caller does not like to retry the allocation in this case. The kernel branches back to the rebalance label that introduces the slow path and remains there until a suitable memory chunk is finally found — with reservations of this size, the kernel can assume that the endless loop won’t last all that long. Beforehand, the kernel invokes congestion_wait to wait for the block layer queues to free up (see Chapter 6) so that it has a chance to swap pages out. The kernel also goes into the above endless loop if the desired allocation order is greater than 3 but the __GFP_NOFAIL flag is set — the flag does not allow failing on any account. 230 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 231 Chapter 3: Memory Management If this is not the case, the kernel gives up and can do nothing more than return a NULL pointer to the user, and print a warning message that a memory request could not be fulfilled. Removing the Selected Pages Two things remain to be done once the kernel has found a suitable zone with sufficient free pages for the allocation. First, it must be checked whether the pages are contiguous (up to now it only knows how many free pages there are). And second, the pages must be removed from the free_lists in the buddy fashion, and this may make it necessary to break up and rearrange memory regions. The kernel delegates this work to buffered_rmqueue as discussed in the previous section. Figure 3-32 shows the essential steps of the function. Fill in per-CPU cache if necessary Appropriate page found? Remove page Return Null pointer Ye s No Yes== 0? No buffered_rmqueue order prep_new_page _ _rmqueue prep_new_page Figure 3-32: Code flow diagram for buffered_rmqueue. The kernel performs optimization if only a single page is to be allocated, that is, if the allocation order is 0 because 20 = 1. The page is not taken directly from the buddy system but from the per-CPU page cache (recall that this cache provides a CPU-local list of cache-hot and cache-cold pages; the required data structures are described in Section 3.2.2). As usual, some variables need to be set up first: mm/page_alloc.c static struct page * buffered_rmqueue(struct zone *zone, int order, gfp_t gfp_flags) { unsigned long flags; struct page *page; int cold = !!(gfp_flags & __GFP_COLD); int migratetype = allocflags_to_migratetype(gfp_flags); If GFP_COLD is set in the allocation flags, then a cache-cold page must be taken from the per-CPU allocator if any exists. The double negation ensures that cold is either 0 or 1.21 It is also essential to determine the 21If just gfp_flags & __GFP_COLD were used, then the numerical value of cold would be the bit value of __GFP_COLD if the flag is set. This would not allow using cold as an index into a binary array. 231 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 232 Chapter 3: Memory Management migrate list from the allocation flags. The previously introduced function allocflags_to_migratetype (see Section 3.5.2) comes in handy here. When only a single page is desired, the kernel tries to speed up the request with the help of the per-CPU cache. If the cache is empty, the kernel takes the opportunity to check the cache fill level. mm/page_alloc.c again: if (order == 0) { struct per_cpu_pages *pcp; page = NULL; pcp = &zone_pcp(zone, get_cpu())->pcp[cold]; if (!pcp->count) pcp->count = rmqueue_bulk(zone, 0, pcp->batch, &pcp->list); if (unlikely(!pcp->count)) goto failed; } ... Once the appropriate (i.e., hot or cold) per-CPU list for the current processor has been selected, rmqueue_bulk is invoked to refill the cache. I won’t reproduce the function here as it simply removes pages from the normal buddy system and adds them to the cache. However, it is important to note that buffered_rmqueue stores the migrate type of the page in the private element of struct page. This will become important when pages are taken off the cache: mm/page_alloc.c /* Find a page of the appropriate migrate type */ list_for_each_entry(page, &pcp->list, lru) if (page_private(page) == migratetype) break; /* Allocate more to the pcp list if necessary */ if (unlikely(&page->lru == &pcp->list)) { pcp->count += rmqueue_bulk(zone, 0, pcp->batch, &pcp->list, migratetype); page = list_entry(pcp->list.next, struct page, lru); } list_del(&page->lru); pcp->count-- } else { page = __rmqueue(zone, order); if (!page) goto failed; } ... The kernel iterates over all pages on the per-CPU cache and checks if the page of the desired migrate type is available. This need not be the case if the cache has been refilled by a previous call with pages of a different migrate type. If no suitable page is found, some more pages with the currently desired migrate 232 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 233 Chapter 3: Memory Management type are added to the cache, and one page is removed from the per-CPU list and processed further below. If more than one page is to be allocated (as handled in the else branch), the kernel calls __rmqueue to select a suitable page block from the zone’s buddy lists. If necessary, the function automatically breaks down larger blocks and puts unused parts back in the lists (how this is done is described below). Caution: It can be the case that there are enough free pages in the zone to satisfy the allocation request, but that the pages are not contiguous.Inthiscase,__rmqueue fails, and a NULL pointer is returned. Since all failures are handled by jumping to the label failed, it is guaranteed that page points to a valid sequence of pages once the kernel gets to the current point. Before the pointer can be returned, prep_new_page has to prepare the pages for life in the kernel (note that the function returns a positive value if something is wrong with the selected pages; in this case, the allocation is restarted from the beginning): mm/page_alloc.c if (prep_new_page(page, order, gfp_flags)) goto again; return page; failed: ... return NULL; } prep_new_page performs several checks on the pages to ensure that they leave the allocator in a perfect state — this means, in particular, that the page must not be in use in existing mappings and no incorrect flags like PG_locked or PG_buddy may be set because this would imply that the page is in use somewhere else and should not be on the free list. Normally, however, no error should occur because this would imply a kernel error elsewhere. The function also sets the following default flags used for each new page: mm/page_alloc.c static int prep_new_page(struct page *page, int order, gfp_t gfp_flags) { page->flags &= ~(1 << PG_uptodate | 1 << PG_error | 1 << PG_readahead | 1 << PG_referenced | 1 << PG_arch_1 | 1 << PG_owner_priv_1 | 1 << PG_mappedtodisk); ... The meanings of the individual bits are given in Section 3.2.2. prep_new_page is also invoked to set the reference counters of the first page instance involved to the initial value of 1. Besides, some more work is required depending on the page flags: mm/page_alloc.c if (gfp_flags & __GFP_ZERO) prep_zero_page(page, order, gfp_flags); if (order && (gfp_flags & __GFP_COMP)) prep_compound_page(page, order); return 0; } 233 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 234 Chapter 3: Memory Management ❑ If __GFP_ZERO is set, prep_zero_page fills the page with zero bytes using an efficient, architecture-specific function. ❑ If __GFP_COMP is set and more than one page has been requested, the kernel must group the pages into compound pages. The first page is called the head page, while all other pages are called tail pages. The structure of compound pages is shown in Figure 3-33. All pages are identified as compound pages by the PG_compound bit. The private elements of the page instance of all pages — even the head page itself — point to the head page. Besides, the kernel needs to store information on how to free the compound page. This requires both a function to free the page and information on how many pages compose the compound page. The LRU list element of the first tail page is abused for this purpose: A pointer to a destructor function is thus kept in lru.next, while the allocation order is stored in lru.prev.Noticethat the lru element cannot be used for this purpose because it is required if the compound page is to be kept on a kernel list. Why is this information required? The kernel can combine multiple adjacent physical pages to a so-called huge-TLB page. When a userland application works with large chunks of data, many processors allow using huge-TLB pages to keep the data in memory. Since the page size of a huge-TLB page is larger than the regular page size, this reduces the amount of information that must be stored in the translation lookaside buffer (TLB), that, in turn, reduces the probability of a TLB cache miss — and thus speeds things up.22 However, huge-TLB pages need to be freed differently than compound pages composed of multiple regular pages, so an explicit destructor is required. free_compound_pages is used for this purpose. The function essentially determines the page order stored in lru.prev and frees the pages one after another when the compound page is freed. The auxiliary function prep_compound_page is used to arrange the described structure. PG_compound private Iru.next Iru.prev PG_compound private Iru.next 2n pages Iru.prev PG_compound private PG_compound private struct page free_compound_page n Figure 3-33: Higher-order allocations generate compound pages in which the individual pages are linked. The __rmqueue Helper Function The kernel uses the __rmqueue function (whose purpose is evident from the preceding description), which acts as a gatekeeper to penetrate into the innermost core of the buddy system: mm/page_alloc.c static struct page *__rmqueue(struct zone *zone, unsigned int order, int migratetype) 22Huge-TLB pages are created at boot time and kept in a special cache. The kernel parameter hugepages allows for specifying how many huge-TLB pages are to be created, and applications can request them via the special filesystem hugetlbfs. The library libhugetlbfs allows userland applications to use huge-TLB pages without direct interference with this filesystem. 234 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 235 Chapter 3: Memory Management { struct page *page; page = __rmqueue_smallest(zone, order, migratetype); if (unlikely(!page)) page = __rmqueue_fallback(zone, order, migratetype); return page; } By reference to the desired allocation order, the zone from which the pages are to be removed, and the migrate type, __rmqueue_smalles scans the page lists until it finds a suitable contiguous chunk of memory. When it does this, buddies can be split as described in Chapter 1. Should the desired migrate list not be able to satisfy the request, then other migrate lists are tried as an emergency measure in __rmqueue_fallback. The implementation of __rmqueue_smallest is not very long. Essentially, it consists of a loop that iterates over the list of migrate-type-specific free pages list of the zone in ascending order until an appropriate entry is found. mm/page_alloc.c static struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, int migratetype) { unsigned int current_order; struct free_area * area; struct page *page; /* Find a page of the appropriate size in the preferred list */ for (current_order = order; current_order < MAX_ORDER; ++current_order) { area = &(zone->free_area[current_order]); if (list_empty(&area->free_list[migratetype])) continue; page = list_entry(area->free_list[migratetype].next, struct page, lru); list_del(&page->lru); rmv_page_order(page); area->nr_free--; __mod_zone_page_state(zone, NR_FREE_PAGES, - (1UL << order)); expand(zone, page, order, current_order, area, migratetype); return page; } return NULL; } The search begins at the entry for the desired allocation order. Smaller areas are of no use because the pages allocated must be contiguous. Recall that all pages of a given allocation order are again subdivided into migrate-type-specific lists, and the proper one is selected. 235 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 236 Chapter 3: Memory Management Checking for a suitable chunk of memory is very simple: If an element is present in the examined list, it can be used because it contains as many contiguous pages as needed. Otherwise, the kernel selects the next higher allocation order and continues the search there. Once a memory chunk has been removed from the list with list_del,itsremovalmustbenotedby decrementing the nr_free element of struct free_area by 1. The per-zone statistics of the current zone must also be updated accordingly with __mod_zone_page_state. rmv_page_order is a helper func- tion that deletes the PG_buddy bit from the page flags — the page is not contained in the buddy system anymore — and sets the private element of struct page to 0. If the memory chunk to be allocated is smaller than the selected range of contiguous pages, that is, if the pages stem from a higher allocation order than required because no suitable smaller block was available, it must be split into smaller segments in accordance with the principles of the buddy system. This is done by the expand function. mm/page_alloc.c static inline struct page * expand(struct zone *zone, struct page *page, int low, int high, struct free_area *area) int migratetype) { unsigned long size = 1 << high; while (high > low) { area--; high--; size >>= 1; list_add(&page[size].lru, &area->free_list[migratetype]); area->nr_free++; set_page_order(&page[size], high); } return page; } This function uses a whole range of parameters. The meanings of page, zone,andarea are obvious. index specifies the index position of the buddy pair in the allocation bitmap, low is the desired allocation order, and high indicates the order from which the memory found was taken. migratetype sticks to its name and denotes the migrate type. It is best to look at the code step-by-step to understand how it works. Let us assume the following sit- uation: A block with order = 3 is to be allocated. There is no block of this size in RAM, so the kernel selects a block with order = 5 instead. For the sake of simplicity, this is located at index = 0. The function is therefore invoked with the following parameters. expand(page,index=0,low=3,high=5,area) Figure 3-34 illustrates the steps described below that are needed to split the page (the previous contents of the free_area lists are not shown, only the new pages). 1. The value of size is initialized to 2high = 25 = 32. The allocated memory area has already been removed from the free_area list in __rmqueue and is therefore shown with dashed lines in Figure 3-34. 2. In the first loop pass, the kernel switches to the migrate-type-specific free_area list with the next smaller memory units, namely, area=4. Analogously, the area size reduces to size=16 236 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 237 Chapter 3: Memory Management (calculated by size >> 1). The front half of the initial area is inserted in the free_area list for order=4. 5 4 3 2 1 page mem_map Area 5 4 3 2 1 page mem_map Area 5 4 3 2 1 page mem_map Area Figure 3-34: Steps performed by expand when splitting a memory area. Only the first page instance of a memory area is needed by the buddy system for management purposes; the size of the area is derived automatically from the list in which the page is located. 3. The index of the rear memory area with size = 16 can be calculated by adding size to index, thus skipping the next 16 bits in the allocation bitmap. Because all page instances are in lin- ear succession in memory, the pointer to page must also be increased by 16 to arrive at the corresponding page instance. The position of the page pointer is indicated by an arrow in Figure 3-34. 4. The next loop pass places the first half of the remaining 16 units on the free_area list with size=8.Bothindex and page are then increased by 8 units. The function has now arrived at the desired size unit, and the page pointer can be returned as the result. From the figure, it is evident that the last 8 pages of the original area of 32 pages are used; all other pages are in the appropriate free_area lists of the buddy system. 237 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 238 Chapter 3: Memory Management The kernel always uses the migrate-type-specific free_area list and does not change the migrate type of any page during the process. The set_page_order helper function invoked in each step is responsible for setting the private flag of the first instance of struct page in each block to the order currently being processed and for assigning the PG_buddy bit to the page. This indicates that the block is managed by the buddy system. If no contiguous memory area is available on the migrate-type-specific list, __rmqueue_smallest returns a NULL pointer. The kernel then tries to fulfill the request using other migrate lists based on the fallback order. The task is delegated to __rmqueue_fallback. Recall from Section 3.5.2 that the fallback order for migrate types is defined in the fallbacks array. First of all, the function iterates once again over the various allocation order lists: mm/page_alloc.c static struct page *__rmqueue_fallback(struct zone *zone, int order, int start_migratetype) { struct free_area * area; int current_order; struct page *page; int migratetype, i; /* Find the largest possible block of pages in the other list */ for (current_order = MAX_ORDER-1; current_order >= order; --current_order) { for (i = 0; i < MIGRATE_TYPES - 1; i++) { migratetype = fallbacks[start_migratetype][i]; ... However, not just the desired migrate type, but also different migrate types as specified in the fallback list are considered. Notice that the function iterates from large to small allocation orders! This is done contrary to the usual strategy, because the kernel wants to take a maximally big block out of foreign allocation lists if this cannot be avoided. If smaller blocks were favored, this would introduce fragmentation into the other zone because blocks of different migrate types would be mixed, and this is clearly undesired. The special zone MIGRATE_RESERVE contains emergency reservations and requires special treatment, discussed below. If the free list for the currently considered migrate type contains free page blocks, the request can be satisfied from there: mm/page_alloc.c /* MIGRATE_RESERVE handled later if necessary */ if (migratetype == MIGRATE_RESERVE) continue; area = &(zone->free_area[current_order]); if (list_empty(&area->free_list[migratetype])) continue; page = list_entry(area->free_list[migratetype].next, struct page, lru); area->nr_free--; ... 238 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 239 Chapter 3: Memory Management Recall that migrate lists are the basis for the page mobility approach that is used to keep memory frag- mentation as low as possible. Low memory fragmentation means that larger contiguous page blocks are available even after the system has been running for a longer time. As discussed in Section 3.5.2, the notion of how big a larger block is given by the global variable pageblock_order, which defines the order for a large block. If it is required to break a block of free pages from another migration list, the kernel has to choose what to do with the remaining pages. If the rest itself qualifies as a large block, it makes sense to transfer the whole block to the migrate list of the allocation type to mitigate fragmentation. The kernel is more aggressive about moving free pages from one migrate list to another if an allocation is performed for reclaimable memory. Allocations of this type often appear in bursts, for instance, when updatedb is running, and could therefore scatter many small reclaimable portions across all migrate lists. To avoid this situation, remaining pages for MIGRATE_RECLAIMABLE allocations are always transferred to the reclaimable migrate list. The kernel implements the described policy as follows: mm/page_alloc.c /* * If breaking a large block of pages, move all free * pages to the preferred allocation list. If falling * back for a reclaimable kernel allocation, be more * agressive about taking ownership of free pages */ if (unlikely(current_order >= (pageblock_order >> 1)) || start_migratetype == MIGRATE_RECLAIMABLE) { unsigned long pages; pages = move_freepages_block(zone, page, start_migratetype); /* Claim the whole block if over half of it is free */ if (pages >= (1 << (pageblock_order-1))) set_pageblock_migratetype(page, start_migratetype); migratetype = start_migratetype; } ... move_freepages tries to move the complete page block with 2pageblock_order pages in which the current allocation is contained to the new migrate list. However, only free pages (i.e., those with the PG_buddy bit set) are moved. Additionally, move_freepages also obeys zone boundaries, so the total number of pages can be smaller than a complete large page block. If, however, more than one-half of a large page block is free, then set_pageblock_migratetype claims the complete block (recall that the function always works on groups with pageblock_nr_pages pages). Finally, the kernel can remove the page block from the list, and use expand to place the unused parts of a largerblockbackonthebuddysystem. mm/page_alloc.c /* Remove the page from the freelists */ list_del(&page->lru); 239 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 240 Chapter 3: Memory Management rmv_page_order(page); __mod_zone_page_state(zone, NR_FREE_PAGES, -(1UL << order)); ... expand(zone, page, order, current_order, area, migratetype); return page; } } ... Notice that the new migrate type is used by expand if the kernel has decided to change it before. Other- wise, the remainders are put back onto their original migrate list. Finally, one more scenario must be considered: What if the allocation cannot be satisfied despite all page orders and all migrate types have been taken into account? In this case, the kernel can try to fulfill the allocation from the MIGRATE_RESERVE list, which serves as a last resort: mm/page_alloc.c /* Use MIGRATE_RESERVE rather than fail an allocation */ return __rmqueue_smallest(zone, order, MIGRATE_RESERVE); } 3.5.6 Freeing Pages __free_pages is the base function used to implement all functions of the kernel API. Its code flow dia- gram is shown in Figure 3-35. Yes No __free_pages _ _free_pages_ok __free_one_page free_hot_pageSingle page? Figure 3-35: Code flow diagram for __free_pages. __free_pages first establishes whether a single page or a larger contiguous block is to be freed. If a single page is freed, it is not returned to the buddy system but is placed in the per-CPU cache — in the warm list for all pages that are highly likely to reside in the CPU cache. For this purpose, the kernel provides the free_hot_page helper function, which is a parameter conversion function for free_hot_cold_page that is invoked in turn. If free_hot_cold_page determines that the number of pages in the per-CPU cache exceeds the limit set by pcp->count, a whole batch of pages — whose size is specified by pcp->batch —isreturned to the buddy system. This strategy is known as lazy coalescing.Itpreventslargenumbersofwasteful coalescing operations that would be carried out if single pages were returned to the buddy system and then immediately split to satisfy subsequent allocation requests. The free_pages_bulk function is used to return pages to the buddy system. 240 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 241 Chapter 3: Memory Management If the lazy coalescing limit is not exceeded, the page is simply kept on the per-CPU cache. However, it is important that the private element be set to the migrate type of the page. As described, this allows allocations fulfilled from the per-CPU cache to pick only pages of the proper migrate type. If more than one page is freed, then __free_pages delegates work (after a detour that is not interesting for our purposes) to __free_pages_ok and finally to __free_one_page. Despite the name, the function not only handles freeing of single pages, but also takes compound pages into account. mm/page_alloc.c static inline void __free_one_page (struct page *page, struct zone *zone, unsigned int order) This function is the cornerstone of memory freeing. The relevant area is added to the appropriate free_area list of the buddy system. When buddy pairs are freed, the function coalesces them into a contiguous area that is then placed in the next higher free_area list. If this reunites a further buddy pair, it is also coalesced and moved to a higher list. This procedure is repeated until all possible buddy pairs have been coalesced and the changes have been propagated upward as far as possible. However, this doesn’t answer the question as to how the kernel knows that both parts of a buddy pair are located on the list of free pages. To place a page group back into the buddy system, the kernel must be able to compute two things: the address of the potential buddy and the index of the combined page group if both buddies can be recombined. Two auxiliary functions are provided for this purpose: mm/page_alloc.c static inline struct page * __page_find_buddy(struct page *page, unsigned long page_idx, unsigned int order) { unsigned long buddy_idx = page_idx ^ (1 << order); return page + (buddy_idx - page_idx); } static inline unsigned long __find_combined_index(unsigned long page_idx, unsigned int order) { return (page_idx & ~(1 << order)); } It is advantageous to remember that the operator ^ performs a bitwise XOR operation. The calculations performed by the function will be clarified by an example immediately. First, we need to introduce one more helper function, though. The page index of the buddy is not enough. The kernel must also ensure that all pages, belonging to the buddy are free and thus contained in the buddy system to be able to merge both pairs: mm/page_alloc.c static inline int page_is_buddy(struct page *page, struct page *buddy, int order) { 241 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 242 Chapter 3: Memory Management ... if (PageBuddy(buddy) && page_order(buddy) == order) { return 1; } return 0; } If the first page of the buddy group is in the buddy system, then the PG_buddy bit of the corresponding struct page instance is set. This, however, is not sufficient to reunite two buddies: When freeing a page group with 2order pages, the kernel must ensure that 2order pages of the second buddy are contained in the buddy system. This is easy to check because the page order of the free group is stored in the first private element of the struct page instance of a free group, and page_order reads this value. Note that page_is_buddy is slightly more complicated in reality because it needs to account for memory holes, but this is omitted to simplify matters. Table 3-8: Calculations When a Page is Placed Back into the Buddy System. order page_idx buddy_index - page_index __find_combined_index 010 1 10 110 -2 8 28 4 8 38 -8 0 The following code determines whether a buddy pair can be coalesced: mm/page_alloc.c static inline void __free_one_page(struct page *page, struct zone *zone, unsigned int order) { int migratetype = get_pageblock_migratetype(page); ... while (order < MAX_ORDER-1) { unsigned long combined_idx; struct page *buddy; buddy = __page_find_buddy(page, page_idx, order); if (!page_is_buddy(page, buddy, order)) break; /* Move the buddy up one level. */ list_del(&buddy->lru); zone->free_area[order].nr_free--; rmv_page_order(buddy); combined_idx = __find_combined_index(page_idx, order); page = page + (combined_idx - page_idx); page_idx = combined_idx; order++; } ... 242 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 243 Chapter 3: Memory Management The routine tries to free a page group of order order. Because it is possible not only that the current group can be merged with its direct buddy, but also that higher-order buddies can be merged, the kernel needs to find the maximal allocation order for which this is possible. The action of the code is best understood by means of an example. Imagine that an order 0 allocation, that is, a single page, is freed, and let this page have the page index 10. The required calculations are found in Table 3-8, and Figure 3-36 visualizes the process step-by-step. We assume that page 10 is the last missing link that allows for coalescing two buddies of order 3 to form a new range of order 4. order 4 0781015 pi = 0 ci = 0 buddy = 0pi = 8order 3 order 2 order 1 order empty page used page returned page 0 ci = 8 buddy = 12pi = 8 ci = 8 buddy = 8pi = 10 pi = 10 ci = 10 buddy = 11 pi = 10 Figure 3-36: Returning a page into the buddy system can cause higher-order allocations to be coalesced. pi stands for page_index, while ci denotes combined_index. The first loop pass finds page number 11 as the buddy for page 10. Since not the page number of the buddy, but a pointer to the corresponding page instance is required, the difference buddy_idx - page_idx is of relevance: It denotes the difference between the current page and its buddy, and adding it to the page pointer will deliver a pointer to the page instance of the buddy. This pointer is required by page_is_buddy to check if the buddy is free. As per Figure 3-36, this is luckily the case, so the buddies can be combined. This requires that page number 11 is temporarily removed from the buddy system because it will be reintegrated as part of a larger block later. The page instance is taken off the free list, and rmv_page_order clears the PG_buddy flag and the private data. Computing the index of the combined group in __find_combined_index delivers 10, because the 2-page buddy block starts at this page number. At the end of each loop step, the page pointer is set to point to the first page in the new buddy group, but in this case, nothing needs to be modified. The next loop pass works similarly, but now for order=1; that is, the kernel tries to combine two 2- page buddies into a 4-page group. The buddy of the [10, 11] page group starts at page number 8, so the difference buddy_index - page_index is negative. Naturally, there’s nothing preventing a buddy from being on the left-hand side of the current page group. The combined index of the merged group is 8, so the page pointer has to be updated accordingly after page_is_buddy has ensured that all pages of the new buddy (i.e., pages 8 and 9) are contained in the buddy system. 243 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 244 Chapter 3: Memory Management The loop continues until the order 4. This page group cannot be merged with its buddy because the buddy is not empty, as the figure shows. Consequently, page_is_buddy does not allow for merging the two regions, and the loop is left. Finally, the 24 = 16 page region must now be placed on the free lists of the buddy system. This is not very complicated: mm/page_alloc.c set_page_order(page, order); list_add(&page->lru, &zone->free_area[order].free_list[migratetype]); zone->free_area[order].nr_free++; } Notice that the allocation order of the page group is preserved in the private element of the first page instance of the group. This way the kernel knows that not only page 0, but also the whole range [0, 15], is free and in the buddy system. 3.5.7 Allocation of Discontiguous Pages in the Kernel Physically contiguous mappings are best for the kernel for the reasons given above, but they cannot always be used successfully. When a larger chunk of memory is allocated, it may be that it is not available in a contiguous block despite all efforts made to avoid this situation. This is not a problem in userspace because normal processes are designed to use the paging mechanism of the processor even though this is costly in terms of speed and TLBs. The same technique can also be applied in the kernel. As discussed in Section 3.4.2, the kernel reserves a chunk of its virtual address space so that it can set up contiguous mappings in them. As Figure 3-37 shows, a memory zone for the management of discontiguous memory on IA-32 follows the direct mapping of the first 892 MiB of RAM after an intervening safety gap of 8 MiB. This segment has all the properties of a linear address space; the pages assigned to it can be located anywhere in RAM memory. This is achieved by modifying the kernel page tables responsible for this area. Direct physical page mappings Protection gap vmalloc-Areas 8 MiB Figure 3-37: vmalloc area in the kernel’s virtual address space on IA-32 systems. A self-contained area separated from the other areas by a memory page is assigned to each vmalloc allocation. Like the boundary between direct mappings and the vmalloc area, the purpose of this is to safeguard against incorrect memory access operations; these occur only as a result of kernel faults and 244 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 245 Chapter 3: Memory Management should be reported by a system error message rather than allowing the data of other kernel parts to be overwritten unnoticed. Because this separation is made in virtual address space, no valuable real memory pages are wasted. Reserving Memory with vmalloc vmalloc is the interface function used by the kernel code to request memory that is not necessarily con- tiguous in physical memory but is always linear in virtual memory. void *vmalloc(unsigned long size); Just one parameter is required to specify the size of the required memory area — in contrast to the func- tions discussed earlier, the size unit is not pages but bytes, as is common in userspace programming. The best-known example of vmalloc use is in the module implementation of the kernel. Because modules can be loaded at any time, there is no guarantee — particularly if the system has been up and running for a long time — that sufficient contiguous memory will be available for the sometimes voluminous module data. This problem can be circumvented by using vmalloc if sufficient memory can be pieced together from smaller chunks. vmalloc is also invoked at about 400 other places in the kernel, particularly in device and sound drivers. Because the memory pages used for vmalloc must in any case be actively mapped in kernel address space, it is obviously preferable to use pages from ZONE_HIGHMEM for this purpose. This allows the kernel to conserve the more valuable lower zones without incurring any added disadvantages. For this reason, vmalloc (along with the mapping functions discussed in Section 3.5.8) is one of the few occasions when the kernel is able to use highmem pages for its own purposes (and not for userspace applications). Data Structures When it manages the vmalloc area in virtual memory, the kernel must keep track of which sections are in use and which are free. To this end, it defines a data structure to hold all used sections in a linked list. The kernel uses an important data structure called vm_area_struct to manage the virtual address space contents of a userspace process. Despite the similarity of name and purpose, these two structures must not be confused. struct vm_struct { struct vm_struct *next; void *addr; unsigned long size; unsigned long flags; struct page **pages; unsigned int nr_pages; unsigned long phys_addr; }; 245 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 246 Chapter 3: Memory Management There is an instance of the structure in kernel memory for each area allocated with vmalloc.Themeanings of the structure elements are as follows: ❑ addr defines the start address of the allocated area in virtual address space; size indicates the size of the area. A complete allocation plan of the vmalloc area can be drawn up on the basis of this information. ❑ flags stores the — almost inevitable — flag set associated with the memory section. It is used only to specify the memory area type and currently accepts one of the three values below. ❑ VM_ALLOC specifies that the area was created by vmalloc. ❑ VM_MAP is set to indicate that an existing collection of pages was mapped into the contigu- ous virtual address space. ❑ VM_IOREMAP indicates that an (almost) random physical memory area was mapped into the vmalloc area; this is an architecture-specific operation. Section 3.5.7 shows how the latter two values are employed. ❑ pages is a pointer to an array of page pointers. Each element represents the page instance of a physical page mapped into virtual address space. nr_pages specifies the number of entries in pages and therefore the number of memory pages involved. ❑ phys_addr is required only if physical memory areas described by a physical address are mapped with ioremap. This information is held in phys_addr. ❑ next enables the kernel to hold all sections in the vmalloc area on a singly linked list. Figure 3-38 shows an example of how the structure is used. Three physical pages whose (fictitious) posi- tions in RAM are 1,023, 725 and 7,311 are mapped one after the other. In the virtual vmalloc area, the kernel sees them as a contiguous memory area starting at the VMALLOC_START + 100. addr=VMALLOC_START+100 size=3*PAGE_SIZE nr_pages=3 pages= mem_map PAGE_OFFSET vmalloc-Area: 725 725 1023 1023 7311 7311 Figure 3-38: Mapping physical pages into the vmalloc area. 246 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 247 Chapter 3: Memory Management Creating a vm_area Before a new virtual memory area can be created, it is necessary to find a suitable location for it. A linked list with instances of vm_area manages the sections already created in the vmalloc area. The global variable vmlist defined in mm/vmalloc is the list head. mm/vmalloc.c struct vm_struct *vmlist; The kernel provides the get_vm_area helper function in mm/vmalloc; it acts as a parameter-preparation front end for __get_vm_area. In turn, the latter function is a frontend for __get_vm_area_node that does the actual work. On the basis of the size information for the area, the function tries to find a suitable place in the virtual vmalloc space. As a safety gap of 1 page (guard page) is inserted between each vmalloc area, the kernel first increases the size specification by the appropriate amount. mm/vmalloc.c struct vm_struct *__get_vm_area_node(unsigned long size, unsigned long flags, unsigned long start, unsigned long end, int node) { struct vm_struct **p, *tmp, *area; ... size = PAGE_ALIGN(size); .... /* * We always allocate a guard page. */ size += PAGE_SIZE; ... The start and end parameters are set to VMALLOC_START and VMALLOC_END, respectively, by the calling functions. A loop then iterates over all elements of the vmlist list to find a suitable entry. mm/vmalloc.c for (p = &vmlist; (tmp = *p) != NULL ;p = &tmp->next) { if ((unsigned long)tmp->addr < addr) { if((unsigned long)tmp->addr + tmp->size >= addr) addr = ALIGN(tmp->size + (unsigned long)tmp->addr, align); continue; } if ((size + addr) < addr) goto out; if (size + addr <= (unsigned long)tmp->addr) goto found; addr = ALIGN(tmp->size + (unsigned long)tmp->addr, align); 247 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 248 Chapter 3: Memory Management if (addr > end - size) goto out; } ... The kernel finds a suitable position if size+addr is less than the start address of the area just examined (held in tmp->addr). The new list element is then initialized with the appropriate values and is added to the vmlist linked list. mm/vmalloc.c found: area->next = *p; *p = area; area->flags = flags; area->addr = (void *)addr; area->size = size; area->pages = NULL; area->nr_pages = 0; area->phys_addr = 0; return area; ... } A null pointer is returned to indicate failure if no suitable memory area is found. The remove_vm_area function removes an existing area from the vmalloc address space. struct vm_struct *remove_vm_area(void *addr); The function expects as a parameter the virtual start address of the area to be removed. To find the area, the kernel must successively scan the list elements of vmlist until it finds a match. The corresponding vm_area instance can then be removed from the list. Allocating a Memory Area Allocation of a non-continuous memory area is initiated by vmalloc. This is simply a front-end function to supply __vmalloc with suitable parameters and to directly invoke __vmalloc_node. The associated code flow diagram is shown in Figure 3-39. Implementation is divided into three parts. First, get_vm_area finds a suitable area in the vmalloc address space. Then individual pages are allocated from physical memory, and finally, these pages are mapped contiguously into the vmalloc area — and VM allocation is done. The full code need not be reproduced here because it is riddled with boring safety checks.23 What is interesting is the allocation of the physical memory area (ignore the possibility that there may not be enough physical pages available). 23This, however, does not mean that you should avoid safety checks in your own code! 248 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 249 Chapter 3: Memory Management mm/vmalloc.c void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, pgprot_t prot, int node) { ... for (i = 0; i < area->nr_pages; i++) { if (node < 0) area->pages[i] = alloc_page(gfp_mask); else area->pages[i] = alloc_pages_node(node, gfp_mask, 0); } ... if (map_vm_area(area, prot, &pages)) goto fail; return area->addr; ... } Return address Allocate as many pages as necessary Allocate memory for page instances vmalloc _vmalloc_node get_vm_area_node alloc_pages_node _vmalloc_area_node _vmalloc Figure 3-39: Code flow diagram for vmalloc. If an explicit node is specified from which the pages are to be allocated, the kernel invokes alloc_pages_node. Otherwise, pages are taken from the current node using alloc_page. The pages are removed from the buddy system of the relevant node; when this is done, vmalloc sets gfp_mask to GFP_KERNEL | __GFP_HIGHMEM — the kernel instructs memory management to take the pages from ZONE_HIGHMEM if possible. The reasons for this were given above: Pages from the lower-memory areas are more valuable and should therefore not be wasted for vmalloc allocations that could just as well be satisfied with high-memory pages. Memory is taken from the buddy system, and gfp_mask is set to GFP_KERNEL | __GFP_HIGHMEM so that the kernel instructs memory management to take the pages from ZONE_HIGHMEM if possible. We have already seen the reasons. 249 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 250 Chapter 3: Memory Management Memory is not allocated from the buddy system in a single chunk but page-by-page. This is a key aspect of vmalloc. If it were certain that a contiguous allocation could be made, there would be no need to use vmalloc. After all, the whole purpose of the function is to reserve large memory chunks even though they may not be contiguous owing to fragmentation of the available memory. Splitting the allocation into the smallest possible units — in other words, individual pages — ensures that vmalloc will still work even when physical memory is fragmented. The kernel invokes map_vm_area to map the scattered physical pages contiguously into the virtual vmalloc area. This function iterates over the reserved physical pages and allocates the required number of entries in the various page directories and in the page tables. Some architectures require flushing of the CPU caches after the page tables have been modified. The kernel therefore invokes the flush_cache_vmap whose definition is architecture-specific. Depending on CPU type, this includes the required low-level assembler statements to flush the cache, an invoca- tion of flush_cache_all (if there is no function to flush selective virtually mapped areas), or an empty procedure if the CPU is not reliant on cache flushing, as is the case with IA-32. Alternative Mapping Methods Besides vmalloc, there are other ways of creating virtually contiguous mappings. All are based on the __vmalloc function discussed above or make use of a very similar mechanism (not discussed here). ❑ vmalloc_32 works in the same way as vmalloc but ensures that the physical memory used can always be addressed by means of regular 32-bit pointers. This is important if an architecture can address more memory than would normally be possible on the basis of its word length; this is the case, for example, on IA-32 systems with enabled PAE. ❑ vmap uses a page array as its starting point to create a virtually contiguous memory area. In con- trast to vmalloc, the physical memory location is not allocated implicitly but must be passed ready-made to the function. Mappings of this kind can be recognized by the VM_MAP flag in their vm_map instance. ❑ Unlike all mapping methods described above, ioremap is a processor-specific function that must be implemented on all architectures. It enables a chunk taken from the physical address space used by the system buses for I/O operations to be mapped into the address space of the kernel. This function is used predominantly in device drivers to make the address areas used for com- munication with the peripherals available to the rest of the kernel (and, of course, to itself). Freeing Memory Two functions return memory to the kernel — vfree for areas allocated by vmalloc and vmalloc_32, and vunmap for mappings created using vmap or ioremap.Bothleadbackto__vunmap. mm/vmalloc.c void __vunmap(void *addr, int deallocate_pages) addr indicates the start address of the area to be freed, and deallocate_pages specifies whether the physical pages associated with the area are to be returned to the buddy system. vfree sets the parameter 250 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 251 Chapter 3: Memory Management to 1, whereas vunmap sets it to 0 because in this case only the mappings are removed but the associ- ated physical pages are not returned to the buddy system. Figure 3-40 shows the code flow diagram for __vunmap. __vunmap remove_vm_area _ _remove_vm_area _ _free_pages unmap_vm_area deallocate_pages set? Find areas Free kernel data structures Figure 3-40: Code flow diagram for __vunmap. It is not necessary to explicitly state the size of the area to be freed because this can be derived from the information in vmlist. The first task of __vunmap is therefore to scan this list in __remove_vm_area (invoked by remove_vm_area after completion of locking) in order to find the associated entry. The vm_area instance found is used by unmap_vm_area to remove the entries no longer needed from the page tables. In the same way as when memory is reserved, the function works its way through the various hierarchy levels of page management, but this time removes the entries involved. It also updates the CPU caches. If the __vunmap function parameter deallocate_pages is set to a true value (in vfree), the kernel iterates over all elements of area->pages in which there are pointers to the page instances of the physical pages involved. __free_page is invoked for each entry to return the page to the buddy system. Finally, the kernel data structures used to manage the memory area must be returned. 3.5.8 Kernel Mappings Although the vmalloc family of functions can be used to map pages from the highmem area into the kernel (these are then not usually directly visible in kernel space), this is not the actual purpose of these functions. It is important to underline this fact because the kernel provides other functions for the explicit mapping of ZONE_HIGHMEM pages into kernel space, and these are unrelated to the vmalloc mechanism; this is, therefore, a common source of confusion. Persistent Kernel Mappings The kmap function must be used if highmem pages are to be mapped into kernel address space for a longer period (as a persistent mapping).Thepagetobemappedisspecifiedbymeansofapointertopage as the function parameter. The function creates a mapping when this is necessary (i.e., if the page really is a highmem page) and returns the address of the data. 251 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 252 Chapter 3: Memory Management This task is simple if highmem support is not enabled. In this case, all pages can be accessed directly so it is only necessary to return the page address; there is no need to create a mapping explicitly. The situation is more complicated if highmem pages are actually present. As with vmalloc,the kernel must first establish an association between the highmem pages and the addresses at which they are mapped. An area in virtual address space must also be reserved to map the pages, and finally, the kernel must keep track of which parts of the virtual area are already in use and which are still free. Data Structures As discussed in Section 3.4.2, the IA-32 kernel reserves a region that follows on from the vmalloc area and extends from PKMAP_BASE to FIXADDR_START. This area is used for persistent mappings. The schemes used by different architectures are similar. pkmap_count (defined in mm/highmem.m) is an integer array with LAST_PKMAP positions that contain an entry for each page that can be persistently mapped. It is, in fact, a usage counter for the mapped pages with slightly unusual semantics. The number of users in the kernel is not counted, but the number of users plus 1. If the value of the counter is 2, the mapped page is used at just one point in the kernel. The counter value 5 indicates that there are four users. Expressed more generally, the counter value n stands for n − 1 users in the kernel. As with classic usage counters, 0 means that the associated page is not in use. Counter value 1 has a special meaning. The page associated with the position has already been mapped but cannot be used because the TLB of the CPU has not been updated and access would either fail or be directed to an incorrect address. The kernel makes use of the following data structure to create the association between the page instances of the physical pages and their position in the virtual memory area: mm/highmem.c struct page_address_map { struct page *page; void *virtual; struct list_head list; }; This structure is used to create the page−→ virtual mapping (hence the name of the structure). page holds a pointer to the page instance in the global mem_map array, and virtual specifies the allocated position in the kernel virtual address space. For ease of organization, the mappings are kept in a hash table where the list element is used to set up an overflow list to handle hash collisions. The hash table is implemented by means of the page_address_htable array, not discussed further here. Thehashfunctionispage_slot from mm/highmen.c, which determines the page address on the basis of the page instance. page_address is the front-end function to determine the address of a given page instance using the data structures just described: 252 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 253 Chapter 3: Memory Management mm/highmem.c void *page_address(struct page *page) Figure 3-41 outlines the interplay between the above data structures. struct page_address_map page_address_map->virtual page_address_map->page LAST_ PKMAP pkmap_count page_address_htable mem_map PKMAP_ BASE LAST_ PKMAP FIXMAP_ START Virtual address space 0 0 1 2 12 Figure 3-41: Data structures for managing persistent mappings. Finding Page Addresses page_address first checks whether the passed page instance is in normal memory or high memory. If the former applies, the page address can be calculated from the position of page in the mem_map array. In the latter case, the above hash table is referenced to find the virtual address. Creating a Mapping The kmap function must be used to create a mapping by means of a page pointer.24 It is only a front end to establish whether the desired page really is in highmem. If not, the address yielded by page_address is returned as the result. Otherwise, the kernel delegates work to kmap_high, which is defined as follows: mm/highmem.c void fastcall *kmap_high(struct page *page) { unsigned long vaddr; vaddr = (unsigned long)page_address(page); if (!vaddr) vaddr = map_new_virtual(page); pkmap_count[PKMAP_NR(vaddr)]++; return (void*) vaddr; } 24This function resides not only in arch/x86/mm/highmem_32.c but also in include/asm-ppc/highmem.h and include/asm-sparc/highmem.h with practically the same definition. 253 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 254 Chapter 3: Memory Management The page_address function discussed above first checks whether the page is already mapped. If it does not return a valid address, the page must be mapped using map_new_virtual. The function performs the following main steps: 1. The pkmap_count array is scanned backward from the last used position (held in the global variable last_pkmap_nr) until a free position is found. If no position is free, the function sleeps until another part of the kernel performs an unmapping. When the maximum index of pkmap_count is reached, the search begins at position 0. In this case, the flush_all_zero_pkmaps function is also invoked to flush the caches (you will see this shortly). 2. The page tables of the kernel are modified so that the page is mapped at the desired position. However, the TLB is not updated. 3. The usage counter for the new position is set to 1. As stated above, this means that the page is reserved but cannot be used because the TLB entries are not current. 4. set_page_address adds the page to the data structures of the persistent kernel mappings. The function returns the virtual address of the newly mapped page as its result. On architectures that do not require high-memory pages (or if CONFIG_HIGHMEM is not set), a generic version of kmap is used to return only the page address without changing anything in virtual memory. static inline void *kmap(struct page *page) { might_sleep(); return page_address(page); } Unmapping Pages mapped with kmap must be unmapped using kunmap when they are no longer needed. As usual, this function first checks whether the relevant page (identified by means of its page instance) is actually in high memory. If so, work is delegated to kunmap_high from mm/highmem.c, whose main task is to decrement the counter at the corresponding position in the pkmap_count array (I won’t discuss the details). This mechanism can never reduce the counter value to less than 1; this means that the associated page is not freed. This is because of the additional usage counter increment required to ensure correct handling of the CPU cache as discussed above. The flush_all_zero_pkmaps also mentioned above is key to the final freeing of a mapping; it is always invoked when the search for a free position in map_new_virtual starts from the beginning. It is responsi- ble for three actions: 1. flush_cache_kmaps performs a flush on the kernel mappings (on most architectures that require explicit flushing, the complete CPU cache is flushed using flush_cache_all) because the global page tables of the kernel are changed.25 25This is a very costly operation that fortunately is not required on many processor architectures. In this case, it is defined as a null operation as described in Section 3.7. 254 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 255 Chapter 3: Memory Management 2. pkmap_count is scanned in full. Entries with counter value 1 are set to 0, and the associated entry is deleted from the page table, thus finally removing the mapping. 3. Finally, all TLB entries present for the PKMAP area are flushed using the flush_tlb_kernel_ range function. Temporary Kernel Mappings The kmap function just described must not be used in interrupt handlers because it can sleep. If there are no free positions in the pkmap array, it goes to sleep until the situation improves. The kernel therefore provides an alternative mapping function that executes atomically and is logically named kmap_atomic. A major advantage of this function is that it is faster than a normal kmap. However, it must not be used in code that can potentially go to sleep. It is therefore ideal for short code sections that quickly require a temporary page. The definition of kmap_atomic is architecture-specific for IA-32, PPC, and Sparc32, but the three imple- mentations differ only in very minor details. Their prototype is identical. void *kmap_atomic(struct page *page, enum km_type type) page is a pointer to the management structure of the highmem page, and type defines the type of map- ping required.26 enum km_type { KM_BOUNCE_READ, KM_SKB_SUNRPC_DATA, ... KM_PTE0, KM_PTE1, ... KM_SOFTIRQ1, KM_TYPE_NR }; The fixmap mechanism discussed in Section 3.4.2 makes the memory needed to create atomic mappings available in the kernel address space. An area that can be used to map highmem pages is set up between FIX_KMAP_BEGIN and FIX_KMAP_END in the fixed_addresses array. The exact position is calculated on the basis of the CPU currently active and the desired mapping type. idx = type + KM_TYPE_NR*smp_processor_id(); vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); In the fixmap area, there is a ‘‘window‘‘for each processor in the system. It contains just one entry for each mapping type, as demonstrated in Figure 3-42 (KM_TYPE_NR is not a separate type but simply indicates how many entries there are in km_type). This arrangement makes it clear why functions may not block when they use kmap_atomic. If they did, another process could create a mapping of the same type behind their backs and overwrite the existing entries. 26The contents of the structure differ according to architecture, but the differences are so insignificant that they are not worth describing. 255 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 256 Chapter 3: Memory Management CPU 0 CPU 1 CPU n FIX_KMAP_BEGIN FIX_KMAP_END 1 = KM_BOUNCE_READ 2 = KM_SKB_SUNRPC_DATA,... 01234567 01234567n n 01234567 n Figure 3-42: Mapping high-memory pages by means of fixed mappings. Once the appropriate index has been calculated using the formula specified above and the associated fixmap address has been found, all the kernel need do is modify the page tables accordingly and flush the TLBs to put the changes into effect. The kunmap_atomic function unmaps an existing atomic mapping from virtual memory by reference to its type and virtual address simply by deleting the corresponding entry in the page tables. Mapping Functions on Machines without Highmem Many architectures do not support high memory because they don’t need it — 64-bit architectures head this list. However, to permit use of the above functions without having to constantly distinguish between highmem and non-highmem architectures, the kernel defines several macros that implement compatible functions in normal memory (these are also used when highmem support is disabled on highmem- capable machines). #ifdef CONFIG_HIGHMEM ... #else static inline void *kmap(struct page *page) { might_sleep(); return page_address(page); } #define kunmap(page) do { (void) (page); } while (0) #define kmap_atomic(page, idx) page_address(page) #define kunmap_atomic(addr, idx) do { } while (0) #endif 3.6 The Slab Allocator Every C programmer is familiar with malloc and all its related functions in the standard library; they are frequently invoked by most programs to reserve a few bytes of memory. The kernel must also frequently allocate memory but cannot resort to the standard library functions. The buddy system resources described above support the allocation of memory in pages, but this unit is much too big. If space is needed for a string with 10 characters, reserving a full page with 4 KiB or more is not only wasteful but absolutely unacceptable. The obvious solution is to split the memory in a page into smaller units that can then hold large numbers of small objects. 256 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 257 Chapter 3: Memory Management To this end, it is necessary to introduce new management mechanisms that place a greater overhead on the kernel. To minimize the impact of this extra burden on system performance, the implementation of the management layer should be as compact as possible so that there is little noticeable effect on the caches and TLBs of the processor. At the same time, the kernel must ensure that memory is utilized speedily and efficiently. Not only Linux but look-alikes and all other operating systems face this prob- lem. Over the course of time, some good solutions and some bad solutions have been proposed and are described in the general operating system literature (e.g., [Tan07]). One such proposal — slab allocation — has proved to be very efficient for many workloads. It was devised and implemented for Solaris 2.4 by Jeff Bonwick, a Sun employee. Because he publicly documented his method [Bon94], it was also possible to implement a version for Linux. The provision of smaller memory blocks is not the only task of the slab allocator. Owing to its structure, it also serves as a cache for objects that are frequently allocated and then released. By setting up a slab cache, the kernel is able to keep a store of objects at the ready for subsequent use, even in an initialized state, if so desired. For instance, the kernel must frequently generate new instances of struct fs_struct to manage the filesystem data associated with a process (see Chapter 8). The memory blocks occupied by instances of this type are reclaimed just as often (when a process terminates). In other words, the kernel tends to allocate and release sizeof{fs_struct} memory blocks with great regularity. The slab allocator keeps the returned memory blocks on an internal list and does not immediately give them back to the buddy system. A recently returned block is then used when a new request is received for a fresh instance of the object. This has two advantages. First, handling time is shorter because the kernel need not apply the buddy system algorithms. Second, because the memory blocks are still ‘‘fresh,’’ there is a strong probability that they are still in one of the CPU caches. The slab allocator also has two further benefits: ❑ Calls to the buddy system are operations that have a considerable impact on the data and instruction caches of the system. The more the kernel wastes these resources, the less they are available for userspace processes. The more lightweight slab allocator dispenses with the need for calls to the buddy system wherever possible and helps prevent undesirable cache ‘‘contamination.’’ ❑ Data stored in pages delivered directly by the buddy system is always clustered around addresses divisible by powers of 2 (many other allocation methods that divide pages into smaller blocks share this characteristic). This has a negative impact on CPU cache utilization because some cache lines are overused owing to this kind of address distribution and others are almost empty. This disadvantage can be even more drastic on multiprocessor systems if different memory addresses are transferred on different buses because some buses may be congested, while others are little used. By means of slab coloring, the slab allocator is able to distribute objects uniformly to achieve uni- form cache utilization, as demonstrated below. That frequently used kernel objects are kept in the CPU cache is a desired effect. The earlier comment that the large cache and TLB footprints of the buddy system are negative in terms of the slab allocator related to the fact that unimportant data land in the CPU cache and important data are displaced — a situation that should naturally be prevented. 257 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 258 Chapter 3: Memory Management The term color is used in the metaphorical sense. It has nothing to do with colors but represents a certain offset by which the objects in the slab are shifted to place them in a different cache line. Where does the name slab allocator come from? The objects managed in each cache are combined into larger groups covering one or more contiguous page frames. Such groups are called slabs; each cache consists of several such slabs. 3.6.1 Alternative Allocators Although the slab allocator works well for many possible workloads, there are naturally situations in which it fails to provide optimal performance. Problems arise when slab allocation is used on machines that range on the borders of the current hardware scale: tiny embedded systems and large, massively parallel systems equipped with huge amounts of RAM. In the second case, the large amount of metadata required by the slab allocator can become a problem: developers have reported that many gigabytes of memory are required only for the slab data structures on large systems. For embedded systems, the total footprint and complexity of slab allocation can simply be too much. To cope with such situations, two drop-in replacements for the slab allocator were added during the development of kernel 2.6: ❑ The slob allocator is especially optimized for low code size. It is centered around a simple linked lists ofblocks (thus its name). To allocate memory, a likewise simple first-fit algorithm is used. With only roughly 600 lines, the total footprint of the slob allocator is very small. Naturally, it is not the most efficient allocator in terms of speed and is definitely not designed to be used on large-scale systems. ❑ The slub allocator tries to minimize the required memory overhead by packing page frames into groups and to manage these groups by overloading unused fields in struct page. While this certainly does not simplify the definition of this structure, as you have seen before, the effort is justified by the better performance of slub in contrast to slab on large machines. Since slab allocation is the default option used by most kernel configurations, alternative allocators are not discussed in detail. It is, however, important to emphasize that the rest of the kernel need not be concerned about which low-level allocator is chosen. The visible front end is identical for all allocators. Each must implement a certain set of functions for memory allocation and caching: ❑ kmalloc, __kmalloc,andkmalloc_node as general (node-specific) allocation functions. ❑ kmem_cache_alloc, kmem_cache_alloc_node as (node-specific) providers of specific kernel caches. The behavior of these functions is included in the following discussion of the slab allocator. Using these standard functions, the kernel can provide further convenience functions that do not require specific knowledge about how memory is managed internally — for instance, kcalloc to allocate memory for arrays, or kzalloc to allocate a memory region that is filled with zero bytes. The situation is illustrated in Figure 3-43. Regular kernel code just needs to include slab.h to enjoy all standard kernel functions for memory allocation. The build system will ensure that the allocator chosen at compile time is used to fulfill the desired requests. 258 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 259 Chapter 3: Memory Management Physical page frames Generic kernel code Buddy system Standard interfaceSlab, slob, or slub allocator Convenience functions Figure 3-43: Connection between the buddy system, general-purpose allocators, and the interface to generic kernel code. 3.6.2 Memory Management in the Kernel The general allocation and freeing functions of the kernel have similar names to their equivalents in the C standard library and are employed in exactly the same way. ❑ kmalloc(size, flags) reserves a memory area of size bytes and returns a void pointer to the start of the area. If insufficient memory is available (a very improbable situation in the kernel but one that must always be catered for), a null pointer is the result. The flags argument specifies the area from which memory is to be selected using the GFP_ con- stants discussed in Section 3.5.4, for example, GFP_DMA for a DMA-suitable memory area. ❑ kfree{*ptr} frees the memory area pointed at by *ptr. In contrast to the situation in userspace programming, the kernel also includes the percpu_alloc and percpu_free functions to reserve and free the desired memory area for each system CPU (and not specif- ically for the CPU currently active).27 kmalloc is used at thousands of places in the kernel sources, but the pattern is always the same. The memory area reserved with kmalloc is converted to the correct type by means of a typecast and is then assigned to a pointer variable. info = (struct cdrom_info *) kmalloc (sizeof (struct cdrom_info), GFP_KERNEL); The task of setting up and using caches is not especially difficult from the programmer’s point of view. A suitable cache must first be created with kmem_cache_create, then the objects it contains can be allocated 27Older kernel versions have used the functions alloc_percpu and free_percpu for this purpose, but since these functions do not support CPU hotplugging, they are only supported for compatibility reasons and should not be used in new code. 259 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 260 Chapter 3: Memory Management and freed using kmem_cache_alloc and kmem_cache_free. The slab allocator is automatically responsible for interaction with the buddy system to reserve the required pages. A list of all active caches is held in /proc/slabinfo (the output below omits a few less important columns for reasons of space).28 wolfgang@meitner> cat /proc/slabinfo slabinfo - version: 2.1 # name : tunables : slabdata nf_conntrack_expect 0 0 224 18 1 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 16 16 960 4 1 : tunables 0 0 0 : slabdata 4 4 0 TCPv6 19 20 1792 4 2 : tunables 0 0 0 : slabdata 5 5 0 xfs_inode 25721 25725 576 7 1 : tunables 0 0 0 : slabdata 3675 3675 0 xfs_efi_item 44 44 352 11 1 : tunables 0 0 0 : slabdata 4 4 0 xfs_efd_item 44 44 360 11 1 : tunables 0 0 0 : slabdata 4 4 0 ... kmalloc-128 795 992 128 32 1 : tunables 0 0 0 : slabdata 31 31 0 kmalloc-64 19469 19584 64 64 1 : tunables 0 0 0 : slabdata 306 306 0 kmalloc-32 2942 2944 32 128 1 : tunables 0 0 0 : slabdata 23 23 0 kmalloc-16 2869 3072 16 256 1 : tunables 0 0 0 : slabdata 12 12 0 kmalloc-8 4075 4096 8 512 1 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-192 2940 3276 192 21 1 : tunables 0 0 0 : slabdata 156 156 0 kmalloc-96 754 798 96 42 1 : tunables 0 0 0 : slabdata 19 19 0 The file columns contain the following information in addition to a string that identifies each cache (and also ensures that no identical caches are created): ❑ Number of active objects in the cache. ❑ Total number of objects in the cache (used and unused). ❑ Size of the managed objects in bytes. ❑ Number of objects in a slab. ❑ Pages per slab. ❑ Number of active slabs. ❑ Object number allocated when the kernel decides to make more memory available to a cache. (A larger memory block is allocated in one chunk so that the required interaction with the buddy system is worthwhile.) This value is also used as the block size when shrinking the cache. 28Additional information on slab allocator statistics is output if the CONFIG_DEBUG_SLAB option is set at compilation time. 260 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 261 Chapter 3: Memory Management In addition to easily identified cache names such as unix_sock (for domain sockets, i.e., objects of type struct unix_sock), there are other fields called kmalloc-size. (Machines that provide DMA memory also include caches for DMA allocations, but these are not present in the above example.) These are the basis of the kmalloc function in which the kernel provides slab caches for various memory sizes that, with few exceptions, are in power-of-2 steps between 25 = 32 (for machines with 4 KiB page size), respective 64 (for all other machines), and 225 bytes. The upper bound can also be considerably smaller and is set by KMALLOC_MAX_SIZE, which, in turn, is computed based on the page size of the system and the maximally allowed allocation order: #define KMALLOC_SHIFT_HIGH ((MAX_ORDER + PAGE_SHIFT - 1) <= 25 ? \ (MAX_ORDER + PAGE_SHIFT - 1) : 25) #define KMALLOC_MAX_SIZE (1UL << KMALLOC_SHIFT_HIGH) #define KMALLOC_MAX_ORDER (KMALLOC_SHIFT_HIGH - PAGE_SHIFT) Each time kmalloc is invoked, the kernel finds the most suitable cache and allocates one of its objects to satisfy the request for memory as best it can (if no cache fits exactly, larger objects are always allocated but never smaller objects). The difference between the slab allocator and cache outlined above quickly disappears in the concrete implementation, so much so that both terms are used synonymously in the further course of the book. Section 3.6.5 looks at the details of kmalloc after discussing the implementation of the slab allocator. 3.6.3 The Principle of Slab Allocation The slab allocator is made up of a closely interwoven network of data and memory structures that is not easy to untangle at first sight. It is therefore important to obtain an overview of the relationships between the structures before moving on to examine the implementation. Basically, the slab cache consists of the two components shown in Figure 3-44: a cache object to hold the management data and slabs to hold the managed objects. Each cache is responsible for just one object type, instances of struct unix_sock, for example, or general buffers. The number of slabs in each cache varies according to the number of pages used, the object size, and the number of objects managed. Section 3.6.4 goes into the details of how cache sizes are calculated. All caches in the system are also kept in a doubly linked list. This gives the kernel the opportunity to traverse all caches one after the other; this is necessary, for example, when shrinking cache memory because of an impending memory shortage. Fine Structure of the Cache If we look more closely at the cache structure, we note further details of importance. Figure 3-45 provides an overview of the cache components. 261 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 262 Chapter 3: Memory Management Cache Cache Cache Slab Slab Slab Cache Object Figure 3-44: Components of the slab allocator. struct kmem_cache free array_ cache array_ cache Arrays with pointers to Slab objects Slab Head Slab Head Slab Head Page Frames on- or off-Slab full partial Figure 3-45: Fine structure of a slab cache. Besides management data (such as the number of used and free objects or flag registers), the cache struc- ture includes two elements of special significance: ❑ A pointer to an array in which the last freed objects can be kept for each specific CPU. ❑ Three list heads per memory node under which slabs can be listed. The first list contains full slabs, the second partially free slabs, and the third free slabs. The cache structure points to an array that contains as many entries as there are CPUs in the system. Each element is a pointer to a further structure known as an array cache, which contains the management data for the particular system CPU (and not for the cache as a whole). The memory area immediately following the management data contains an array with pointers to as-yet-unused objects in the slabs. The per-CPU pointers are important to best exploit the CPU caches. The LIFO principle (last in, first out) is applied when objects are allocated and returned. The kernel assumes that an object just returned is still 262 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 263 Chapter 3: Memory Management in the cache and allocates it again as quickly as possible (in response to the next request). Only when the per-CPU caches are empty are free objects from the slabs used to refill them. This results in a three-level hierarchy for object allocation within which both the allocation cost and the negative impact of the operation on caches and TLBs rise from level to level: 1. Per-CPU objects in the CPU cache. 2. Unused objects from an existing slab. 3. Unused objects from a new slab just reserved using the buddy system. Fine Structure of Slabs Objects are not listed continuously in slabs but are distributed according to a rather complicated scheme. Figure 3-46 illustrates the details. Manage- ment head ... BUFCTL_END Management of free objects Color Space Unused Memory ...Free Used Free Fill bytes for alignment Object Size Figure 3-46: Fine structure of a slab. The size used for each object does not reflect its exact size. Instead, the size is rounded to fulfill certain alignment criteria. Two alternatives are possible: ❑ Using the flag SLAB_HWCACHE_ALIGN at slab creation time, the slab user can request that objects are aligned to hardware cache lines. The alignment is then performed along the value returned by cache_line_size, which returns the processor-specific size of the L1 cache. If objects are smaller than half of the cache line size, then more than one object is fit into one cache line. ❑ If alignment along hardware cache lines is not requested, then the kernel ensures that objects are aligned with BYTES_PER_WORD — the number of bytes needed to represent a void pointer. On 32-bit processors, 4 bytes are required for a void pointer. Consequently, for an object with 6 bytes, 8 = 2 × 4 bytes are needed, and objects with 15 bytes require 16 = 4 × 4 bytes. The superfluous bytes are referred to as fill bytes. 263 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 264 Chapter 3: Memory Management Fill bytes speed access to the objects in a slab. Memory access is faster on almost all architectures if aligned addresses are used. This compensates for the disadvantage of higher memory requirements entailed by the use of fill bytes. The management structure holding all the management data (and the list element to link with the cache lists) is located at the start of each slab. It is immediately followed by an array that includes an (integer) entry for each object in the slab. The entries are only of significance if the associated object is not allocated. In this case, it specifies the index of the next free object. Because the number of the free object with the lowest number is also stored in the management structure at the start of the slab, the kernel is easily able to find all objects currently available without having to use linked lists or other complicated associations.29 The last array entry is always an end marker with the value BUFCTL_END. Figure 3-47 illustrates the situation graphically. Slab objectsManagement array Used objectUnused object 24 6 Figure 3-47: Management of the free objects in a slab. In most cases, the size of the slab area (minus the management head) is not divisible by the (possibly padded) object size without a remainder. Consequently, a little superfluous memory is available to the kernel and is used to give the slab a ‘‘color‘‘ in the form of an offset as described above. The slab members of a cache are given different offsets to position the data in different cache lines with the result that the free memory at the start and end of a slab varies. When the offset is calculated, the ker- nel must take other alignment factors into account, for instance, alignment of the data on the L1 cache (discussed below). The management data can be positioned either on the slab itself or in an external memory area allocated using kmalloc.30 Which alternative the kernel selects depends on the size of the slab and of the objects used. The corresponding selection criteria are discussed shortly. The association between the manage- ment data and slab memory is easy to establish because the slab header contains a pointer to the start of the slab data area (regardless of whether it is on-slab or off-slab). Figure 3-48 shows the situation when the data are not on the slab itself (as it is in Figure 3-46) but in external memory. And finally, the kernel needs a way of identifying the slab (and therefore the cache in which an object resides) by reference to the object itself. On the basis of an object’s physical memory address, it is not difficult to find the associated page and therefore the matching page instance in the global mem_map array. As we already know, the page structure includes a list element used to manage the page in 29The original implementation of the slab allocator in the SunOS operating system kernel uses a linked list to keep track of the free objects. 30This requires special precautions when the kmalloc caches are initialized because obviously kmalloc cannot be invoked there yet. This and other chicken-and-egg problems of slab initialization are discussed below. 264 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 265 Chapter 3: Memory Management various lists. As this is not necessary for pages on the slab cache, the pointers can be used for other purposes: ❑ page->list.next points to the management structure of the cache in which the page resides. ❑ page->list.prev points to the management structure of the slab on which the page is held. Management area Figure 3-48: Slab with external (off-slab)slabheader. Setting or reading this information is concealed behind the set_page_slab and get_page_slab,respec- tively, _cache functions to lower the hack value of this convention. mm/slab.c void page_set_cache(struct page *page, struct kmem_cache *cache) struct kmem_cache *page_get_cache(struct page *page) void page_set_slab(struct page *page, struct slab *slab) struct slab *page_get_slab(struct page *page) Additionally, the kernel sets the page flag PG_SLAB for each physical page, that is allocated for the slab allocator. 3.6.4 Implementation Various data structures are used to implement the slab allocator as described above. Although this does not appear to be difficult, the code is not always easy to read or understand. This is because many mem- ory areas need to be manipulated using pointer arithmetic and type-casting — not necessarily one of the areas of C famed for its clarity. The code is also pervaded with pre-processor statements because the slab system features numerous debugging options. 31 Some of these are listed below: ❑ Red Zoning — An additional memory area filled with a known byte pattern is placed at the start and end of each object. If this pattern is overwritten, programmers will note when analyzing kernel memory that their code accesses memory areas that don’t belong to them. ❑ Object Poisoning — Objects are filled with a predefined pattern when a slab is created and freed. If it is noted at object allocation that this pattern is changed, programmers know that unauthorized access has already taken place. For the sake of simplicity and to focus attention on the big picture rather than minor details, let’s restrict our description below to a ‘‘pure‘‘ slab allocator that doesn’t make use of the above options. 31The CONFIG_DEBUG_SLAB configuration option must be set at compilation time to enable debugging. However, this significantly slows allocator performance. 265 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 266 Chapter 3: Memory Management Data Structures Each cache is represented by an instance of the kmem_cache structure defined in mm/slab.c. The structure is not normally visible at other points of the kernel because it is defined in a C header and not in a header file. This is because users of the cache need not know in detail how the cache is implemented. It is sufficient to regard slab caches as mechanisms for the efficient creation and release of objects of a specific type by means of a set of standard functions. The contents of the structure are as follows: mm/slab.c struct kmem_cache { /* 1) per-cpu data, touched during every alloc/free */ struct array_cache *array[NR_CPUS]; /* 2) Cache tunables. Protected by cache_chain_mutex */ unsigned int batchcount; unsigned int limit; unsigned int shared; unsigned int buffer_size; u32 reciprocal_buffer_size; /* 3) touched by every alloc & free from the backend */ unsigned int flags; /* constant flags */ unsigned int num; /* # of objs per slab */ /* 4) cache_grow/shrink */ /* order of pgs per slab (2^n) */ unsigned int gfporder; /* force GFP flags, e.g. GFP_DMA */ gfp_t gfpflags; size_t colour; /* cache colouring range */ unsigned int colour_off; /* colour offset */ struct kmem_cache *slabp_cache; unsigned int slab_size; unsigned int dflags; /* dynamic flags */ /* constructor func */ void (*ctor)(struct kmem_cache *, void *); /* 5) cache creation/removal */ const char *name; struct list_head next; /* 6) statistics */ ... struct kmem_list3 *nodelists[MAX_NUMNODES]; }; 266 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 267 Chapter 3: Memory Management This lengthy structure is divided into multiple parts as indicated by the comments in the kernel sources.32 The initial elements are concerned with CPU-specific data accessed by the kernel during each allocation, touched upon in Section 3-46. ❑ array is a pointer to an array with an entry for each CPU in the system. Each entry contains a further pointer to an instance of the array_cache structure discussed below. ❑ batchcount specifies the number of objects to be taken from the slabs of a cache and added to the per-CPU list if it is empty. It also indicates the number of objects to be allocated when a cache is grown. ❑ limit specifies the maximum number of objects that may be held in a per-CPU list. If this value is exceeded, the kernel returns the number of objects defined in batchcount to the slabs (if the kernel then shrinks the caches, memory is returned from the slabs to the buddy system). ❑ buffer_size specifies the size of the objects managed in the cache.33 ❑ Suppose that the kernel has a pointer to an element in a slab and wants to determine the corre- sponding object index. The easiest way to do this is to divide the offset of the pointer compared to the start of the slab area by the object size. Consider, for example, that a slab area starts at memory location 100, each object requires 5 bytes, and the object in question is located at mem- ory position 115. The offset between the slab start and the object is 115 − 100 = 15, so the object index is 15/5 = 3. Unfortunately, divisions are slow on some older machines. Since multiplications are much faster on these machines, the kernel uses the so-called Newton-Raphson technique, which requires only multiplications and bit shifts. While the mathematical details are not interesting for our purposes (they can be found in any standard textbook), we need to know that instead of computing C = A/B, the kernel can also employ C = reciprocal_divide(A, reciprocal_value(B)) — both functions are provided as library routines. Since the object size in a slab is constant, the kernel can store the recpirocal value of buffer_size in recpirocal_buffer_size, which can be used later when the division must be computed. The kernel provides an instance of array_cache for each system processor. This structure is defined as follows: mm/slab.c struct array_cache { unsigned int avail; unsigned int limit; unsigned int batchcount; unsigned int touched; spinlock_t lock; void *entry[]; }; 32If slab debugging is enabled, another part with statistical information gathered by the kernel concludes the structure. 33If slab debugging is enabled, the buffer size can differ from the object size because extra padding (in addition to the padding used to align the objects properly) is introduced per element. In this case, a second variable denotes the real size of the object. 267 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 268 Chapter 3: Memory Management The meanings of batchcount and limit are as given above. The values of kmem_cache_s are applied (normally unchanged) as defaults for the per-CPU values used for cache refill or emptying. avail holds the number of elements currently available. touched is set to 1 when an element is removed from the cache, whereas cache shrinking causes touched to be set to 0. This enables the kernel to establish whether a cache has been accessed since it was last shrunk and is an indicator of the importance of the cache. The last element is a dummy array without an entry to facilitate access to the cache elements following each array_cache instance in memory. The third and fourth parts of kmem_cache contain all the variables needed to manage the slabs and are required when the per-CPU caches are filled or emptied. ❑ nodelists is an array that contains an entry for each possible node in the system. Each entry holds an instance of struct kmem_list3 that groups the three slab lists (full, free, partially free) together in a separate structure discussed below. The element must be placed at the end of the structure. While it formally always has MAX_NUMNODES entries, it is possible that fewer nodes are usable on NUMA machines. The array thus requires fewer entries, and the kernel can achieve this at run time by simply allocating less memory than the array formally requires. This would not be possible if nodelists were placed in the middle of the structure. On UMA machines, this is not much of a concern because only a single node will ever be available. ❑ flags is a flag register to define the global properties of the cache. Currently, there is only one flag bit. CFLGS_OFF_SLAB is set when the management structure is stored outside the slab. ❑ objsize is the size of the objects in the cache, including all fill bytes added for alignment purposes. ❑ num holds the maximum number of objects that fit into a slab. ❑ free_limit specifies the upper limit of free objects in a cache after it has been shrunk (if there is no reason to shrink the cache during normal operation, the number of free objects may exceed this value). The list heads to manage the slab lists are kept in a separate data structure defined as follows: mm/slab.c struct kmem_list3 { struct list_head slabs_partial; /* partial list first, better asm code */ struct list_head slabs_full; struct list_head slabs_free; unsigned long free_objects; unsigned int free_limit; unsigned int colour_next; /* Per-node cache coloring */ spinlock_t list_lock; struct array_cache *shared; /* shared per node */ struct array_cache **alien; /* on other nodes */ unsigned long next_reap; /* updated without locking */ int free_touched; /* updated without locking */ }; 268 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 269 Chapter 3: Memory Management The meanings of the first three list heads are clear from the explanations in the above sections. free_objects indicates the total number of free objects in all slabs of slabs_partial and slabs_free. free_touched indicates whether the cache is active or not. When an object is taken from the cache, the kernel sets the value of this variable to 1; when the cache is shrunk, the value is reset to 0. However, the kernel only shrinks a cache if free_touched has been set to 0 beforehand, because the value 1 indicates that another part of the kernel has just taken objects from the cache and thus it is not advisable to shrink it. This variable applies for the whole cache unlike the per-CPU touched element. next_reap defines a time interval that the kernel must allow to elapse between two attempts to shrink the cache. The idea is to prevent degradation of system performance due to frequent cache shrinking and growing operations as can happen in certain load situations. This technique is only used on NUMA systems and will thus not concern us any further. free_limit specifies the maximum number of unused objects permitted on all slabs. The structure is concluded by pointers to array_cache instances that are either shared per node or origi- nate from other nodes. This is of relevance on NUMA machines but, for the sake of simplicity, this won’t be discussed in detail. The third part of kmem_cache contains all variables needed to grow (and shrink) the cache. ❑ gfporder specifies the slab size as a binary logarithm of the number of pages, or, expressed dif- ferently, the slab comprises 2gfporder pages. ❑ The three colour elements hold all relevant data for slab coloring. colour specifies the maximum number of colors and colour_next the color to use for the next slab created by the kernel. Note, however, that this value is specified as an element of kmem_list3. colour_off is the basic offset multiplied by a color value to obtain the absolute offset. This is again required for NUMA machines — UMA systems could keep colour_next in struct kmem_cache. Placing the next color in a node-specific structure, however, allows coloring slabs added on the same node sequentially, which is beneficial for the local caches. Example: If there are five possible colors (0, 1, 2, 3, 4) and the offset unit is 8 bytes, the kernel can use the following offset values: 0 × 8 = 0, 1 × 8 = 8, 2 × 8 = 16, 3 × 8 = 24 and 4 × 8 = 32 bytes. Section 3.6.4 examines how the kernel determines the possible settings for slab colors. Besides, note that the kernel sources, in contrast to this book, spell colour properly, at least from the British point of view. ❑ If the slab head is stored outside the slab, slabp_cache points to the general cache from which the required memory is taken. If the slab head is on-slab, slabp_cache contains a null pointer. ❑ dflags is a further set of flags that describe the ‘‘dynamic properties‘‘ of the slab, but currently no flags are defined. ❑ ctor is a pointer to a constructor function that is invoked when objects are created. This method is well known in object-oriented languages such as C++ and Java. Former kernel versions did offer the ability to specify an additional destructor function, but since this opportunity was not used, it has been dropped during the development of kernel 2.6.22. 269 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 270 Chapter 3: Memory Management The fifth and last part (statistics fields that are of no further interest for our purposes) of struct kmem_cache consists of two further elements: ❑ name is a string containing a human-readable name for the cache. It is used, for example, to list the available caches in /proc/slabinfo. ❑ next is a standard list element to keep all instances of kmem_cache on the global list cache_chain. Initialization At first sight, initialization of the slab system does not appear to be especially complicated because the buddy system is already fully enabled and no other particular restrictions are imposed on the kernel. Nevertheless, there is a chicken-and-egg problem34 because of the structure of the slab allocator. To initialize the slab data structures, the kernel needs memory blocks that are much smaller than a com- plete page and are therefore best allocated by kmalloc. And here’s the crux: kmalloc only functions if the slab system is already enabled. To be more accurate, the problem lies with the initialization of the per-CPU caches for kmalloc.Before these caches can be initialized, kmalloc must be available to reserve the required memory space, and kmalloc itself is just in the process of being initialized. In other words, kmalloc can only be initialized once kmalloc has been initialized — an impossible scenario. The kernel must therefore resort to a few tricks. The kmem_cache_init function is used to initialize the slab allocator. It is invoked during the kernel initialization phase (start_kernel) once the buddy system is enabled. However, on multiprocessor systems, the boot CPU is running and the other CPUs are not yet initialized. kmem_cache_init employs a multistep process to activate the slab allocator step-by-step: 1. kmem_cache_init creates the first slab cache in the system to generate memory for instances of kmem_cache. To this end, the kernel uses mainly static data created at compilation time; in fact, a static data structure (initarray_cache) is used as a per-CPU array. The name of this cache is cache_cache. 2. kmem_cache_init then initializes the general caches that serve as a source for kmalloc.For this purpose, kmem_cache_create is invoked for each cache size required. The function first needs only the cache_cache cache already created; however, when the per-CPU caches are to be initialized, the function must resort to kmalloc, and this is not yet possible. To resolve this problem, the kernel uses the g_cpucache_up variable, which can accept one of four values (NONE, PARTIAL_AC, PARTIAL_L3,orFULL) to reflect the state of kmalloc initial- ization. Initially the state of the kernel is NONE. When the smallest kmalloc cache (which provides memory blocks of 32 bytes on machines with 4 KiB memory pages; if other page sizes are used, the smallest allocation size is 64 bytes; the exact definition of existing sizes is given in Section 3.6.5) is initialized, a static variable is again used for the per-CPU cache data. 34Chicken-and-egg problems are encountered where something cannot happen until a second thing does, and the second thing cannot happen until the first does. For example, B must be present in order to initialize A, but A must be present to initialize B. It’s the age- old question of which came first, the chicken or the egg? If you are a scientist, you can also use the term causality dilemma, which expresses exactly the same, but sounds much more educated ... . 270 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 271 Chapter 3: Memory Management The state in g_cpucache_up is then set to PARTIAL_AC, meaning that array_cache instances can be allocated immediately. If the initialized size is also sufficient to allocate kmem_list3 instances, the state immediately changes to PARTIAL_L3. Otherwise, this only happens when the next larger cache has been initialized. The per-CPU data of the remaining kmalloc caches can now be created with kmalloc as an instance of arraycache_init, as only the smallest kmalloc area is needed for this purpose. mm/slab.c struct arraycache_init { struct array_cache cache; void * entries[BOOT_CPUCACHE_ENTRIES]; }; 3. In the last step of kmem_cache_init, all statically instantiated elements of the data struc- tures used up to present are replaced with dynamically allocated version created using kmalloc. The state of g_cpucache_up is now FULL, indicating that the slab allocator is ready for use. Creating Caches kmem_cache_create must be invoked to create a new slab cache. This function requires a large set of parameters. mm/slab.c struct kmem_cache * kmem_cache_create (const char *name, size_t size, size_t align, unsigned long flags, void (*ctor)(struct kmem_cache *, void *)) Besides a human-readable name that subsequently appears in /proc/slabinfo, the function requires the size of the managed objects in bytes, an offset used when aligning data (align, in almost all cases 0), a set of flags in flags, and constructor/destructor functions in ctor and dtor. Creation of a new cache is a lengthy procedure, as the code flow diagram for kmem_cache_create in Figure 3-49 shows. Several parameter checks are carried out to ensure that no invalid specifications are used (e.g., an object size with fewer bytes than a processor word, a slab without name, etc.) before the first important step is carried out — calculation of the required alignment. First, the object size is rounded up to a multiple of the word length of the processor used: mm/slab.c kmem_cache_t * kmem_cache_create (...) { ... if (size & (BYTES_PER_WORD-1)) { size += (BYTES_PER_WORD-1); size &= ~(BYTES_PER_WORD-1); } Object alignment (in align) is typically also based on the processor word length. However, if the SLAB_HWCACHE_ALIGN flag is set, the kernel aligns the data as recommended by the architecture-specific function cache_line_size. It also attempts to pack as many objects as possible in a cache line by halving 271 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 272 Chapter 3: Memory Management the alignment factor as long as this is possible for the given object size. As a result, 2, 4, ...objects fit into a cache line instead of a single object. mm/slab.c /* 1) arch recommendation: */ if (flags & SLAB_HWCACHE_ALIGN) { /* Default alignment: as specified by the arch code. * Except if an object is really small, then squeeze multiple * objects into one cacheline. */ ralign = cache_line_size(); while (size <= ralign/2) ralign /= 2; } else { ralign = BYTES_PER_WORD; } ... Sanity checks Calculate alignment Allocate cache structure Determine where to store slab head Compute cache size iteratively with cache_estimate Compute colors Insert cache in cache_chain kmem_cache_create calculate_slab_order enable_cpucache do_tune_cpucache Figure 3-49: Code flow diagram for kmem_cache_create. The kernel also takes account of the fact that some architectures require a minimum boundary for the alignment of data as defined by ARCH_SLAB_MINALIGN; the alignment required by users is also accepted. mm/slab.c /* 2) arch mandated alignment */ if (ralign < ARCH_SLAB_MINALIGN) { ralign = ARCH_SLAB_MINALIGN; } /* 3) caller mandated alignment */ if (ralign < align) { 272 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 273 Chapter 3: Memory Management ralign = align; } /* 4) Store it. */ align = ralign; ... A new instance of struct kmem_cache is allocated once the data alignment has been calculated (a separate slab cache named cache_cache is provided to perform allocation). The decision as to whether to store the slab head on-slab or off-slab (see Section 3.6.3) is relatively simple. If the object size is more than one-eighth of a page frame, the head is stored off-slab, otherwise on the slab itself. mm/slab.c if (size >= (PAGE_SIZE>>3)) /* * Size is large, assume best to place the slab management obj * off-slab (should allow better packing of objs). */ flags |= CFLGS_OFF_SLAB; size = ALIGN(size, align); ... The slab header can also be stored off-slab for smaller objects by explicitly setting CFLGS_OFF_SLAB in the kmem_cache_create call. Finally, the object size is increased until it corresponds to the alignment calculated above. Up to now we have only defined the size of the objects but not of the slab. In the next step, an attempt is therefore made to find a suitable number of pages that is neither too small nor too big. Too few objects on a slab increase administrative overhead and render the method less efficient, while overlarge slab areas are detrimental to the buddy system. The kernel tries to find the ideal slab size in an iterative process implemented in calculate_slab_order. Based on the given object size, cache_estimate calculates the number of objects, the wasted space, and the space needed for coloring for a specific number of pages. The function is invoked in a loop until the kernel is satisfied with the results. By systematic trial and error, cache_estimate findsaslabarrangementthatcanbedescribedbythe following elements: size is the object size, gfp_order the order for page allocation, num the number of objects on the slab, and wastage the space that is ‘‘wasted‘‘ with this order and is therefore no longer available for useful data (of course, wastage < size always applies; otherwise, another object could be fitted on the slab). head specifies how much space is required for the slab head. This layout corresponds to the following formula: PAGE_SIZE<num*sizeof(kmem_bufctl_t) + sizeof(struct slab), align); ... ALIGN(x,y) is a standard macro provided by the kernel that computes the required space that is sufficient to store the object x, but is additionally an integer-valued multiple of align. Table 3-9 provides some exemplary alignment calculations. If sufficient free space is available to store the slab head on-slab although it should actually be stored off-slab, the kernel gladly makes use of the opportunity. The CFLGS_OFF_SLAB is deleted, and the head is stored on the slab despite the earlier decision to do the opposite or despite the default setting. The following steps are performed to color the slab: mm/slab.c cachep->colour_off = cache_line_size(); /* Offset must be a multiple of the alignment. */ 274 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 275 Chapter 3: Memory Management if (cachep->colour_off < align) cachep->colour_off = align; cachep->colour = left_over/cachep->colour_off; ... The kernel uses the size of an L1 cache that can be determined using the architecture-specific cache_line_size function as an offset. It must also be ensured that the offset is a multiple of the alignment used — otherwise, the alignment effect would be lost. Table 3-9: Examplary Calculations of the Alignment on 4- and 8-Byte Boundaries Object size x Alignment y ALIGN(x,y) 148 448 588 888 91216 12 12 16 13 16 16 16 16 16 17 20 24 19 20 24 The color of the slab (i.e., the number of potential offset values) is calculated by dividing the free space on the slab (known as the left_over) by the color offset (colour_off)withouta remainder. For example, on an older IA-32 machine, the kernel produces the following results for a cache that manages 256-byte objects aligned on the hardware cache with SLAB_HWCACHE_ ALIGN: ❑ 15 objects are managed on a slab (num = 15). ❑ Onepageisused(gfp_order = 0). ❑ There are five possible colors (colour = 5),andanoffsetof32bytesisusedforeachcolor (colour_off = 32). ❑ Theslabheadisstoredon-slab. 275 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 276 Chapter 3: Memory Management Now that we have dealt with the slab arrangement, there are still two more things to do when creating a new slab cache in kmem_cache_create: ❑ The per-CPU caches must be generated. This task is delegated to enable_cpucache (the layout and structure of these caches are described in Section 3.6.4). First, the kernel defines the number of object pointers in the cache depending on the object size: 0 < size ≤ 256 :120 objects 256 < size ≤ 1024 :54 objects 1024 < size ≤ PAGE_SIZE :24 objects PAGE_SIZE < size :8 objects size > 131072 :1 object Allocation of the required memory for each processor — an instance of array_cache and an array of pointers to objects with the calculated number of elements — as well as initialization of the data structures is delegated to do_tune_cpucache. A particularly interesting aspect is that the batchcount field is always set to half the calculated number of objects in the cache. This regulates the number of objects processed in one go when a cache is filled. ❑ To conclude initialization, the initialized kmem_cache instance is added to a globally linked list whose list head (cache_chain)isdefinedinmm/slab.c. Allocating Objects kmem_cache_alloc is invoked to obtain objects from a specific cache. Like all malloc functions, it yields either a pointer to the reserved memory area or a null pointer if allocation fails. The function requires two parameters — the cache from which the object is to be obtained and a flag variable to accurately describe the allocation characteristics. void *kmem_cache_alloc (kmem_cache_t *cachep, gfp_t flags) Any of the GFP_ values mentioned in Section 3.5.4 can be specified for the flags.35 As the code flow diagram in Figure 3-50 shows, kmem_cache_alloc is based on the internal function __cache_alloc that requires the same parameters and can be invoked without further ado (this structure was adopted to merge the implementation of kmalloc and kmem_cache_alloc as quickly as possible, as demonstrated in Section 3.6.5). However, __cache_allloc is also only a front-end function to perform all necessary locking operations. The actual work is delegated to ____cache_alloc (with four underscores), as shown in Figure 3-50 (actually, the function do_cache_alloc stands between __cache_alloc and ____cache_alloc, but is only required on NUMA systems). The figure clearly shows that work can follow one of two paths; the first, which is the more frequent and more convenient of the two, is taken if there are free objects in the per-CPU cache. However, if all objects are in use, the cache must be refilled, and in the worst-case scenario, this means that a new slab must be created. 35Notice that the kernel used to provide a differently named set of constants (SLAB_ATOMIC, SLAB_DMA, etc.) with the same numerical values. These have been dropped during the development of kernel 2.6.20 and cannot be used anymore. 276 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 277 Chapter 3: Memory Management kmalloc kmem_cache_alloc _ _cache_alloc _ _ _ _cache_alloc cache_alloc_refill cache_grow Object in per-CPU Cache Ja Take object from cache Return object Return object Nein Find object and take it from the cache Insufficient space in available slabs? Figure 3-50: Code flow diagram for kmem_cache_alloc. Selecting a Cached Object ____cache_alloc can check relatively easily if an object is in the per-CPU cache, as the following code excerpt shows: mm/slab.c static inline void *____cache_alloc(kmem_cache_t *cachep, gfp_t flags) { ac = ac_data(cachep); if (likely(ac->avail)) { ac->touched = 1; objp = ac->entry[--ac->avail]; } else { objp = cache_alloc_refill(cachep, flags); } return objp; cachep is a pointer to the kmem_cache_t instance of the cache used. The ac_data macro yields the asso- ciated array_cache instance for the currently active CPU by returning cachep->array[smp_processor_ id()]. As the objects in memory immediately follow the array_cache instance, the kernel can access them easily with the help of the dummy array at the end of the structure without the explicit need for pointer arithmetic. The object is removed from the cache by decrementing ac->avail. Refilling the Per-CPU Cache The workload is heavier when there are no more objects in the per-CPU cache. The refill operations needed in this situation are located in cache_alloc_refill, which is invoked when the allocation cannot be satisfied directly from the per-CPU cache. 277 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 278 Chapter 3: Memory Management The kernel must now find array_cache->batchcount unused objects to refill the per-CPU cache by first scanning the list of all partially free slabs (slabs_partial) and then taking all free objects one after another by slab_get_obj until no more objects are free in the relevant slab. The kernel then performs the same procedure on all other slabs in the slabs_partial list. If this finds the desired number of objects, the kernel iterates over the slabs_free list of all unused slabs. When objects are taken from a slab, the kernel must also ensure that it places them on the correct slab list (slabs_full or slabs_partial, depending on whether the slab was totally emptied or still contains some objects). The above is implemented by the following code: mm/slab.c static void *cache_alloc_refill(kmem_cache_t *cachep, gfp_t flags) { ... while (batchcount > 0) { /* Select list from which slabs are to be taken (first slabs_partial, then slabs_free) */ ... slabp = list_entry(entry, struct slab, list); while (slabp->inuse < cachep->num && batchcount--) { /* get obj pointer */ ac->entry[ac->avail++] = slab_get_obj(cachep, slabp, node); } check_slabp(cachep, slabp); /* move slabp to correct slabp list: */ list_del(&slabp->list); if (slabp->free == BUFCTL_END) list_add(&slabp->list, &l3->slabs_full); else list_add(&slabp->list, &l3->slabs_partial); } ... } The key to removing one slab element after another is in slab_get_obj: mm/slab.c static void *slab_get_obj(struct kmem_cache *cachep, struct slab *slabp, int nodeid) { void *objp = index_to_obj(cachep, slabp, slabp->free); kmem_bufctl_t next; slabp->inuse++; next = slab_bufctl(slabp)[slabp->free]; slabp->free = next; return objp; } 278 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 279 Chapter 3: Memory Management Recall from Figure 3-47 that the kernel uses an interesting system to keep track of free entries: The index of the free object that is currently under consideration is stored in slabp->free, and the index of the next free object, is kept in the management array. Obtaining the object that belongs to a given index is a matter of some simple pointer manipulation per- formed in index_to_obj. slab_bufctl is a macro that yields a pointer to the kmem_bufctl array after slabp. Let us return to cache_alloc_grow. If no free object is found although all slabs have been scanned, the cache must be enlarged using cache_grow. This is a costly operation examined in the next section. Growing the Cache Figure 3-51 shows the code flow diagram for cache_grow. Compute offset and next color Set page pointer Add slab to cache cache_grow kmem_getpages alloc_slabmgt alloc_pages_node cache_init_objs Figure 3-51: Code flow diagram for cache_grow. The arguments of kmem_cache_alloc are passed to cache_grow. It is also possible to specify an explicit node from which the fresh memory pages are to be supplied. The color and offset are first calculated: mm/slab.c static int cache_grow(struct kmem_cache *cachep, gfp_t flags, int nodeid, void *objp) { ... l3 = cachep->nodelists[nodeid]; ... offset = l3->colour_next; l3->colour_next++; if (l3->colour_next >= cachep->colour) l3->colour_next = 0; offset *= cachep->colour_off; ... } 279 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 280 Chapter 3: Memory Management The kernel restarts counting at 0 when the maximum number of colors is reached; this automatically results in a zero offset. The required memory space is allocated page-by-page by the buddy system using the kmem_getpages helper function. The sole purpose of this function is to invoke the alloc_pages_node function discussed in Section 3.5.4 with the appropriate parameters. The PG_slab bit is also set on each page to indicate that the page belongs to the buddy system. When a slab is used to satisfy short-lived or reclaimable allocations, the flag __GFP_RECLAIMABLE is passed down to the buddy system. Recall from Section 3.5.2 that this is important to allocate the pages from the appropriate migrate list. The allocation of the management head for the slab is not very exciting. The relevant alloc_slabmgmt function reserves the required space if the head is stored off-slab; if not, the space is already reserved on the slab. In both situations, the colouroff, s_mem,andinuse elements of the slab data structure must be initialized with the appropriate values. The kernel then establishes the associations between the pages of the slab and the slab or cache structure by invoking slab_map_pages. This function iterates over all page instances of the pages newly allo- cated for the slab and invokes page_set_cache and page_set_slab for each page. These two functions manipulate (or misuse) the lru element of a page instance as follows: mm/slab.c static inline void page_set_cache(struct page *page, struct kmem_cache *cache) { page->lru.next = (struct list_head *)cache; } static inline void page_set_slab(struct page *page, struct slab *slab) { page->lru.prev = (struct list_head *)slab; } cache_init_objs initializes the objects of the new slab by invoking the constructor for each object assum- ing it is present. (As only a very few parts of the kernel make use of this option, there is normally little to do in this respect.) The kmem_bufctl list of the slab is also initialized by storing the value i + 1 at array position i: because the slab is as yet totally unused, the next free element is always the next consecutive element. As per convention, the last array element holds the constant BUFCTL_END. The slab is now fully initialized and can be added to the slabs_free list of the cache. The number of new objects generated is also added to the number of free objects in the cache (cachep->free_objects). Freeing Objects When an allocated object is no longer required, it must be returned to the slab allocator using kmem_cache_free. Figure 3-52 shows the code flow diagram of this function. kmem_cache_free immediately invokes __cache_free and forwards its arguments unchanged. (Again the reason is to prevent code duplication in the implementation of kfree, as discussed in Section 3.6.5.) As with allocation, there are two alternative courses of action depending on the state of the per-CPU cache. If the number of objects held is below the permitted limit, a pointer to the object in the cache is stored. 280 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 281 Chapter 3: Memory Management Include object into cache Move remaining cache entries upward Yes No kmem_cache_free _ _cache_free cache_flush_array free_block Space in per-CPU array available? Figure 3-52: Code flow diagram for kmem_cache_free. mm/slab.c static inline void __cache_free(kmem_cache_t *cachep, void *objp) { ... if (likely(ac->avail < ac->limit)) { ac->entry[ac->avail++] = objp; return; } else { cache_flusharray(cachep, ac); ac->entry[ac->avail++] = objp; } } If not, some objects (the exact number is given by array_cache->batchcount) must be moved from the cache back into the slabs starting with the array elements with the lowest numbers — because the cache implementation applies the LIFO principle, these are objects that have been in the array longest and whose data are therefore least likely still to be held in the CPU cache. Implementation is delegated to cache_flusharray. In turn, this function invokes free_block to move the objects from the cache to their original slabs and shifts the remaining objects in the cache to the start of the array. For example, if there is space for 30 objects in the cache and the batchcount is 15, the objects at positions 0 to 14 are moved back into the slabs. The remaining objects numbered 15 to 29 are shifted upward in the cache so that they now occupy positions 0 to 14. Moving objects from the cache back onto the slabs is instructive, so it’s well worth taking a closer look at free_block. The arguments required by this function are the kmem_cache_t instance of the cache, a pointer to an array consisting of pointers to the objects in the cache, an integer to indicate the number of objects in the array, and the node whose memory is just being processed. The function iterates over all objects in objpp after the number of unused objects in the cache data struc- ture has been updated. mm/slab.c static void free_block(kmem_cache_t *cachep, void **objpp, int nr_objects, int node) { int i; 281 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 282 Chapter 3: Memory Management struct kmem_list3 *l3; for (i = 0; i < nr_objects; i++) { void *objp = objpp[i]; struct slab *slabp; ... The following operations must be performed for each object: mm/slab.c slabp = virt_to_slab(objp) l3 = cachep->nodelists[node]; list_del(&slabp->list); slab_put_obj(cachep, slabp, objp, node); slabp->inuse--; l3->free_objects++; Before it can be established to which slab an object belongs, it is first necessary to invoke virt_to_page to find the page in which the object is located. The association with the slab is determined using page_get_slab as discussed above. The slab is (temporarily) removed from the lists of the cache. slab_put_obj is used to reflect this action in the freelist: The first object to be used for allocation purposes is the one just removed, and the next object in the list is the one that was previously first. Thereafter, the slab is reinserted in the linked lists of the cache: mm/slab.c ... /* fixup slab chains */ if (slabp->inuse == 0) { if (l3->free_objects > l3->free_limit) { l3->free_objects -= cachep->num; slab_destroy(cachep, slabp); } else { list_add(&slabp->list, &l3->slabs_free); } } else { list_add(&slabp->list, &l3->slabs_partial); } } } The slab is normally placed on the slabs_free list if, after deletion, all objects in the slab are unused (slab->inuse == 0). Exception: The number of free objects in the cache is above the predefined limit cachep->free_limit. In this case, the complete slab is returned to the buddy system using slab_destroy. The slab is inserted into the slabs_partial list of the cache if it contains both used and unused objects. 282 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 283 Chapter 3: Memory Management Destroying Caches The kmem_cache_destroy function must be invoked to destroy a complete cache in which there are only unused objects. This function is needed primarily when removing modules that want to return all memory space allocated to them.36 Since the implementation itself reveals nothing new, we will confine ourselves to outlining the main steps needed to remove a cache: ❑ The slabs on the slabs_free list are scanned successively. The destructor is first invoked for each object on each slab, and then the slab memory space is returned to the buddy system. ❑ The memory space for the per-CPU caches is freed. ❑ The data are removed from the cache_cache list. 3.6.5 General Caches The kmalloc and kfree functions must be used to allocate and free memory in the classic sense rather than objects. These are the kernel equivalents to the malloc and free functions from the C standard library in userspace.37 I have already noted several times that kmalloc and kfree are implemented as slab allocator front-ends and mimic the semantics of malloc/free as best they can. We can therefore deal with their implementa- tion succinctly. Implementation of kmalloc Thebaseofkmalloc is an array that groups slab caches for memory areas of graded sizes. The array entries are instances of the cache_sizes data structure that is defined as follows: struct cache_sizes { size_t cs_size; kmem_cache_t *cs_cachep; kmem_cache_t *cs_dmacachep; #ifdef CONFIG_ZONE_DMA struct kmem_cache *cs_dmacachep; #endif } size specifies the size of the memory area for which the entry is responsible. There are two slab caches for each size, one of which supplies DMA-suitable memory. The statically defined malloc_sizes array groups the available sizes essentially using powers of 2 between 25 = 32 and 225 = 131, 072, depending on the setting of KMALLOC_MAX_SIZE as discussed above. 36This is not mandatory. If a module wants to obtain persistent memory that is preserved between unloading a module and reload- ing the next time (assuming, of course, that the system is not rebooted in the meantime), it can retain a cache so that the data it contains are available for reuse. 37Use of printk, kmalloc and kfree in userspace programs is an unmistakable sign of too much contact with kernel programming. 283 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 284 Chapter 3: Memory Management mm/slab.c static struct cache_sizes malloc_sizes[] = { #define CACHE(x) { .cs_size = (x) }, #if (PAGE_SIZE == 4096) CACHE(32) #endif CACHE(64) #if L1_CACHE_BYTES < 64 CACHE(96) #endif CACHE(128) #if L1_CACHE_BYTES < 128 CACHE(192) #endif CACHE(256) CACHE(512) CACHE(1024) CACHE(2048) CACHE(4096) CACHE(8192) CACHE(16384) CACHE(32768) CACHE(65536) CACHE(131072) #if KMALLOC_MAX_SIZE >= 262144 CACHE(262144) #endif #if KMALLOC_MAX_SIZE >= 524288 CACHE(524288) #endif ... #if KMALLOC_MAX_SIZE >= 33554432 CACHE(33554432) CACHE(ULONG_MAX) There is always a cache for allocations up to the maximum size that can be represented in an unsigned long variable. However, this cache (in contrast to all others) is not filled with elements in advance; this allows the kernel to ensure that each giant allocation is satisfied with freshly allocated memory pages. As allocations of this size can request the entire memory of the system, a corresponding cache would not be particularly useful. However, this kernel approach makes sure that very large allocation requests can be satisfied if sufficient memory is available. The pointers to the corresponding caches are not initially filled. They are assigned their correct value when initialization is performed with kmem_cache_init. kmalloc from first checks whether a constant is specified as the memory size; in this case, the required cache can be determined statically at compilation time, and this delivers speed gains. If not, __kmalloc is invoked to find the cache of matching size. The function is a parameter-conversion front end for __do_kmalloc: mm/slab.c void *__do_kmalloc(size_t size, gfp_t flags) { kmem_cache_t *cachep; 284 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 285 Chapter 3: Memory Management cachep = __find_general_cachep(size, flags); if (unlikely(ZERO_OR_NULL_PTR(cachep))) return NULL; return __cache_alloc(cachep, flags); } Once __find_general_cachep has found a suitable cache (it iterates over all possible kmalloc sizes to find a matching cache), the heavy work is delegated to the __cache_alloc function discussed above. Implementation of kfree kfree is likewise easy to implement: mm/slab.c void kfree(const void *objp) { kmem_cache_t *c; unsigned long flags; if (unlikely(ZERO_OR_NULL_PTR(objp))) return; c = virt_to_cache(objp)); __cache_free(c, (void*)objp); } kfree hands over the actual work to the __cache_free function also discussed above once the cache associated with the memory pointer has been found. 3.7 Processor Cache and TLB Control Caches are crucial in terms of overall system performance, which is why the kernel tries to exploit them as effectively as possible. It does this primarily by skillfully aligning kernel data in memory. A judicious mix of normal functions, inline definitions, and macros also helps extract greater performance from the processor. The compiler optimizations discussed in Appendix C also make their contribution. However, the above aspects affect the cache only indirectly. Use of the correct alignment for a data structure does indeed have an effect on the cache but only implicitly — active control of the processor cache is not necessary. Nevertheless, the kernel features some commands that act directly on the cache and the TLB of the pro- cessor. However, they are not intended to boost system efficiency but to maintain the cache contents in a consistent state and to ensure that no entries are incorrect and out-of-date. For example, when a mapping is removed from the address space of a process, the kernel is responsible for removing the corresponding entries from the TLBs. If it failed to do so and new data were added at the position previously occupied by the mapping, a read or write operation to the virtual address would be redirected to the incorrect location in physical memory. The hardware implementation of caches and TLBs differs significantly from architecture to architecture. The kernel must therefore create a view on TLBs and caches that takes adequate account of the different approaches without neglecting the specific properties of each architecture. 285 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 286 Chapter 3: Memory Management ❑ The meaning of the translation lookaside buffer is abstracted to refer to a mechanism that translates a virtual address into a physical address.38 ❑ The kernel regards a cache as a mechanism that provides rapid access to data by reference to a virtual address without the need for a request to RAM memory. There is not always an explicit difference between data and instruction caches. The architecture-specific code is responsible for any differentiation if its caches are split in this manner. It is not necessary for each processor type to implement every control function defined by the kernel. If a function is not required, its invocation can be replaced with an empty operation (do {} while (0))thatis optimized away by the compiler. This is very frequently the case with cache-related operations because, as above, the kernel assumes that addressing is based on virtual addresses. The resultant problems do not occur in physically organized caches so that it is not usually necessary to implement the cache control functions. The following functions must be made available (even if only as an empty operation) by each CPU- specific part of the kernel in order to control the TLBs and caches39: ❑ flush_tlb_all and flush_cache_all flush the entire TLB/cache. This is only required when the page tables of the kernel (and not of a userspace process) are manipulated because a modification of this kind affects not only all processes but also all processors in the system. ❑ flush_tlb_mm(struct mm_struct *mm) and flush_cache_mm flush all TLB/cache entries belong- ing to the address space mm. ❑ flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) and flush_cache_range(vma, start, end) flush all entries from the TLB/cache between the start and end virtual addresses in the address range vma->vm_mm. ❑ flush_tlb_page(struct vm_area_struct *vma, unsigned long page) and flush_cache_page(vma, page) flush all entries from the TLB/cache whose virtual addresses are in an interval that begins at page and consists of PAGE_SIZE bytes. ❑ update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t pte) is invoked after a page fault has been handled. It inserts information in the memory management unit of the processor so that the entry at the virtual address address is described by the page table entry pte. This function is needed only if there is an external MMU. Typically, the MMU is integrated into the processor, but MIPS processors, for example, have external MMUs. The kernel makes no distinction between data and instruction caches. If a distinction is required, the processor-specific code can reference the VM_EXEC flag in vm_area_struct->flags to ascertain whether the cache contains data or instructions. The flush_cache_ and flush_tlb_ functions very often occur in pairs; for instance, when the address space of a process is duplicated using fork. kernel/fork.c flush_cache_mm(oldmm); ... 38Whether TLBs are the only hardware resource for doing this or whether other alternatives (e.g., page tables) are provided is irrele- vant. 39The following description is based on the documentation by David Miller [Mil] in the kernel sources. 286 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 287 Chapter 3: Memory Management /* Manipulate page tables */ ... flush_tlb_mm(oldmm); The sequence of operations — cache flushing, memory manipulation, and TLB flushing — is important for two reasons: ❑ If the sequence were reversed, another CPU in a multiprocessor system could take the wrong information from the process table after the TLBs have been flushed but before the correct infor- mation is supplied. ❑ Some architectures require the presence of ‘‘virtual-to-physical‘‘ transformation rules in the TLB when the cache is flushed (caches with this property are referred to as strict). flush_tlb_mm must execute after flush_cache_mm to guarantee that this is the case. Some control functions apply specifically to data caches (flush_dcache_ ... ) or instruction caches (flush_icache_ ... ). ❑ flush_dcache_page(struct page *page) helps prevent alias problems that arise if a cache may contain several entries (with different virtual addresses) that point to the same page in memory. It is always invoked when the kernel writes to a page in the page cache or when it wants to read data from a page that is also mapped in userspace. This routine gives each architecture in which alias problems can occur an opportunity to prevent such problems. ❑ flush_icache_range(unsigned long start, unsigned long end) is invoked when the kernel writes data to kernel memory (between start and end) for subsequent execution. A standard example of this scenario is when a module is loaded into the kernel. The binary data are first copied to RAM and are then executed. flush_icache_range ensures that data and instruction caches do not interfere with each other if implemented separately. ❑ flush_icache_user_range(*vma, *page, addr, len) is a special function for the ptrace mecha- nism. It is needed to propagate changes to the address space of a traced process. It is beyond the scope of this book to discuss the implementation details of the cache and TLB control functions. Too much background knowledge on the structure of the underlying processor (and the subtle problems involved) would be required for a full understanding of the implementation details. 3.8 Summary This chapter has discussed many aspects of memory management. Our focus lies on physical memory management, but the connection between virtual and physical memory via page tables has also been cov- ered. Although the architecture-specific details in this area differ greatly among the various architectures supported by Linux, an architecture-independent set of data structures and functions allows generic code to manipulate the page tables. However, some architecture-specific code is required before the generic view is enabled, and this code runs during the boot process. Once the kernel is up and running, memory management is handled by two layers: The buddy system is responsible for the management of physical page frames, while the slab allocator must handle small allocations and provides an in-kernel equivalent to the malloc function family known from userland programming. 287 Mauerer runc03.tex V2 - 09/04/2008 4:52pm Page 288 Chapter 3: Memory Management The buddy system is centered around the idea of splitting and recombining larger continuous blocks of pages. When a continuous area becomes free, the kernel notices this automatically, and can use it once the need for a corresponding allocation arises. Since this is unfortunately not sufficient to prevent fragmen- tation of physical memory after longer uptimes in a satisfactory manner, recent kernels have acquired anti-fragmentation techniques that allow grouping pages by their mobility, on the one hand, and aug- ment the kernel with a new virtual memory zone, on the other hand. Both help to avoid fragmentation by essentially decreasing the chance that coalescing of larger regions is prohibited by allocated blocks in their middle. The slab allocator is implemented on top of the buddy system. It does not only allow to allocate small chunks of memory for arbitrary use, but additionally offers the possibility to create specific caches for often used data structures. Initializing memory management is challenging because the data structures employed by the subsystem itself also require memory, which must be allocated from somewhere. We have seen how the kernel solves the situation by introducing a very simple boot memory allocator that is shut down after the proper allocation routines function fully. While we have mostly focused on physical memory here, the next chapter will discuss how the virtual address space is managed by the kernel. 288 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 289 Virtual Process Memory The virtual address space of userland processes is an important abstraction of Linux: It allows the same view of the system to each running process, and this makes it possible for multiple processes to run simultaneously without interfering with the memory contents of the others. Additionally, it allows various advanced programming techniques like memory mappings. In this chapter, I will discuss how these concepts are realized in the kernel. This also requires an examination of the connection between page frames of the available physical RAM and pages in all virtual process address spaces: The reverse mapping technique helps to track which virtual pages are backed by which physical page, and page fault handling allows filling the virtual address space with data from block devices on demand. 4.1 Introduction All the memory management methods discussed in the preceding chapter were concerned either with the organization of physical memory or management of the virtual kernel address space. This section examines the methods required by the kernel to manage the virtual user address space. For a variety of reasons, some of which are given below, this is more complex than managing kernel address space: ❑ Each application has its own address space that is segregated from all other applications. ❑ Usually only a few sections of the large linear address space available to each userspace process are actually used, and they may also be some distance from each other. The kernel needs data structures to efficiently manage these (randomly) spread sections. ❑ Only the smallest part of the address space is directly associated with physical pages. Infre- quently used parts are linked with pages only when necessary. ❑ The kernel has trust in itself, but not in user processes. For this reason, each operation to manipulate user address space is accompanied by various checks to ensure that programs cannot acquire more rights than are due to them and thus endanger system stability and security. Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 290 Chapter 4: Virtual Process Memory ❑ The fork-exec model used under Unix to generate new processes (described in Chapter 2) is not very powerful if implemented carelessly. The kernel must therefore concentrate on managing user address spaces as efficiently as possible by resorting to a few tricks. Most of the ideas discussed below are based on the assumption that the system has a memory management unit (or MMU) that supports the use of virtual memory. This is, in fact, the situation on all ‘‘normal‘‘ processors. However, during the development of Linux 2.5, three architectures that do not provide an MMU were added to the kernel sources — V850E, H8300, and m68knommu. Another one (blackfin) was added during the development of kernel 2.6.22. Some of the functions examined below are not available on these CPUs, and the interface to the outside returns error messages because the underlying mechanisms are not implemented in the kernel and cannot be implemented owing to the lack of processor support. The information below covers only machines with MMU. I do not deal with the oddities and modifications needed for MMU-less architectures. 4.2 Virtual Process Address Space The virtual address space of each process starts at address 0 and extends to TASK_SIZE - 1;thekernel address space begins above this. On IA-32 systems with 232 = 4 GiB, the total address space is usually split in a 3:1 ratio on which we focus in the information below. The kernel is assigned 1 GiB, while 3 GiB is available to each userspace process. Other ratios are possible but yield benefits only on very specific configurations and with certain work loads, as discussed above. A very important aspect relating to system integrity is that user programs may access only the lower part of the overall address space but not the kernel part. Neither is it possible for a user process to manipulate parts of the address space of another process without previous ‘‘agreement,’’ simply because these parts are invisible to it. The contents of the virtual address space portion of the kernel are always the same regardless of which user process is currently active. Depending on hardware, this is achieved either by manipulating the page tables of user processes so that the upper part of the virtual address space always appears to be identical or by instructing the processor itself to provide a separate address space for the kernel, which is mapped above each user address space. Recall that this is visualized in Figure 1-3 in the Introduction. The virtual address space is made up of many sections of varying sizes that serve different purposes and must be handled differently. For example, in most cases, it is not permitted to modify the text segment, but it must be possible to execute its contents. On the other hand, it must be possible to modify the contents of a text file mapped into the address space but not to execute such contents as this doesn’t make sense — it’s just data and not machine code. 4.2.1 Layout of the Process Address Space The virtual address space is populated by a number of regions. How they are distributed is architecture- specific, but all approaches have the following elements in common: ❑ The binary code of the code currently running. This code is normally referred to as text and the area of virtual memory in which it is located as a text segment.1 ❑ The code of dynamic libraries used by the program. 1This is not the same as a hardware segment, which is featured in some architectures and acts as a separate address space. It is simply the linear address space area used to hold the data. 290 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 291 Chapter 4: Virtual Process Memory ❑ The heap where global variables and dynamically generated data are stored. ❑ The stack used to hold local variables and to implement function and procedure calls. ❑ Sections with environment variables and command-line arguments. ❑ Memory mappings that map the contents of files into the virtual address space. Recall from Chapter 2 that each process in the system is equipped with an instance of struct mm_struct that can be accessed via the task structure. This instance holds memory management information for the process: struct mm_struct { ... unsigned long (*get_unmapped_area) (struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); ... unsigned long mmap_base; /* base of mmap area */ unsigned long task_size; /* size of task vm space */ ... unsigned long start_code, end_code, start_data, end_data; unsigned long start_brk, brk, start_stack; unsigned long arg_start, arg_end, env_start, env_end; ... } The start and end of the virtual address space area consumed by the executable code are marked by start_code and end_code. Similarly, start_data and end_data mark the region that contains initialized data. Notice that the size of these areas does not change once an ELF binary has been mapped into the address space. The start address of the heap is kept in start_brk, while brk denotes the current end of the heap area. While the start is constant during the lifetime of a process, heap size and thus the value of brk will vary. The position of the argument list and the environment is described by arg_start and arg_end,respec- tively, env_start and env_end. Both regions reside in the topmost area of the stack. mmap_base denotes the starting point for memory mappings in the virtual address space, and get_ unmapped_area is invoked to find a suitable place for a new mapping in the mmap area. task_size — variable names don’t lie — stores the task size of the corresponding process. For native applications, this will usually be TASK_SIZE. However, 64-bit architectures are often binary-compatible with their predecessors. If a 32-bit binary is executed on a 64-bit machine, then task_size describes the effective task size visible to the binary. The individual architectures can influence the layout of the virtual address space by several configuration options: ❑ If an architecture wants to choose between different possibilities for how the mmap area is arranged, it needs to set HAVE_ARCH_PICK_MMAP_LAYOUT and provide the function arch_ pick_mmap_layout. 291 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 292 Chapter 4: Virtual Process Memory ❑ When a new memory mapping is created, the kernel needs to find a suitable place for it unless a specific address has been specified by the user. If the architecture wants to choose the proper location itself, it must set the pre-processor symbol HAVE_ARCH_UNMAPPED_AREA and define the function arch_get_unmapped_area accordingly. ❑ New locations for memory mappings are usually found by starting the search from lower mem- ory locations and progressing toward higher addresses. The kernel provides the default func- tion arch_get_unmapped_area_topdown to perform this search, but if an architecture wants to provide a specialized implementation, it needs to set the pre-processor symbol HAVE_ARCH_ GET_UNMAPPED_AREA. ❑ Usually, the stack grows from bottom to top. Architectures that handle this differently need to set the configuration option CONFIG_STACK_GROWSUP.2 In the following, only stacks that grow from top to bottom are considered. Finally, we need to consider the task flag PF_RANDOMIZE. If it is set, the kernel does not choose fixed locations for stack and the starting point for memory mappings, but varies them randomly each time a new process is started. This complicates, for instance, exploiting security holes that are caused by buffer overflows. If an attacker cannot rely on a fixed address where the stack can be found, it will be much harder to construct malicious code that deliberately manipulates stack entries after access to the memory region has been gained by a buffer overflow. Figure 4-1 illustrates how the aforementioned components are distributed across the virtual address space on most architectures. Text Heap MMAP Stack TASK_SIZE STACK_TOP-randomized_variable mm->mmap_base (TASK_UNMAPPED_SIZE) Gap already used Figure 4-1: Composition of the linear process address space. 2Currently only PA-Risc processors require this option. The constants in the kernel thus have a slight tendency toward a sit- uation where the stack grows from downward, albeit the PA-Risc code is not quite satisfied with that, as we can read in include/asm-parisc/a.out.h: /* XXX: STACK_TOP actually should be STACK_BOTTOM for parisc. * prumpf *\ The funny thing is that ‘‘prumpf’’ is not a grumpy sign of discontent, but an abbreviation for a developer, Philipp Rumpf :-) 292 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 293 Chapter 4: Virtual Process Memory How the text segment is mapped into the virtual address space is determined by the ELF standard (see Chapter E for more information about this binary format). A specific starting address is specified for each architecture: IA-32 systems start at 0x08048000, leaving a gap of roughly 128 MiB between the lowest possible address and the start of the text mapping that is used to catch NULL pointers. Other architectures keep a similar hole: UltraSparc machines use 0x100000000 as the starting point of the text segment, while AMD64 uses 0x0000000000400000. The heap starts directly above the text segment and grows upward. The stack starts at STACK_TOP, but the value is decremented by a small random amount if PF_RANDOMIZE is set. STACK_TOP must be defined by each architecture, and most set it to TASK_SIZE — the stack starts at the highest possible address of the user address space. The argument list and environment of a process are stored as initial stack elements. The region for memory mappings starts at mm_struct->mmap_base, which is usually set to TASK_ UNMAPPED_BASE, needing to be defined by every architecture. In nearly all cases, TASK_SIZE/3 is chosen. Note that the start of the mmap region is not randomized if the default kernel approach is used. Using the described address space layout works very well on machines that provide a large virtual address space. However, problems can arise on 32-bit machines. Consider the situation on IA-32: The virtual address space ranges from 0 to 0xC0000000, so 3 GiB are available for each user process. TASK_ UNMAPPED_BASE starts at 0x4000000, that is, at 1 GiB. Unfortunately, this implies that the heap can only consume roughly 1 GiB before it crashes right into the mmap area, which is clearly not a desirable situation. The problem is caused by the memory mapping region that is located in the middle of the virtual address space. This is why a new virtual address space layout for IA-32 machines (in addition to the classical one, which can still be used) was introduced during the development of kernel 2.6.7. It is illustrated in Figure 4-2. Gap already used Text 0×0804 80000 0 MMAP Heap Stack TASK_SIZE Random offset mm->mmap_base STACK_TOP-randomized_variable Figure 4-2: Layout of the virtual address space on IA-32 machines when the mmap region is expanded from top to bottom. 293 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 294 Chapter 4: Virtual Process Memory The idea is to limit the maximal stack size to a fixed value. Since the stack is bounded, the region into which memory mappings are installed can then be started immediately below the end of the stack. In contrast to the classical approach, it now expands from top to bottom. Since the heap is still located in the lower region of the virtual address space and grows upward, both mmap region and heap can expand until there is really no portion of the virtual address space left. To ensure that the stack does not collide with the mmap region, a safety gap is installed between both. 4.2.2 Creating the Layout The address space of a task is laid out when an ELF binary is loaded with load_elf_binary —recallthat the function is used by the exec system call. Loading an ELF file is cluttered with numerous technical details that are not interesting for our purposes, so the code flow diagram in Figure 4-3 concentrates on the steps required to set up the virtual memory region. Set PF_RANDOMIZE if required load_elf_binary arch_pick_mmap_layout setup_arg_pages Figure 4-3: Code flow diagram for load_elf_binary. Address space randomization is enabled if the global variable randomize_va_space is set to 1. This is usually the case, but is disabled for Transmeta CPUs because it has a negative speed impact on such machines. Besides, the user can use /proc/sys/kernel/randomize_va_space to disable the feature. The address space layout is selected in arch_pick_mmap_layout. If the architecture does not provide a specific function, the kernel’s default routine sets up the address space as shown in Figure 4-1. It is, however, more interesting to observe how IA-32 chooses between the classical and the new alternative: arch/x86/mm/mmap_32.c void arch_pick_mmap_layout(struct mm_struct *mm) { /* * Fall back to the standard layout if the personality * bit is set, or if the expected stack growth is unlimited: */ if (sysctl_legacy_va_layout || (current->personality & ADDR_COMPAT_LAYOUT) || current->signal->rlim[RLIMIT_STACK].rlim_cur == RLIM_INFINITY) { mm->mmap_base = TASK_UNMAPPED_BASE; mm->get_unmapped_area = arch_get_unmapped_area; mm->unmap_area = arch_unmap_area; } else { mm->mmap_base = mmap_base(mm); 294 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 295 Chapter 4: Virtual Process Memory mm->get_unmapped_area = arch_get_unmapped_area_topdown; mm->unmap_area = arch_unmap_area_topdown; } } The old layout is chosen if the user has explicitly instructed to do so via /proc/sys/kernel/legacy_ va_layout, if a binary that was compiled for a different Unix flavor that requires the old layout is exe- cuted, or — most importantly — the stack may grow infinitely. This makes it difficult to find a bound for the stack below which the mmap region can start. In the classical case, the start of the mmap area is at TASK_UNMAPPED_BASE, which resolves to 0x4000000, and the standard function arch_get_unmapped_area (despite its name, the function is not necessarily architecture-specific, but there’s also a standard implementation available in the kernel) is used to grow new mappings from bottom to top. When the new layout is used, memory mappings grow from top to bottom. The standard function arch_get_unmapped_area_topdown (which I will not consider in detail) is responsible for this. More interesting is how the base address for memory mappings is chosen: arch/x86/mm/mmap_32.c #define MIN_GAP (128*1024*1024) #define MAX_GAP (TASK_SIZE/6*5) static inline unsigned long mmap_base(struct mm_struct *mm) { unsigned long gap = current->signal->rlim[RLIMIT_STACK].rlim_cur; unsigned long random_factor = 0; if (current->flags & PF_RANDOMIZE) random_factor = get_random_int() % (1024*1024); if (gap < MIN_GAP) gap = MIN_GAP; else if (gap > MAX_GAP) gap = MAX_GAP; return PAGE_ALIGN(TASK_SIZE - gap - random_factor); } The lowest possible stack location that can be computed from the maximal stack size can be used as the start of the mmap area. However, the kernel ensures that the stack spans at least 128 MiB. Additionally, it is ensured that at least a small portion of the address space is not taken up by the stack if a gigantic stack limit is specified. If address space randomization is requested, the position is modified by a random offset of maximally 1 MiB. Additionally, the kernel ensures that the region is aligned along the page frame size because this is required by the architecture. At a first glance, one could assume that life is easier for 64-bit architectures because they should not have to choose between different address layouts — the virtual address space is so large that collisions of heap and mmap region are nearly impossible. 295 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 296 Chapter 4: Virtual Process Memory However, the definition of arch_pick_mmap_layout for the AMD64 architecture shows that another complication arises: arch/x86_64/mmap.c void arch_pick_mmap_layout(struct mm_struct *mm) { #ifdef CONFIG_IA32_EMULATION if (current_thread_info()->flags & _TIF_IA32) return ia32_pick_mmap_layout(mm); #endif mm->mmap_base = TASK_UNMAPPED_BASE; if (current->flags & PF_RANDOMIZE) { /* Add 28bit randomness which is about 40bits of address space because mmap base has to be page aligned. or ~1/128 of the total user VM (total user address space is 47bits) */ unsigned rnd = get_random_int() & 0xfffffff; mm->mmap_base += ((unsigned long)rnd) << PAGE_SHIFT; } mm->get_unmapped_area = arch_get_unmapped_area; mm->unmap_area = arch_unmap_area; } If binary emulation for 32-bit applications is enabled, any process that runs in compatibility mode should see the same address space as it would encounter on a native machine. Therefore, ia32_pick_ mmap_layout is used to lay out the address space for 32-bit applications. The function is an identical copy of arch_pick_mmap_layout for IA-32 systems, as discussed above. The classic layout for virtual address space is always used on AMD64 systems so that there is no need to distinguish between the various options. Address space randomization is performed by shifting the otherwise fixed mmap_base if the PF_RANDOMIZE flag is set. Let us go back to load_elf_binary. Finally, the function needs to create the stack at the appropriate location: static int load_elf_binary(struct linux_binprm *bprm, struct pt_regs *regs) { ... retval = setup_arg_pages(bprm, randomize_stack_top(STACK_TOP), executable_stack); ... } The standard function setup_arg_pages is used for this purpose. I will not discuss it in detail because it is only technical. The function requires the top of the stack as a parameter. This is given by the architecture- specific constant STACK_TOP,butrandomize_stack_top ensures that the address is changed by a random amount if address space randomization is required. 296 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 297 Chapter 4: Virtual Process Memory 4.3 Principle of Memory Mappings Because the total virtual address space of all user processes is substantially larger than the available RAM memory, only the most frequently used elements can be associated with a physical page frame. This is not a problem because most programs occupy only a small part of the memory actually available to them. Let’s look at the situation in which a file is manipulated by a text editor. Typically, the user is only bothered with the end of the file so although the complete file is mapped into memory, only a few pages are actually used to store the data at the end of the file. As for the beginning of the file, the kernel need only keep the information in address space about where on the disk to find the data and how to read them when they are required. The situation is similar with the text segment — only part of it is always needed. If we stay with the example of the text editor, only the code for the central editing function is required. Other parts — the Help system or the obligatory Web and e-mail client common to all programs — are only loaded when explicitly required by the user.3 The kernel must provide data structures to establish an association between the regions of the virtual address space and the places where the related data are located. In the case of a mapped text file, for example, the virtual memory area must be associated with the area on the hard disk in which the filesys- tem has stored the contents of the file. This is illustrated in Figure 4-4. Virtual address space File on hard disk Figure 4-4: Mapping a file into virtual memory. Of course, I have shown the situation in simplified form because file data are not generally stored con- tiguously on hard disk but are distributed over several smaller areas (this is discussed in Chapter 9). The kernel makes use of the address_space data structure4 to provide a set of methods to read data from a backing store — from a filesystem, for example. address_spaces therefore form an auxiliary layer to represent the mapped data as a contiguous linear area to memory management. Allocating and filling pages on demand is known as demand paging. It is based on interaction between the processor and the kernel using a variety of data structures as shown in Figure 4-5. 3I assume that all program parts reside in a single, large binary file. Of course, program parts can also be loaded at the explicit request of the program itself, but I do not discuss this here. 4Unfortunately, the names for the virtual address space and the address space indicating how the data are mapped are identical. 297 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 298 Chapter 4: Virtual Process Memory Backing Store Virtual address space Addres space region Physical page frames Not mapped, in use Mapped, in use Not mapped, not in use Page tables Figure 4-5: Interaction of data structures during demand paging. ❑ A process tries to access a memory address that is in user address space but cannot be resolved using the page tables (there is no associated page in RAM memory). ❑ The processor then triggers a page fault that is forwarded to the kernel. ❑ The kernel runs through the process address space data structures responsible for the area in which the fault occurred to find the appropriate backing store or to establish that access was, indeed, incorrect. ❑ A physical page is allocated and filled with the requisite data from the backing store. ❑ The physical page is incorporated into the address space of the user process with the help of the page tables, and the application resumes. These actions are transparent to user processes; in other words, the processes don’t notice whether a page is actually available or must first be requested by means of demand paging. 4.4 Data Structures Recall that struct mm_struct is important — it provides all necessary information to lay out a task in memory as discussed before. Additionally, it includes the following elements for management of all memory regions in the virtual address space of a user process. struct mm_struct { struct vm_area_struct * mmap; /* list of VMAs */ struct rb_root mm_rb; struct vm_area_struct * mmap_cache; /* last find_vma result */ ... } The following sections discuss the meanings of the entries. 298 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 299 Chapter 4: Virtual Process Memory 4.4.1 Trees and Lists Each region is described by an instance of vm_area_struct, and the regions of a process are sorted in two ways: 1. On a singly linked list (starting with mm_struct->mmap). 2. In a red-black tree whose root element is located in mm_rb. mmap_cache is a cache for the region last handled; its meaning will become clear in Section 4.5.1. Red-black trees are binary search trees whose nodes also have a color (red or black). They exhibit all the properties of normal search trees (and can therefore be scanned very efficiently for a specific element). The red-black property also simplifies re-balancing.5 Readers unfamiliar with this concept are referred to Appendix C, which deals extensively with the structure, properties, and implementation of red-black trees. The start and end addresses describe each region in virtual user address space. The existing regions are included in the linked list in ascending order of start address. Scanning the list to find the region associated with a particular address is a very inefficient operation if there are a very large number of regions (as is the case with data-intensive applications). The individual instances of vm_area_struct are therefore also managed in a red-black tree, which speeds up scanning considerably. To add a new region, the kernel first searches the red-black tree for the region immediately preceding the new region. With its help, it can add the new region to the tree and also to the linear list without having to explicitly scan the list (the algorithm used by the kernel to add new regions is discussed at length in Section 4.5.3). Finally, the situation in memory is illustrated in Figure 4-6. Notice that the representation of the tree is only symbolic and does not reflect the real layout, which is more complicated. mm mmap struct vm_area_struct Red_Black tree mmap_rb struct task_struct struct mm_struct Manage vm_area_structs associated with a process Figure 4-6: Association of vm_area_struct instances with the virtual process space of a process. 5All important tree operations (add, delete, find) can be performed in O(logn),wheren is the number of elements in the tree. 299 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 300 Chapter 4: Virtual Process Memory 4.4.2 Representation of Regions Each region is represented by an instance of vm_area_struct, which is defined (in simplified form) as follows: struct vm_area_struct { struct mm_struct * vm_mm; /* The address space we belong to. */ unsigned long vm_start; /* Our start address within vm_mm. */ unsigned long vm_end; /* The first byte after our end address within vm_mm. */ /* linked list of VM areas per task, sorted by address */ struct vm_area_struct *vm_next; pgprot_t vm_page_prot; /* Access permissions of this VMA. */ unsigned long vm_flags; /* Flags, listed below. */ struct rb_node vm_rb; /* * For areas with an address space and backing store, * linkage into the address_space->i_mmap prio tree, or * linkage to the list of like vmas hanging off its node, or * linkage of vma in the address_space->i_mmap_nonlinear list. */ union { struct { struct list_head list; void *parent; /* aligns with prio_tree_node parent */ struct vm_area_struct *head; } vm_set; struct raw_prio_tree_node prio_tree_node; } shared; /* * A file’s MAP_PRIVATE vma can be in both i_mmap tree and anon_vma * list, after a COW of one of the file pages. A MAP_SHARED vma * can only be in the i_mmap tree. An anonymous MAP_PRIVATE, stack * or brk vma (with NULL file) can only be in an anon_vma list. */ struct list_head anon_vma_node; /* Serialized by anon_vma->lock */ struct anon_vma *anon_vma; /* Serialized by page_table_lock */ /* Function pointers to deal with this struct. */ struct vm_operations_struct * vm_ops; /* Information about our backing store: */ unsigned long vm_pgoff; /* Offset (within vm_file) in PAGE_SIZE units, *not* PAGE_CACHE_SIZE */ struct file * vm_file; /* File we map to (can be NULL). */ void * vm_private_data; /* was vm_pte (shared mem) */ }; 300 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 301 Chapter 4: Virtual Process Memory The individual elements have the following meanings: ❑ vm_mm is a back-pointer to the mm_struct instance to which the region belongs. ❑ vm_start and vm_end specify the virtual start and end addresses of the region in userspace. ❑ The linear linking of all vm_area_struct instances of a process is achieved using vm_next, whereas incorporation in the red-black tree is the responsibility of vm_rb. ❑ vm_page_prot stores the access permissions for the region in the constants discussed in Section 3.3.1, which are also used for pages in memory. ❑ vm_flags is a set of flags describing the region. I discuss the flags that can be set below. ❑ A mapping of a file into the virtual address space of a process is uniquely determined by the interval in the file and the corresponding interval in memory. To keep track of all intervals asso- ciated with a process, the kernel uses a linked list and a red-black tree as described above. However, it is also necessary to go the other way round: Given an interval in a file, the kernel sometimes needs to know all processes into which the interval is mapped. Such mappings are called shared mappings, and the C standard library, which is used by nearly every process in the system, is a prime example of why such mappings are necessary. To provide the required information, all vm_area_struct instances are additionally managed in a priority tree, and the elements required for this are contained in shared. As you can easily imagine from the rather complicated definition of this structure member, this is a tricky business, which is discussed in detail in Section 4.4.3 below. ❑ anon_vma_node and anon_vma are used to manage shared pages originating from anonymous mappings. Mappings that point to the same pages are held on a doubly linked list, where anon_vma_node acts as the list element. There are several of these lists, depending on how many sets of mappings there are that share different physical pages. The anon_vma element serves as a pointer to the management structure that is associated with each list and comprises a list head and an associated lock. ❑ vm_ops is a pointer to a collection of methods used to perform various standard operations on the region. struct vm_operations_struct { void (*open)(struct vm_area_struct * area); void (*close)(struct vm_area_struct * area); int (*fault)(struct vm_area_struct *vma, struct vm_fault *vmf); struct page * (*nopage)(struct vm_area_struct * area, unsigned long address, int *type); ... }; ❑ open and close are invoked when a region is created and deleted, respectively. They are not normally used and have null pointers. ❑ However, fault is very important. If a virtual page is not present in an address space, the automatically triggered page fault handler invokes this function to read the corresponding data into a physical page that is mapped into the user address space. ❑ nopage is the kernel’s old method to respond to page faults that is less flexible than fault. The element is still provided for compatibility reasons, but should not be used in new code. 301 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 302 Chapter 4: Virtual Process Memory ❑ vm_pgoffset specifies an offset for a file mapping when not all file contents are to be mapped (the offset is 0 if the whole file is mapped). The offset is not expressed in bytes but in multiples of PAGE_SIZE. On a system with pages of 4 KiB, an offset value of 10 equates to an actual byte offset of 40,960. This is reasonable because the kernel only supports mappings in whole-page units, and smaller values would make no sense. ❑ vm_file points to the file instance that describes a mapped file (it holds a null pointer if the object mapped is not a file). Chapter 8 discusses the contents of the file structure at length. ❑ Depending on mapping type, vm_private_data can be used to store private data that are not manipulated by the generic memory management routines. (The kernel ensures only that the element is initialized with a null pointer when a new region is created.) Currently, only a few sound and video drivers make use of this option. vm_flags stores flags to define the properties of a region. They are all declared as pre-processor constants in . ❑ VM_READ, VM_WRITE, VM_EXEC,andVM_SHARED specify whether page contents can be read, written, executed, or shared by several processes. ❑ VM_MAYREAD, VM_MAYWRITE, VM_MAYEXEC,andVM_MAYSHARE determine whether the VM_* flagsmaybeset.Thisisrequiredforthemprotect system call. ❑ VM_GROWSDOWN and VM_GROWSUP indicate whether a region can be extended downward or upward (to lower/higher virtual addresses). Because the heap grows from bottom to top, VM_GROWSUP is set in its region; VM_GROWSDOWN is set for the stack, which grows from top to bottom. ❑ VM_SEQ_READ is set if it is likely that the region will be read sequentially from start to end; VM_RAND_READ specifies that read access may be random. Both flags are intended as ‘‘prompts‘‘ for memory management and the block device layer to improve their optimizations (e.g., page readahead if access is primarily sequential. Chapter 8 takes a closer look at this technique). ❑ If VM_DONTCOPY is set, the relevant region is not copied when the fork system call is executed. ❑ VM_DONTEXPAND prohibits expansion of a region by the mremap system call. ❑ VM_HUGETLB is set if the region is based on huge pages as featured in some architectures. ❑ VM_ACCOUNT specifies whether the region is to be included in the calculations for the overcommit features. These features restrict memory allocations in various ways (refer to Section 4.5.3 for more details). 4.4.3 The Priority Search Tree Priority search trees are required to establish a connection between a region in a file and all virtual address spaces into which the region is mapped. To understand how this connection is established, we need to introduce some data structures of the kernel, which will be discussed in more detail and within a more general context in the following chapters. 302 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 303 Chapter 4: Virtual Process Memory Additional Data Structures Every open file (and every block device, because these can also be memory-mapped via device special files) is represented by an instance of struct file. This structure, in turn, contains a pointer to an address space object as represented by struct address_space.Thisobjectisthebasisoftheprioritysearchtree (prio tree) by which the connection between mapped intervals and the address spaces into which these are mapped is established. The definition of both structures is as follows (I only show the elements required for our purposes here): struct address_space { struct inode *host; /* owner: inode, block_device */ ... struct prio_tree_root i_mmap; /* tree of private and shared mappings */ struct list_head i_mmap_nonlinear;/*list VM_NONLINEAR mappings */ ... } struct file { ... struct address_space *f_mapping; ... } Additionally, each file and each block device are represented by an instance of struct inode. In contrast to struct file, which is the abstraction for a file opened by the open system call, the inode represents the object in the filesystem itself. struct inode { ... struct address_space *i_mapping; ... } Notice that only mapped file intervals are discussed below although, it is also possible to map different things, for instance, direct intervals in raw block devices, without a detour over filesystems. When a file is opened, the kernel sets file->f_mapping to inode->i_mapping. This allows multiple processes to access the same file without directly interfering with the other processes: inode is a file-specific data structure, while file is local to a given process. These data structures are connected with each other, and Figure 4-7 provides an overview about the situation in memory. Notice that the representation of the tree is only symbolic and does not reflect the actual, complicated tree layout. Given an instance of struct address_space, the kernel can infer the associated inode, which, in turn, allows for access to the backing store on which the file data are stored. Usually, the backing store will be a block device; the details are discussed in Chapter 9. Section 4.6 and Chapter 16 are devoted to discussing more about address spaces. Here it suffices to know that the address space is the base element of a 303 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 304 Chapter 4: Virtual Process Memory priority tree that contains all vm_area_struct instances describing the mapping of an interval of the file associated with the inode into some virtual address space. Since each instance of struct vm_area contains a pointer to the mm_struct of the process to which it belongs, the desired connection is set up! Note that vm_area_structs can also be associated with an address space via a doubly linked list headed by i_mmap_nonlinear.Thisisrequiredfornonlinear mappings, which I neglect for now. I will come back to them in Section 4.7.3, though. i_mapping struct file i_mapping i_mmap i_mmap_nonlinear host struct file i_mapping struct inode vm_mm vm_mm vm_mm mm_struct Backing device mm_struct mm_struct struct address_space Manage vm_area_structs associated with a file struct vm_area_struct Figure 4-7: Tracking the virtual address spaces into which a given interval of a file is mapped with the help of a priority tree. Recall that Figure 4-6 shows how vm_area_struct instances are organized in a linked list and a red-black tree. It is important to realize that these are the same vm_area_struct instances that are managed in the prio tree. While keeping vm_area_structs in two or more data structures at the same time is no problem for the kernel at all, it is nearly impossible to visualize. Therefore, keep in mind that a given instance of struct vm_area can be contained in two data structures: One establishes a connection between a region in the virtual address space of a process to the data in the underlying file, and one allows for finding all address spaces that map a given file interval. Representing Priority Trees Priority trees allow for management of the vm_area_struct instances that represent a particular interval of the given file. This requires that the data structure cannot only deal with overlapping, but also with identical file intervals. The situation is illustrated in Figure 4-8: Two processes map the region [7, 12] of a file into their virtual address space, while a third process maps the interval [10, 30]. Managing overlapping intervals is not much of a problem: The boundaries of the interval provide a unique index that allows for storing each interval in a unique tree node. I will not discuss in detail how this is implemented by the kernel because it rather similar to radix trees (see Appendix C for more details). It suffices to know that if intervals B, C,andD are completely contained in another interval A, then A will be the parent element of B, C,andD. However, what happens if multiple identical intervals must be included in the prio tree? Each prio tree node is represented by the raw_prio_tree_node instance, which is directly included in each 304 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 305 Chapter 4: Virtual Process Memory vm_ area_struct instance. Recall, however, that it is in a union with a vm_set. This allows for associating a list of vm_sets(andthusvm_area_structs) with a prio tree node. Figure 4-9 illustrates the situation in memory. Process 1 Process 2 Process 3 File 7101230 Figure 4-8: Multiple processes can map identical or overlapping regions of a file into their virtual address space. prio_tree_root vm_set vm_set vm_set vm_set raw_prio_ tree_node raw_prio_tree_node Figure 4-9: Interrelation of data structures in the management of shared identical mappings. When an interval is inserted into the prio tree, the kernel proceeds as follows: ❑ When the vm_area_struct instance is linked into the prio tree as a node, prio_tree_node is used to establish the necessary associations. To check whether there is a vm_area_struct in the tree, the kernel exploits the fact that the parent element of vm_set coincides with the last structure element of prio_tree_node — the data structures are coordinated accordingly. Since parent is not used within vm_set, the kernel can use parent != NULL to check whether the cur- rent vm_area_struct member is in a tree. The definition of prio_tree_node also ensures that the head element of vmset does not overlap with prio_tree_node so that both can be used together, although they are actually combined in a union. The kernel therefore uses vm_set.head to point to the first element on the list of vm_area_struct instances that belong to a shared mapping. ❑ If the above list of shared mappings contains a vm_area_struct, vm_set.list is used as the list head to list all regions affected. Section 4.5.3 discusses the technical details of how the kernel goes about inserting new regions. 305 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 306 Chapter 4: Virtual Process Memory 4.5 Operations on Regions The kernel provides various functions to manipulate the regions of a process. Creating and deleting regions (and finding a suitable memory location for a new region) are standard operations needed when setting up or removing a mapping. The kernel is also responsible for performing optimizations when managing the data structures, as shown in Figure 4-10. Existing region New region Region is being deleted Figure 4-10: Operations on regions. ❑ When a new region is added immediately before or after an existing region (and therefore also between two existing regions), the kernel merges the data structures involved into a single structure — but, of course, only if the access permissions for all the regions involved are identical and contiguous data are mapped from the same backing store. ❑ If a deletion is made at the start or end of a region, the existing data structure must be truncated accordingly. ❑ If a region between two other regions is deleted, the existing data structure is reduced in size, and a new data structure is created for the resultant new region. A further important standard operation is the search for a region associated with a specific virtual address in userspace. Before explaining the optimizations mentioned above, let’s discuss the helper function used to do this. 4.5.1 Associating Virtual Addresses with a Region By reference to a virtual address, find_vma finds the first region in user address space whose end is after the given address and therefore satisfies the addr < vm_area_struct->vm_end condition. As parameters, the function requires not only the virtual address (addr) but also a pointer to the mm_struct instance of the relevant process whose address space is to be scanned. struct vm_area_struct * find_vma(struct mm_struct * mm, unsigned long addr) { struct vm_area_struct *vma = NULL; if (mm) { /* Check the cache first. */ 306 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 307 Chapter 4: Virtual Process Memory /* (Cache hit rate is typically around 35%.) */ vma = mm->mmap_cache; if (!(vma && vma->vm_end > addr && vma->vm_start <= addr)) { struct rb_node * rb_node; rb_node = mm->mm_rb.rb_node; vma = NULL; while (rb_node) { struct vm_area_struct * vma_tmp; vma_tmp = rb_entry(rb_node, struct vm_area_struct, vm_rb); if (vma_tmp->vm_end > addr) { vma = vma_tmp; if (vma_tmp->vm_start <= addr) break; rb_node = rb_node->rb_left; } else rb_node = rb_node->rb_right; } if (vma) mm->mmap_cache = vma; } } return vma; } The kernel first checks whether the region last processed and now held in mm->mmap_cache contains the required address — that is, whether its end is after the required address and its start is before. If so, the kernel does not execute the if block and immediately returns the pointer to the region. If not, the red-black tree must be searched step by step. rb_node is the data structure used to represent each node in the tree. rb_entry enables the ‘‘useful data‘‘ (in this case, an instance of vm_area_struct) to be removed from the node. The root element of the tree is located in mm->mm_rb.rb_node. If the end address of the associated region is less than the required address and the start address is greater than the required address, the kernel has found the appropriate element and can exit the while loop to return a pointer to the vm_area_struct instance. Otherwise, the search is resumed at the ❑ left child if the end address of the current region is after the required address, or at the ❑ right child if the end address of the region is before the required address. As the root elements of the tree have null pointers as child elements, it is easy for the kernel to decide when to terminate the search and return a null pointer as an error message. If a suitable region is found, a pointer to it is stored in mmap_cache because there is a strong likelihood that the next find_vma call will search for a neighboring address in the same region. 307 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 308 Chapter 4: Virtual Process Memory find_vma_intersection is another helper function to establish whether an interval bounded by start_ addr and end_addr is fully within an existing region. It builds on find_vma and is easily implemented as follows: static inline struct vm_area_struct * find_vma_intersection(struct mm_struct * mm, unsigned long start_addr, unsigned long end_addr) { struct vm_area_struct * vma = find_vma(mm,start_addr); if (vma && end_addr <= vma->vm_start) vma = NULL; return vma; } 4.5.2 Merging Regions When a new region is added to the address space of a process, the kernel checks whether it can be merged with one or more existing regions as shown in Figure 4-10. vm_merge merges a new region with the surrounding regions if this is possible. It requires numerous parameters. mm/mmap.c struct vm_area_struct *vma_merge(struct mm_struct *mm, struct vm_area_struct *prev, unsigned long addr, unsigned long end, unsigned long vm_flags, struct anon_vma *anon_vma, struct file *file, pgoff_t pgoff, struct mempolicy *policy) { pgoff_t pglen = (end - addr) >> PAGE_SHIFT; struct vm_area_struct *area, *next; ... mm is the address space instance of the relevant process and prev the region immediately before the new region. rb_parent is the parent element of the region in the red-black search tree. addr, end,andvm_flags describe the start, end, and flags of the new region as their names suggest. If the region belongs to a file mapping, file contains a pointer to the file instance that identifies the file. pgoff specifies the offset of the mapping within the file data. Since policy is required on NUMA systems only, I won’t discuss it further. The technical details of implementation are very straightforward. A check is first made to ascertain whether the end address of the predecessor region corresponds to the start address of the new region. If so, the kernel must then check that the flags and the mapped file are identical for both regions, that the offsets of file mappings are such that a contiguous region results, that both regions do not con- tain anonymous mappings, and that both regions are mutually compatible.6 This is done using the 6The regions cannot be merged if two file mappings follow each other without a hole but map non-contiguous sections of the file. 308 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 309 Chapter 4: Virtual Process Memory can_vma_merge_after helper function. The work of merging a region with its predecessor region looks like this: mm/mmap.c if (prev && prev->vm_end == addr && can_vma_merge_after(prev, vm_flags, anon_vma, file, pgoff)) { ... If it can, the kernel then checks whether the successor region can and must be merged. mm/mmap.c /* * OK, it can. Can we now merge in the successor as well? */ if (next && end == next->vm_start && can_vma_merge_before(next, vm_flags, anon_vma, file, pgoff+pglen) && is_mergeable_anon_vma(prev->anon_vma, next->anon_vma)) { vma_adjust(prev, prev->vm_start, next->vm_end, prev->vm_pgoff, NULL); } else vma_adjust(prev, prev->vm_start, end, prev->vm_pgoff, NULL); return prev; } The first difference as compared to the previous case is that can_vma_merge_before is used instead of can_vma_merge_after to check whether the two regions can be merged. If both the predecessor and the successor region can be merged with the current region, it must also be ensured that the anonymous mappings of the predecessor can be merged with those of the successor before a single region consisting of all three regions can be created. In both cases, the helper function vma_adjust is invoked to perform final merging; it appropriately modifies all data structures involved — the priority tree and the vm_area_struct instances — as well as deallocating the instances of these structures that are no longer needed. 4.5.3 Inserting Regions insert_vm_struct is the standard function used by the kernel to insert new regions. The actual work is delegated to two helper functions, as the code flow diagram in Figure 4-11 shows. find_vma_prepare is first invoked to obtain the information listed below by reference to the start address of the new region and of the address space involved (mm_struct). ❑ The vm_area_struct instance of the preceding address space. ❑ The parent node (in the red-black tree) in which the node for the new region is held. ❑ The leaf node (of the red-black tree) that contains the region itself. 309 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 310 Chapter 4: Virtual Process Memory insert_vm_struct find_vma_prepare vma_link _ _vma_link _ _vma_link_list _ _vma_link_rb _ _anon_vma_link _ _vma_link_file Figure 4-11: Code flow diagram for insert_vm_struct. It’s common knowledge that C permits functions to return just one variable — consequently, the above function returns only a pointer to the successor region as its direct result; the remaining information is supplied by means of pointer arguments. The information found is sufficient to incorporate the new region into the existing data structures of the process using vma_link. After some preparatory work, the function delegates the real work to insert_vm_struct, which performs three insert operations as the code flow diagram shows. ❑ __vma_link_list puts the new region on the linear list of regions of the process; only the prede- cessor and successor region found using find_vma_prepare are needed to do this.7 ❑ __vma_link_rb links the new region into the data structures of the red-black tree, as the name suggests. ❑ __anon_vma_link adds the vm_area_struct instance to the linked list of anonymous mappings discussed above. Finally, __vma_link_file links the relevant address_space and the mapping in the case of file mappings and also adds the region to the prio tree using vma_prio_tree_insert, which handles multiple identical regions as described above. 4.5.4 Creating Regions Before a new memory region can be inserted into the data structures, the kernel must establish where there is enough free space in virtual address space for a region of a given size. This job is assigned to the get_unmapped_area helper function. 7If there is no predecessor region because the new region is the new start region or because no regions are defined for the address space, the information in the red-black tree is used to set the pointers correctly. 310 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 311 Chapter 4: Virtual Process Memory mm/mmap.c unsigned long get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) The arguments are self-explanatory. The implementation of the function is of no further interest as the actual work is delegated to the architecture-specific helper function stored in the mm_struct instance of the current process.8 Recall from Section 4.2 that different mapping functions are used depending on the layout of the virtual process address space. Here I consider the standard function arch_get_unmapped_area that is employed on most systems. arch_get_unmapped_area first has to check whether the MAP_FIXED flag is set, indicating that the map- ping is to be created at a fixed address. If so, the kernel ensures only that the address satisfies alignment requirements (page-by-page) and that the interval is fully within the available address space. If no desired area was specified, the kernel tries to find a suitable section in the virtual memory area of the process by invoking arch_get_unmapped_area. If a particular preferred (as opposed to a fixed) address is specified, the kernel checks whether the region overlaps with an existing region. If not, the address can be returned as the target. mm/mmap.c unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { struct mm_struct *mm = current->mm; ... if (addr) { addr = PAGE_ALIGN(addr); vma = find_vma(mm, addr); if (TASK_SIZE - len >= addr && (!vma || addr + len <= vma->vm_start)) return addr; } ... Otherwise, the kernel must try to find a free area of the right size by iterating over the available regions of the process. In doing so, it checks whether a cached area from previous scans could be used. mm/mmap.c if (len > mm->cached_hole_size) { start_addr = addr = mm->free_area_cache; } else { start_addr = addr = TASK_UNMAPPED_BASE; mm->cached_hole_size = 0; } ... 8Files can also be equipped with a special-purpose mapping function. This is, for instance, used by the frame-buffer code to allow direct manipulation of the video memory when a frame-buffer device file is mapped into memory. However, because the kernel gen- erally uses the standard implementation, I won’t bother to discuss other more specific routines. 311 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 312 Chapter 4: Virtual Process Memory The actual iteration begins either at the address of the last ‘‘hole‘‘ in the virtual address space or at the global start address TASK_UNMAPPED_BASE. mm/mmap.c full_search: for (vma = find_vma(mm, addr); ; vma = vma->vm_next) { /* At this point: (!vma || addr < vma->vm_end). */ if (TASK_SIZE - len < addr) { /* * Start a new search - just in case we missed * some holes. */ if (start_addr != TASK_UNMAPPED_BASE) { addr = TASK_UNMAPPED_BASE; start_addr = addr; mm->cached_hole_size = 0; goto full_search; } return -ENOMEM; } if (!vma || addr + len <= vma->vm_start) { /* * Remember the place where we stopped the search: */ mm->free_area_cache = addr + len; return addr; } if (addr + mm->cached_hole_size < vma->vm_start) mm->cached_hole_size = vma->vm_start - addr; addr = vma->vm_end; } } If the search continues to the end of the user address space (TASK_SIZE) and no suitable area is found, the kernel returns an -ENOMEM error that must be forwarded to userspace for processing by the relevant appli- cation, as it indicates that insufficient virtual address space memory is available to satisfy the request. If memory is found, its virtual start address is returned. The version for top-down allocation, arch_get_unmapped_area_topdown, progresses similarly, but the search direction is, of course, reversed. We need not bother with the details of implementation here. 4.6 Address Spaces Memory mappings of files can be regarded as mappings between two different address spaces to simplify the work of (system) programmers. One address space is the virtual memory address space of the user process, the other is the address space spanned by the filesystem. When the kernel creates a mapping, it must create a link between the address spaces to support com- munication between the two — in the form of read and write requests. The vm_operations_struct structure with which we are familiar from Section 4.4.2 is first used to do this. It provides an operation to read pages not yet in physical memory although their contents have already been mapped there. 312 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 313 Chapter 4: Virtual Process Memory However, the operation has no information on the mapping type or on its properties. As there are numer- ous kinds of file mappings (regular files on different filesystem types, device files, etc.), more information is required. In fact, the kernel needs a more detailed description of the address space of the data source. The address_space structure mentioned briefly above is defined for this purpose and contains additional information on a mapping. Recall that the connection between files, address spaces, and inodes has been shown in Figure 4-7. Some of the data structures involved are explained in future chapters, and thus their relationships are not dealt with here; let us simply state that each file mapping has an associated instance of address_space. Neither is the exact definition of struct address_space relevant at this point; it is discussed in more detail in Chapter 16. Here it is sufficient to know that each address space has a set of address space operations held as function pointers in the structure shown below (only the most important entries are reproduced). struct address_space_operations { int (*writepage)(struct page *page, struct writeback_control *wbc); int (*readpage)(struct file *, struct page *); ... /* Write back some dirty pages from this mapping. */ int (*writepages)(struct address_space *, struct writeback_control *); /* Set a page dirty. Return true if this dirtied it */ int (*set_page_dirty)(struct page *page); int (*readpages)(struct file *filp, struct address_space *mapping, struct list_head *pages, unsigned nr_pages); ... }; A detailed description of the structure can also be found in Chapter 16. ❑ readpage reads a single page from the underlying block medium into RAM memory; readpages performs the same task for several pages at once. ❑ writepage writes the contents of a page from RAM memory back to the corresponding location on a block device to permanently save changes. ❑ set_page_dirty indicates that the contents of a page have been changed and no longer match the original contents on the block device. How is the link between vm_operations_struct and address_space established? There is no static link to assign an instance of each structure to the other structure. Nevertheless, both are linked by the standard implementations that the kernel provides for vm_operations_struct and that are used by almost all filesystems. mm/filemap.c struct vm_operations_struct generic_file_vm_ops = { .fault = filemap_fault, }; 313 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 314 Chapter 4: Virtual Process Memory The implementation of filemap_fault uses the readpage method of the underlying mapping and there- fore adopts the above address_space concept, as you will see in the concept description in Chapter 8. 4.7 Memory Mappings Now that we are familiar with the data structures and address space operations related to memory mappings, we move on in this section to examine the interaction between the kernel and the applications when mappings are created. As we know, the C standard library features the mmap function to install mappings. Two system calls — mmap and mmap2 — are provided on the kernel side. Some architectures implement both versions [e.g., IA-64, and Sparc(64)], others only the first (AMD64) or only the second (IA-32). Both have the same set of parameters. asmlinkage unsigned long sys_mmap{2}(unsigned long addr, unsigned long len, unsigned long prot, unsigned long flags, unsigned long fd, unsigned long off) Both calls create a mapping of length len at position pos in the virtual user address space whose access permissions are defined in prot. flags is a flag set used to set a number of parameters. The relevant file is identified by means of its file descriptor in fd. The difference between mmap and mmap2 lies in the meaning of the offset (off). In both calls, it indicates the point in the file at which mapping is to begin. For mmap, the position is specified in bytes, whereas the unit used by mmap2 is pages (PAGE_SIZE) — this enables file sections to be mapped even if the file is larger than the address space available. Typically, the C standard library provides only a single function for the creation of memory mappings by applications. This function call is then translated internally to the system call appropriate to the archi- tecture. The munmap system call is invoked to remove a mapping. There is no need for a munmap2 system call because no file offset is required — just the virtual address of the mapping. 4.7.1 Creating Mappings The call syntax for mmap and mmap2 has already been introduced above, so I only need briefly list the most important flags that can be set: ❑ MAP_FIXED specifies that no other address than the one given may be used for the mapping. If this flag is not set, the kernel is free to change the desired address if, for example, a mapping already resides there (the existing mapping would otherwise be overwritten). ❑ MAP_SHARED must be used when an object (usually a file) is to be shared between several pro- cesses. ❑ MAP_PRIVATE creates a private mapping that is separated from the contents of the source — write operations on the mapped region have no effect on the data in the file. ❑ MAP_ANONYMOUS creates an anonymous mapping that is not associated with any data source — the fd and off parameters are ignored. This type of mapping can be used to allocate malloc-like memory for applications. 314 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 315 Chapter 4: Virtual Process Memory A combination of PROT_EXEC, PROT_READ, PROT_WRITE,andPROT_NONE values can be used to define access permission in prot. Not all combinations are implemented for all processors, with the result that the region may be granted more rights than those specified. Although the kernel does its best to set the desired mode, it can only guarantee that the access permissions set are not more restrictive than those specified. For the sake of simplicity, the description below deals only with sys_mmap2 (sys_mmap behaves in a very similar way on most other architectures: all arrive in the do_mmap_pgoff function discussed below). In line with the convention discussed in Chapter 13, the function serves as the entry point for the mmap2 system call and immediately delegates work to do_mmap2. There the kernel references the file descriptor to find the file instance with all the characteristic data of the file being processed (Chapter 8 examines this data structure more closely). The remaining work is delegated to do_mmap_pgoff. do_mmap_pgoff is an architecture-independent function defined in mm/mmap.c. Figure 4-12 shows the asso- ciated code flow diagram. Already existing region? Compute flags do_mmap_pgoff get_unmapped_area mmap_region find_vma_prepare do_munmap file->f_op->mmap make_pages_present Check memory limits Create a new vm_area_struct VM_LOCKED set? Return start address of mapping Figure 4-12: Code flow diagram for do_mmap_pgoff. do_mmap_pgoff used to be one of the longest functions in the kernel. It is now effectively split into two parts, which are, however, still rather voluminous. One part has to thoroughly check the parameters of the user application, and the second part has to take a very large number of special situations and subtleties into consideration. As the latter make no valuable contribution to a general understanding of the mechanism involved, we look only at a representative standard situation — mapping of a regular file with MAP_SHARED — to avoid bloating our description, and the code flow diagram also applies just for this case. 315 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 316 Chapter 4: Virtual Process Memory The get_unmapped_area function described in Section 4.5.4 is first invoked to find a suitable area for the mapping in the virtual address space. Recall that the application may specify a fixed address for the mapping, suggest an address, or leave the choice of address to the kernel. calc_vm_prot_bits and calc_vm_flag_bits combine the flags and access permission constants speci- fied in the system call in a joint flag set that is easier to handle in the subsequent operations (the MAP_ and PROT_ flags are ‘‘translated‘‘into flags with the prefix VM_). mm/mmap.c vm_flags = calc_vm_prot_bits(prot) | calc_vm_flag_bits(flags) | mm->def_flags | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC; What is most interesting is that the kernel includes the value of def_flags in the flag set after removing it from the mm_struct instance of the currently running process. def_flags has the value 0 or VM_LOCK.The former brings about no change to the resulting flag set, whereas VM_LOCK means that pages subsequently mapped in cannot be swapped out (the implementation of swapping is discussed in Chapter 18). To set the value of def_flags, the process must issue the mlockall system call, which uses the mechanism described above to prevent all future mappings from being swapped out, even if this was not requested explicitly by means of the VM_LOCK flag at creation time. After the arguments have been checked and all required flags have been set up, the remaining work is delegated to mmap_region.Thefind_vma_prepare function with which we are familiar from Section 4.5.3 is invoked to find the vm_area_struct instances of the predecessor and successor areas and the data for the entry in the red-black tree. If a mapping already exists at the specified mapping point, it is removed by means of do_munmap (as described in the section below). vm_enough_memory is invoked9 if either the MAP_NORESERVE flag is not set or the value of the kernel parameter sysctl_overcommit_memory10 is set to OVERCOMMIT_NEVER, that is, when overcommiting is not allowed. The function chooses whether to allocate the memory needed for the operation. If it selects against, the system call terminates with -ENOMEM. 9Using security_vm_enough_memory, which calls __vm_enough_memory over varying paths depending on the security framework in use. 10sysctl_overcommit_memory can be set with the help of the /proc/sys/vm/overcommit_memory. Currently there are three overcommit options. 1 allows an application to allocate as much memory as it wants, even more than is permitted by the address space of the system. 0 means that heuristic overcommitting is applied with the result that the number of usable pages is determined by adding together the pages in the page cache, the pages in the swap area, and the unused page frames; requests for allocation of a smaller number of pages are permitted. 2 stands for the strictest mode, known as strict overcommitting,inwhichthe permitted number of pages that can be allocated is calculated as follows: allowed = (totalram_pages - hugetlb) * sysctl_overcommit_ratio / 100; allowed += total_swap_pages; Here sysctl_overcommit_ratio is a configurable kernel parameter that is usually set to 50. If the total number of pages used exceeds this value, the kernel refuses to perform further allocations. Why does it make sense to allow an application to allocate more pages than can ever be handled in principle? This is sometimes required for scientific applications. Some tend to allocate huge amounts of memory without actually requiring it — but, in the opinion of the application authors, it seems good to have it just in case. If the memory will, indeed, never be used, no physical page frames will ever be allocated, and no problem arises. Such a programming style is clearly bad practice, but unfortunately this is often no criterion for the value of software. Writing clean code is usually not rewarding in the scientific community outside computer science. There is only immediate interest that a program works for a given configuration, while efforts to make programs future-proof or portable do not seem to provide immediate benefits and are therefore often not valued at all. 316 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 317 Chapter 4: Virtual Process Memory Once the kernel has granted the desired memory, the following steps are taken: 1. Allocation and initialization of a new vm_area_struct instance that is inserted in the list/tree data structures of the process. 2. Creation of the mapping with the file-specific function file->f_op->mmap. Most filesystems use generic_file_mmap for this purpose; all it does is set the vm_ops element of the mapping to generic_file_vm_ops. vma->vm_ops = &generic_file_vm_ops; The definition of generic_file_vm_ops is given in Section 4.5.3. Its key element is filemap_ fault, which is invoked when an application accesses the mapped area but the correspond- ing data are not yet in RAM memory. filemap_fault enlists the help of low-level routines of the underlying filesystem to fetch the desired data and — transparently to the application — read them into RAM memory. In other words, the mapped data are not read in immediately when the mapping is created but only when they are actually needed. Chapter 8 takes a closer look at the implementation of filemap_fault. If VM_LOCKED is set — either explicitly with system call flags or implicitly by means of the mlockall mechanism — the kernel invokes make_pages_present to successively scan the pages of the mapping and to trigger a page fault for each so that their data are read in. Of course, this means that the perfor- mance gain of deferred reading is lost, but the kernel makes sure that the pages are always in memory after a mapping has been created — after all, the VM_LOCKED flag prevents them from being swapped out, so they must be first in. The start address of the new mapping is then returned to conclude the system call. do_mmap_pgoff performs several checks (not described in detail here) at various points in addition to the actions described above. If one of the checks fails, the operation is terminated, and the system call returns to userspace with an error code. ❑ Accounting — The kernel keeps statistics on the number of pages a process uses for mappings. As it is possible to limit process resources, the kernel must always ensure that the permitted value is not exceeded. There is also a maximum number of mappings per process. ❑ Extensive security and plausibility checks must be carried out to prevent the applications from setting invalid parameters or parameters that could threaten system stability. For example, no mappings may be created that are larger than the virtual address space or extend beyond the boundaries of virtual address space. 4.7.2 Removing Mappings The munmap system call, which requires two parameters — the start address and length of the area to be unmapped, must be used to remove an existing mapping from virtual address space. sys_munmap is the entry point for the system call; it delegates its work in the usual way to the do_munmap function defined in mm_mmap.c. (Further implementation information is shown in the associated code flow diagram in Figure 4-13.) 317 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 318 Chapter 4: Virtual Process Memory Split region? do_munmap find_vma_prev find_vma split_vma split_vma detach_vmas_to_be_unmapped unmap_region remove_vma_list Another split necessary? Figure 4-13: Code flow diagram for do_munmap. Thekernelmustfirstinvokefind_vma_prev to try to find the vm_area_struct instance for the region to be unmapped. This function operates in exactly the same way as find_vma discussed in Section 4.5.1, but it not only finds the vm_area_struct matching the address, but also returns a pointer to the predecessor region. If the start address of the area to be unmapped is not precisely at the start of the region found by find_vma_prev, only part but not the whole of the mapping is unmapped. Before the kernel does this, it must first divide the existing mapping into several parts. The front part of the mapping that is not to be unmapped is first split off by split_vma. This is a helper function I won’t bother discussing because all it does is perform standard operations on familiar data structures. It simply allocates a new instance of vm_area_struct, fills it with the data of the old region, and adjusts the boundaries. The new region is inserted into the data structures of the process. The same procedure is repeated for the rear part of the mapping if the old region is not to be unmapped right up to its end. The kernel then invokes detach_vmas_to_be_unmapped to draw up a list of all regions to be unmapped. Because an unmapping operation can involve any area of address space, it may well be that several suc- cessive regions are affected. The kernel has ensured that only complete regions are affected by splitting the areas at the start and the end. detach_vmas_to_be_unmapped iterates over the linear list of vm_area_struct instances until the whole area is covered. The vm_next element of the structures is briefly ‘‘misused‘‘to link the regions to be unmapped with each other. The function also sets the mmap cache to NULL, thus invalidating it. Two final steps follow. First, unmap_region is invoked to remove all entries from the page tables associ- ated with the mapping. When this is done, the kernel must also make sure that the relevant entries are removed from the translation lookaside buffer or are rendered invalid. Second, the space occupied by the vm_area_struct instances is freed with remove_vma_list to finally remove the mapping from the kernel. 318 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 319 Chapter 4: Virtual Process Memory 4.7.3 Nonlinear Mappings As just demonstrated, normal mappings map a continuous section from a file into a likewise continuous section of virtual memory. If various parts of a file are mapped in a different sequence into an otherwise contiguous area of virtual memory, it is generally necessary to use several mappings, which is more costly in terms of resources (particularly in vm_area_structs). A simpler way of achieving the same result11 is to use nonlinear mappings as introduced during the development of 2.5. The kernel features a separate system call specifically for this purpose. mm/fremap.c long sys_remap_file_pages(unsigned long start, unsigned long size, unsigned long __prot, unsigned long pgoff, unsigned long flags) The system call allows for rearranging pages in a mapping such that the order in memory is not identical with the order in the file. This is achieved without moving the memory contents around, but is instead performed by manipulating the page tables of the process. sys_remap_file_pages enables an existing mapping at position pgoff and with a size of size to be moved to a new position in virtual memory. start identifies the mapping whose pages are to be moved, and thus must fall into the address of an already existing mapping. It also specifies the new position into which the pages identified by pgoff and size are supposed to be moved. If a nonlinear mapping is swapped out, the kernel must ensure that the offsets are still present when the mapping is swapped back in again. The information needed to do this is stored in the page table entries of the pages swapped out and must be referenced when they are swapped back in, as we shall see below. But how is the information encoded? Two components are used: 1. The vm_area_struct instances of all installed nonlinear mappings are stored in a list headed by the i_mmap_nonlinear element of struct address_space. The individual vm_area_ structs on the list can employ shared.vm_set.list as list element because a nonlinear VMA will not be present on the standard prio tree. 2. The page table entries for the region in question are populated with special entries. These are constructed such that they look like PTEs of pages that are not present, but contain additional information identifying them as PTEs for nonlinear mappings. When the page described by the PTE is accessed, a page fault is generated, and the correct page can be read in. Naturally, page table entries cannot be modified at will, but must adhere to conventions imposed by the underlying architecture. To create nonlinear PTEs, help by the architecture-specific code is required, and three functions must be defined: 1. pgoff_to_pte takes a file offset encoded as a page number and encodes it into a format that can be stored in a page table. 2. pte_to_pgoff can decode a file offset encoded in a page table. 11Even though there appears to be very little need for this, there are various large databases that use operations of this kind to rep- resent data transactions. 319 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 320 Chapter 4: Virtual Process Memory 3. pte_file(pte) checks if a given page table entry is used to represent a nonlinear mapping. This especially allows for distinguishing a page table entry of a nonlinear mapping from a page table entry for a regular swapped-out page when a page fault occurs. The pre-processor constant PTE_FILE_MAX_BITS denotes how many bits of a page table entry can be used to store a file offset. Since this constant will usually be smaller than the word size of the processor because some status bits in the PTE are required by the architecture and to distinguish it from swap-PTEs, the range of a file that can be remapped is, in general, smaller than the maximally possible file size. Since the layout of non-present page table entries is not plagued by any historical oddities on IA-64, the way nonlinear PTEs are implemented is particularly clean, so I present it as an example, which is illustrated in Figure 4-14. include/asm-ia64/pgtable.h #define PTE_FILE_MAX_BITS 61 #define pte_to_pgoff(pte) ((pte_val(pte) << 1) >> 3) #define pgoff_to_pte(off) ((pte_t) { ((off) << 2) | _PAGE_FILE }) _PAGE_PRESENT_PAGE_PROTNONE _PAGE_FILE 01 63 Nonlinear page offset Figure 4-14: Representing nonlinear mappings in page table entries on IA-64 systems. Swap identifiers are 64 bits long. Bit 0 must be zero because the page is not present, and bit 1 represents _PAGE_FILE to indicate that the entry belongs to a nonlinear mapping in contrast to a swap specifier. The last bit, that is, 63, is reserved for the _PAGE_PROTNONE bit.12 Consequently,thisleaves61bitsrawcapacity to represent the nonlinear page offset. pte_to_pgoff first extracts the value stored in the page table entry with pte_val as provided by the architecture-specific code. Performing one left-shift and two right-shifts is a simple method to extract the bits at position [2, 62]. When a PTE representing a nonlinear mapping is constructed, the kernel needs to shift the offset into the bit range starting at bit 2, and must additionally ensure that _PTE_FILE is set to identify it as a nonlinear mapping in contrast to a regular swapped-out identifier. The essential steps of sys_remap_file_pages are summarized in the code flow diagram in Figure 4-15. 12A page with this bit set was marked as completely inaccessible by the mmap system call. While such pages do not need to be backed by a physical page frame (they are not accessible, so what should be read from or written to the page?), the kernel never- theless has to mark somehow that they must not be accessed, and the aforementioned bit provides this capability. 320 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 321 Chapter 4: Virtual Process Memory VM_NONLINEAR notset? Check flags and sizes vma_prio_tree_remove vma_nonlinear_insert populate_range make_pages_present sys_remap_file_pages Set VM_NONLINEAR MAP_NONBLOCK not set? Figure 4-15: Code flow diagram for sys_remap_file_pages. After all flags have been checked and the kernel has ensured that the range to be remapped is valid, the vm_area_struct instance of the target region is selected by find_vma. If the destination has not been nonlinearly remapped before, the flag VM_NONLINEAR is not set in vm_area_struct->vm_flags.Inthis case, the linear mapping has to be removed from the prio tree with vma_prio_tree_remove, and it is inserted into the list of nonlinear mappings using vma_nonlinear_insert. The crucial step is to install the modified page table entries. The auxiliary routine populate_range is responsible for this: mm/fremap.c static int populate_range(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long size, pgoff_t pgoff) { int err; ... The mapping is described by vma. The region starting currently at page offset pgoff with length length is to be remapped to address addr. Since this can involve multiple pages, the kernel needs to iterate over all of them and install new page table entries with install_file_pte: mm/fremap.c do { err = install_file_pte(mm, vma, addr, pgoff, vma->vm_page_prot); if (err) return err; size -= PAGE_SIZE; addr += PAGE_SIZE; pgoff++; } while (size); return 0; } 321 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 322 Chapter 4: Virtual Process Memory install_file_pte first removes any existing page table entry with zap_file_pte and then constructs a new entry using the helper function pgoff_to_pte, which encodes a given file offset into a format suitable for a PTE: mm/fremap.c static int install_file_pte(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long pgoff, pgprot_t prot) { pte_t *pte; ... if (!pte_none(*pte)) zap_pte(mm, vma, addr, pte); set_pte_at(mm, addr, pte, pgoff_to_pte(pgoff)); ... } The final step in sys_remap_file_pages is to read in the pages of the mapping if this is desired (it can be prevented by setting the flag MAP_NONBLOCK). This is done using make_present_pages, which acts as if a page fault would have occurred for each single page in the mapping, and triggers reading the data from the underlying block device. 4.8 Reverse Mapping The data structures already discussed enable the kernel to establish a link between a virtual and a physi- cal address (via the page tables) and between a memory region of a process and its virtual page addresses. What is still missing is a link between a physical page and the processes to which the page belongs (or, to be more accurate, to the page table entries of all processes that use the page). This is the very link that is needed when swapping pages out (see Chapter 18) in order to update all processes that use the page because the fact that the page has been swapped out must be noted in their page tables. In this context, it is necessary to distinguish between two similar terms: 1. When a page is mapped, it is associated with a process but need not necessarily be in active use. 2. The number of references to a page indicates how actively the page is used. In order to deter- mine this number, the kernel must first establish a link between a page and all its users and must then resort to a few tricks to find out how actively the page is used. The first task is therefore to create a link between a page and all points at which it is mapped. To do this, the kernel uses a few additional data structures and functions and adopts a reverse mapping approach.13 All mapping actions described above are concerned only with virtual pages, and there was therefore no need (and no way) to create reverse mappings. The discussion of how the kernel handles page faults and 13Reverse mappings were first introduced during the development of kernel 2.5. They were available as separate patches for 2.4 but had never been included in the standard sources. Swapping-out of shared pages is much more complicated and inefficient without this mechanism because the shared page had to be kept in a special cache until the kernel had chosen separately (and independently) to swap the page out for all processes involved. The implementation of the reverse mapping algorithm was also heavily revised dur- ing the development of kernel 2.6. 322 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 323 Chapter 4: Virtual Process Memory assigns physical pages to hold mapping data in Section 4.10 notes that there is then a need for reverse mapping. 4.8.1 Data Structures The kernel uses lean data structures to minimize management overhead for reverse mappings. The page structure (discussed in Section 3.2.2) contains a single element to implement reverse mapping. mm.h struct page { .... atomic_t _mapcount; /* Count of ptes mapped in mms, * to show when page is mapped * & limit reverse map searches. */ ... }; _mapcount indicates at how many points the page is shared. The original value of the counter is −1.Itis assigned the value 0 when the page is inserted in the reverse mapping data structures and is incremented by 1 for each additional user. This enables the kernel to check quickly how many users are using the page in addition to the owner. Obviously, this isn’t much help because the purpose of reverse mapping is to find all points at which the physical page is used by reference to a given page instance. Consequently, two other data structures have aroletoplay: 1. The priority search tree in which each region belonging to a non-anonymous mapping is embedded 2. The linked lists of anonymous areas that lead back to the same pages in memory The elements needed to generate both data structures are integrated in vm_area_struct — these are the shared union as well as anon_vma_node and anon_vma. To refresh the reader’s memory, I reproduce the corresponding section from vm_area_struct below. mm.h struct vm_area_struct { ... /* * For areas with an address space and backing store, * linkage into the address_space->i_mmap prio tree, or * linkage to the list of like vmas hanging off its node, or * linkage of vma in the address_space->i_mmap_nonlinear list. */ union { struct { struct list_head list; void *parent; /* aligns with prio_tree_node parent */ struct vm_area_struct *head; } vm_set; struct raw_prio_tree_node prio_tree_node; 323 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 324 Chapter 4: Virtual Process Memory } shared; /* * A file’s MAP_PRIVATE vma can be in both i_mmap tree and anon_vma * list, after a COW of one of the file pages. A MAP_SHARED vma * can only be in the i_mmap tree. An anonymous MAP_PRIVATE, stack * or brk vma (with NULL file) can only be in an anon_vma list. */ struct list_head anon_vma_node; /* Serialized by anon_vma->lock */ struct anon_vma *anon_vma; /* Serialized by page_table_lock */ ... } The trick employed by the kernel when implementing reverse mapping is not to store a direct link between a page and the associated users but only the association between a page and the region in which the page is located. All other regions in which the page is included (and therefore all users) can be found by means of the data structures just mentioned. This method is also known as object-based reverse mapping because no direct link between page and user is stored; instead, a further object (the regions in which the page is located) is interposed between the two. 4.8.2 Creating a Reverse Mapping When a reverse mapping is created, it is necessary to distinguish between two alternatives — anonymous pages and pages with file-based mappings. This is understandable because the data structures used to manage both alternatives also differ. The information below only covers working with page instances to be inserted into the reverse mapping scheme. Other parts of the kernel are responsible for adding the relevant vm_area_structs to the data structures discussed above (priority tree and anon list); for example, by invoking vma_prio_tree_insert that is used (directly or indirectly) at several places in the kernel. Anonymous Pages There are two ways of inserting an anonymous page into the reverse mapping data structures. page_ add_new_anon_rmap must be invoked for new anonymous pages. page_add_anon_rmap is the right option for pages that are already reference-counted. The only difference between these alternatives is that the former sets the mapping counter page->_mapcount to 0 (reminder: the initial value of _mapcount is 0 for newly initialized pages), and the latter increments the counter by 1. Both functions then merge into __page_set_anon_rmap. mm/rmap.c void __page_set_anon_rmap(struct page *page, struct vm_area_struct *vma, unsigned long address) { struct anon_vma *anon_vma = vma->anon_vma; anon_vma = (void *) anon_vma + PAGE_MAPPING_ANON; page->mapping = (struct address_space *) anon_vma; page->index = linear_page_index(vma, address); } 324 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 325 Chapter 4: Virtual Process Memory The address of the anon_vma list head is stored in the mapping element of the page instance after PAGE_ MAPPING_ANON has been added to the pointer. This enables the kernel to distinguish between anony- mous pages and pages with a regular mapping by checking whether the least significant bit is 0 (if PAGE_MAPPING_ANON is not set) or 1 (if PAGE_MAPPING_ANON is set) as discussed above. Recall that this trick is valid because the lowest-order bit of a page pointer is guaranteed to be zero because of alignment requirements. Pages with a File-Based Mapping Work is even simpler for pages of this type, as the following code excerpt shows: mm/rmap.c void page_add_file_rmap(struct page *page) { if (atomic_inc_and_test(&page->_mapcount)) __inc_zone_page_state(page, NR_FILE_MAPPED); } Basically, all that needs to be done is to increment the _mapcount variable atomically and update the per-zone statistics. 4.8.3 Using Reverse Mapping The real benefits of reverse mapping do not become clear until Chapter 18, which examines the imple- mentation of swapping. There we will see that the kernel defines the try_to_unmap function, which is invoked to delete a specific physical page from the page tables of all processes by which the page is used. It is apparent that this is only possible with the data structures just described. Nevertheless, the implementation is influenced by many details of the swap layer, and this is why I won’t go into how try_to_unmap worksatthispoint. page_referenced is an important function that puts the data structures of the reverse mapping scheme to good use. It counts the number of processes that have actively used a shared page recently by accessing it — this is different from the number of regions into which the page is mapped. Whereas the second quantity is mostly static, the first changes rapidly if the page is in active use. The function is a multiplexer that invokes page_referenced_anon for anonymous pages or page_ referenced_file for pages from a file-based mapping. Both try to establish at how many places a page is used, but each adopts a different approach owing to the different underlying data structures. Let’s first look at the version for anonymous pages. We first need the page_lock_anon_vma helper function to find the associated list of regions by reference to a specific page instance (by reading the information discussed in the previous section from the data structure). static struct anon_vma *page_lock_anon_vma(struct page *page) { struct anon_vma *anon_vma = NULL; unsigned long anon_mapping; anon_mapping = (unsigned long) page->mapping; if (!(anon_mapping & PAGE_MAPPING_ANON)) goto out; 325 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 326 Chapter 4: Virtual Process Memory if (!page_mapped(page)) goto out; anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON); return anon_vma; } Oncethecodehasensuredthatthepage->mapping pointer actually points to an anon_vma instance using the by-now-familiar trick (the least significant bit of the pointer must be set), page_mapped checks whether the page has been mapped at all (page->_mapcount must then be greater than or equal to 0). If so, the function returns a pointer to the anon_vma instance associated with the page. page_referenced_anon makes use of this knowledge as follows: mm/rmap.c static int page_referenced_anon(struct page *page) { unsigned int mapcount; struct anon_vma *anon_vma; struct vm_area_struct *vma; int referenced = 0; anon_vma = page_lock_anon_vma(page); if (!anon_vma) return referenced; mapcount = page_mapcount(page); list_for_each_entry(vma, &anon_vma->head, anon_vma_node) { referenced += page_referenced_one(page, vma, &mapcount); if (!mapcount) break; } return referenced; } Once the matching anon_vma instance has been found, the kernel iterates over all regions in the list and invokes page_referenced_one for each one to return the number of places at which the page is used (some corrections are required when the system is swapping pages in and out, but these are not of interest here and are discussed in Section 18.7). The results are added together for all pages before the total is returned.14 page_referenced_one performs its task in two steps: 1. It finds the page table entry that points to the page. This is possible because not only the page instance but also the associated vm_area_struct is passed to page_referenced_one. The position in virtual address space at which the page is mapped can be determined from the latter variable. 14The kernel terminates its work when the number of references reaches the number of mappings held in mapcount as it makes no sense to continue searching. page_referenced_one automatically decrements the mapcount counter passed for each referenced page. 326 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 327 Chapter 4: Virtual Process Memory 2. It checks whether the _PAGE_ACCESSED bit is set in the page table entry and then deletes the bit. This flag is set on each access to the page by the hardware (with the additional support of the kernel if required by the particular architecture). The reference counter is incremented by 1 if the bit is set; otherwise, it is left unchanged. As a result, frequently used pages have a high number of references, and the opposite is true for rarely used pages. The kernel is therefore able to decide immediately whether a page is important based on the number of references. The approach adopted for checking the number of references for pages with file-based mapping is similar. mm/rmap.c static int page_referenced_file(struct page *page) { ... mapcount = page_mapcount(page); vma_prio_tree_foreach(vma, &iter, &mapping->i_mmap, pgoff, pgoff) { if ((vma->vm_flags & (VM_LOCKED|VM_MAYSHARE)) == (VM_LOCKED|VM_MAYSHARE)) { referenced++; break; } referenced += page_referenced_one(page, vma, &mapcount); if (!mapcount) break; } ... return referenced; } The kernel invokes vm_prio_tree_foreach to iterate over all elements of the priority tree that store a region where the relevant page is included. As above, page_referenced_one is invoked for each page in order to collect all references. If a page is locked into memory (with VM_LOCKED) and may be shared by processes (VM_MAYSHARE), the reference value is increased further because pages of this kind should not be swapped out and are therefore given a bonus. 4.9 Managing the Heap Managing the heap — the memory area of a process used to dynamically allocate variables and data — is not directly visible to application programmers because it relies on various helper functions of the stan- dard library (the most important of which is malloc) to reserve memory areas of any size. The classic interface between malloc and the kernel is the brk system call that expands/shrinks the heap. Recent malloc implementations (such as those of the GNU standard library) now use a combined approach that operates with brk and anonymous mappings. This approach delivers better performance and certain advantages when returning larger allocations. 327 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 328 Chapter 4: Virtual Process Memory The heap is a contiguous memory area that grows from bottom to top when expanded. The mm_struct structure already mentioned includes the start and the current end address (start_brk and brk)ofthe heap in virtual address space. struct mm_struct { ... unsigned long start_brk, brk, start_stack; ... }; The brk system call expects just a single parameter to specify the new end address of the heap in virtual address space (it can, of course, be smaller than the previous value if the heap is to be shrunk). As usual, the entry point for the implementation of the brk system call is the sys_brk function, whose code flow diagram is shown in Figure 4-16. Check resource limits sys_brk find_vma_intersection do_brk do_munmap Align brk value per page Increase of brk value? Return new brk value Return new brk value No Yes Figure 4-16: Code flow diagram for sys_brk. The brk mechanism is not another independent kernel concept but is implemented on the basis of anony- mous mappings to reduce internal overhead. Many of the functions to manage memory mappings discussed in the preceding sections can therefore be reused to implement sys_brk. After it has been checked that the new desired address for brk is actually within the heap limits, the first important action of sys_brk is to align the request to page size. mm/mmap.c asmlinkage unsigned long sys_brk(unsigned long brk) { unsigned long rlim, retval; unsigned long newbrk, oldbrk; struct mm_struct *mm = current->mm; 328 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 329 Chapter 4: Virtual Process Memory ... newbrk = PAGE_ALIGN(brk); oldbrk = PAGE_ALIGN(mm->brk); ... This code ensures that the new (and, as a precaution, the old) value of brk is a multiple of the system page size. In other words, a page is the smallest memory area that can be reserved with brk.15 do_munmap, with which we are familiar from Section 4.7.2, is invoked when it is necessary to shrink the heap. /* Always allow shrinking brk. */ if (brk <= mm->brk) { if (!do_munmap(mm, newbrk, oldbrk-newbrk)) goto set_brk; goto out; } ... If the heap is to be enlarged, the kernel must first check whether the new size is outside the limit set as the maximum heap size for the process. find_vma_intersection then checks whether the enlarged heap would overlap with an existing mapping of the process; if so, it returns without doing anything. /* Check against existing mmap mappings. */ if (find_vma_intersection(mm, oldbrk, newbrk+PAGE_SIZE)) goto out; ... Otherwise, the actual work of enlarging the heap is delegated to do_brk. The new value of mm->brk is always returned regardless of whether it is larger, smaller, or unchanged as compared to the old value. /* Ok, looks good - let it rip. */ if (do_brk(oldbrk, newbrk-oldbrk) != oldbrk) goto out; set_brk: mm->brk = brk; out: retval = mm->brk; return retval; } We need not discuss do_brk separately as essentially it is a simplified version of do_mmap_pgoff and reveals no new aspects. Like the latter, it creates an anonymous mapping in user address space but omits some safety checks and the handling of special situations to improve code performance. 15It is therefore essential to interpose a further allocator function in userspace to split the page into smaller areas; this is the task of the C standard library. 329 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 330 Chapter 4: Virtual Process Memory 4.10 Handling of Page Faults The association between virtual and physical memory is not established until the data of an area are actually needed. If a process accesses a part of virtual address space not yet associated with a page in memory, the processor automatically raises a page fault that must be handled by the kernel. This is one of the most important and complex aspects of memory management simply because a myriad of details must be taken into account. For example, the kernel must ascertain the following: ❑ Was the page fault caused by access to a valid address from the user address space, or did the application try to access the protected area of the kernel? ❑ Does a mapping exist for the desired address? ❑ Which mechanism must be used to obtain the data for the area? Figure 4-17 shows an initial overview of the potential paths the kernel may follow when handling page faults. Kernel or User- space address? Kernel mode? Synchronize with reference page table Segmentation Fault Yes No Sufficient privileges? Handle request Demand Paging/Allocation, Swapping or COW Mapping exists? Segmentation Fault Segmentation Fault NoYes Yes No Kernel User Figure 4-17: Potential options for handling page faults. As demonstrated below, the individual actions are much more complicated because the kernel must not only guard against malicious access from userspace but must also take note of many minor details; on top of this, it must not allow the page handling operations to degrade system performance unnecessarily. The implementation of page fault handling varies from processor to processor. Because the CPUs employ different memory management concepts, the details of page fault generation also differ. Consequently, the handler routines in the kernel are located in the architecture-specific source code segments. We confine ourselves below to a detailed description of the approach adopted on the IA-32 architecture. Implementation on most other CPUs is at least similar. An assembler routine in arch/x86/kernel/entry_32.S serves as the entry point for page faults but immediately invokes the C routine do_page_fault from arch/x86/mm/fault_32.c. (A routine of the 330 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 331 Chapter 4: Virtual Process Memory same name is present in the architecture-specific sources of most CPUs.16,17)Figure4-18showsthecode flow diagram of this extensive routine. Save faulting address Interrupt handler or no context? Usermode access Segmentation FaultYes No Address > TASK_SIZE and no protection fault and kernel mode? vmalloc-HandlerYes No vm_area_struct exists? Stack? No unsuccesssful erfolgreich Allowed read access, page not present Alllowed write access page not present Not allowed write access Not allowed read access successsful Yes No Yes do_page_fault find_vma fixup_exception expand_stack fixup_exception handle_mm_fault Figure 4-18: Code flow diagram for do_page_fault on IA-32 processors. This situation is complex, so it is necessary to examine the implementation of do_page_fault very closely. Two parameters are passed to the routine — the register set active at the time of the fault, and an error code (long error_code) that supplies information on the cause of the fault. Currently, only the first three bits(0,1,and2)oferror_code are used; their meanings are given in Table 4-1. arch/x86/mm/fault_32.c fastcall void __kprobes do_page_fault(struct pt_regs *regs, unsigned long error_code) { struct task_struct *tsk; struct mm_struct *mm; 16As usual, Sparc processors are the odd man out. There the name of the function is do_sparc_fault (Sparc32), do_sun4c_fault (Sparc32 sun4c), or do_sparc64_fault (UltraSparc). ia64_do_page_fault is used on IA-64 systems. 17Note that the code for IA-32 and AMD64 will be unified in kernel 2.6.25, which was still under development when this book was written. The remarks given here also apply for the AMD64 architecture. 331 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 332 Chapter 4: Virtual Process Memory struct vm_area_struct * vma; unsigned long address; unsigned long page; int write, si_code; int fault; ... /* get the address */ address = read_cr2(); ... Table 4-1: Meaning of Page Fault Error Codes on IA-32 Bit Set (1) Not set (0) 0 No page present in RAM Protection fault (insufficient access permission) 1 Read access Write access 2 Privileged kernel mode User mode Once a large number of variables have been declared for subsequent use, the kernel stores the address of the location that triggered the fault in address.18 arch/i386/mm/fault.c tsk = current; si_code = SEGV_MAPERR; /* * We fault-in kernel-space virtual memory on-demand. The * ’reference’ page table is init_mm.pgd. * * NOTE! We MUST NOT take any locks for this case. We may * be in an interrupt or a critical region, and should * only copy the information from the master page table, * nothing more. * * This verifies that the fault happens in kernel space * (error_code & 4) == 0, and that the fault was not a * protection error (error_code & 9) == 0. */ if (unlikely(address >= TASK_SIZE)) { if (!(error_code & 0x0000000d) && vmalloc_fault(address) >= 0) return; /* * Don’t take the mm semaphore here. If we fixup a prefetch * fault we could otherwise deadlock. 18On IA-32 processors, the address is held in register CR2, whose contents are copied to address by read_cr2. The processor- specific details are of no interest. 332 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 333 Chapter 4: Virtual Process Memory */ goto bad_area_nosemaphore; } ... A vmalloc fault is indicated if the address is outside user address space. The page tables of the process must therefore be synchronized with the information in the kernel’s master page table. Naturally, this is only permitted if access took place in kernel mode and the fault was not triggered by a protection error; in other words, neither bit 2 nor bits 3 and 0 of the error code may be set.19 The kernel uses the auxiliary function vmalloc_fault to synchronize the page tables. I won’t show the code in detail because all it does is copy the relevant entry from the page table of init —thisisthe kernel master table on IA-32 systems — into the current page table. If no matching entry is found there, the kernel invokes fixup_exception in a final attempt to recover the fault; I discuss this shortly. The kernel jumps to the bad_area_nosemaphore label if the fault was triggered during an interrupt (see Chapter 14) or in a kernel thread (see Chapter 14) that does not have its own context and therefore no separate instance of mm_struct. arch/i386/mm/fault.c mm = tsk->mm; /* * If we’re in an interrupt, have no user context or are running in an * atomic region then we must not take the fault.. */ if (in_atomic() || !mm) goto bad_area_nosemaphore; ... bad_area_nosemaphore: /* User mode accesses just cause a SIGSEGV */ if (error_code & 4) { ... force_sig_info_fault(SIGSEGV, si_code, address, tsk); return; } no_context: /* Are we prepared to handle this kernel fault? */ if (fixup_exception(regs)) return; A segmentation fault is output if the fault originates from userspace (indicated by the fact that bit 4 is set in error_code).If,however,thefaultoriginatesfromkernelspace,fixup_exception is invoked. I describe this function below. If the fault does not occur in an interrupt or without a context, the kernel checks whether the address space of the process contains a region in which the fault address lies. It invokes the find_vma function, which we know from Section 4.5.1 to do this. 19This is checked by !(error_code & 0x0000000d).Because20 + 22 + 23 = 13 = 0xd, neither bit 2 nor bits 3 and 0 may be set. 333 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 334 Chapter 4: Virtual Process Memory arch/i386/mm/fault.c vma = find_vma(mm, address); if (!vma) goto bad_area; if (vma->vm_start <= address) goto good_area; if (!(vma->vm_flags & VM_GROWSDOWN)) goto bad_area; ... if (expand_stack(vma, address)) goto bad_area; good_area and bad_area are labels to which the kernel jumps once it has discovered whether the address is valid or invalid. The search can yield various results: ❑ No region is found whose end address is after address, in which case access is invalid. ❑ The fault address is within the region found, in which case access is valid and the page fault is corrected by the kernel. ❑ A region is found whose end address is after the fault address but the fault address is not within the region. There may be two reasons for this: 1. The VM_GROWSDOWN flag of the region is set; this means that the region is a stack that grows from top to bottom. expand_stack is then invoked to enlarge the stack accordingly. If it succeeds, 0 is returned as the result, and the kernel resumes execution at good_area.Oth- erwise, access is interpreted as invalid. 2. The region found is not a stack, so access is invalid. good_area follows on immediately after the above code. arch/i386/mm/fault.c ... good_area: si_code = SEGV_ACCERR; write = 0; switch (error_code & 3) { default: /* 3: write, present */ /* fall through */ case 2: /* write, not present */ if (!(vma->vm_flags & VM_WRITE)) goto bad_area; write++; break; case 1: /* read, present */ goto bad_area; case 0: /* read, not present */ if (!(vma->vm_flags & (VM_READ | VM_EXEC))) goto bad_area; } ... 334 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 335 Chapter 4: Virtual Process Memory The presence of a mapping for the fault address does not necessarily mean that access is actually permit- ted. The kernel must check the access permissions by examining bits 0 and 1 (because 20 + 21 = 3). The following situations may apply: ❑ VM_WRITE must be set in the event of a write access (bit 1 set, cases 3 and 2). Otherwise, access is invalid, and execution resumes at bad_area. ❑ In the event of a read access to an existing page (Case 1), the fault must be a permission fault detected by the hardware. Execution then resumes at bad_area. ❑ If a read access is made to a page that doesn’t exist, the kernel must check whether VM_READ or VM_EXEC is set, in which case access is valid. Otherwise, read access is denied, and the kernel jumps to bad_area. If the kernel does not explicitly jump to bad_area, it works its way down through the case statement and arrives at the handle_mm_fault call that immediately follows; this function is responsible for correcting the page fault (i.e., reading the required data). arch/i386/mm/fault.c ... survive: /* * If for any reason at all we couldn’t handle the fault, * make sure we exit gracefully rather than endlessly redo * the fault. */ fault = handle_mm_fault(mm, vma, address, write); if (unlikely(fault & VM_FAULT_ERROR)) { if (fault & VM_FAULT_OOM) goto out_of_memory; else if (fault & VM_FAULT_SIGBUS) goto do_sigbus; BUG(); } if (fault & VM_FAULT_MAJOR) tsk->maj_flt++; else tsk->min_flt++; return; ... } handle_mm_fault is an architecture-independent routine for selecting the appropriate fault correction method (demand paging, swap-in, etc.) and for applying the method selected (we take a close look at the implementation and the various options of handle_mm_fault in Section 4.11). If the page is created successfully, the routine returns either VM_FAULT_MINOR (the data were already in memory) or VM_FAULT_MAJOR (the data had to be read from a block device). The kernel then updates the process statistics. However, faults may also occur when a page is created. If there is insufficient physical memory to load the page, the kernel forces termination of the process to at least keep the system running. If a permitted 335 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 336 Chapter 4: Virtual Process Memory access to data fails for whatever reason — for instance, if a mapping is accessed but has been shrunk by another process in the meantime and is no longer present at the given address — the SIGBUS signal is sent to the process. 4.11 Correction of Userspace Page Faults Once the architecture-specific analysis of the page fault has been concluded and it has been established that the fault was triggered at a permitted address, the kernel must decide on the appropriate method to read the required data into RAM memory. This task is delegated to handle_mm_fault, which is no longer dependent on the underlying architecture but is implemented system-independently within the memory management framework. The function ensures that page table entries for all directory levels that lead to the faulty PTE are present. The function handle_pte_fault analyzes the reason for the page fault. entry is a pointer to the relevant page table element (pte_t). mm/memory.c static inline int handle_pte_fault(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, pte_t *pte, pmd_t *pmd, int write_access) { pte_t entry; spinlock_t *ptl; if (!pte_present(entry)) { if (pte_none(entry)) { if (vma->vm_ops) { return do_linear_fault(mm, vma, address, pte, pmd, write_access, entry); } return do_anonymous_page(mm, vma, address, pte, pmd, write_access); } if (pte_file(entry)) return do_nonlinear_fault(mm, vma, address, pte, pmd, write_access, entry); return do_swap_page(mm, vma, address, pte, pmd, write_access, entry); } ... } Three cases must be distinguished if the page is not present in physical memory [!pte_present(entry)]. 1. If no page table entry is present (page_none), the kernel must load the page from scratch — this is known as demand allocation for anonymous mappings and demand paging for file-based mappings. This does not apply if there is no vm_operations_struct regis- tered in vm_ops — in this case, the kernel must return an anonymous page using do_ anonymous_page. 2. If the page is marked as not present but information on the page is held in the page table, this means that the page has been swapped out and must therefore be swapped back in from one of the system swap areas (swap-in or demand paging). 336 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 337 Chapter 4: Virtual Process Memory 3. Parts of nonlinear mappings that have been swapped out cannot be swapped in like regular pages because the nonlinear association must be restored correctly. The function pte_file allows for checking if the PTE belongs to a nonlinear mapping, and do_nonlinear_fault handles the fault. A further potential case arises if the region grants write permission for the page but the access mecha- nisms of the hardware do not (thus triggering the fault). Notice that since the page is present in this case, the above if case is executed and the kernel drops right through to the following code: mm/memory.c if (write_access) { if (!pte_write(entry)) return do_wp_page(mm, vma, address, pte, pmd, ptl, entry); entry = pte_mkdirty(entry); } ... do_wp_page is responsible for creating a copy of the page and inserting it in the page tables of the process — with write access permission for the hardware. This mechanism is referred to as copy on write (COW, for short) and is discussed briefly in Chapter 1. When a process forks, the pages are not copied immediately but are mapped into the address space of the process as ‘‘read-only‘‘ copies so as not to spend too much time in the (wasteful) copying of information. A separate copy of the page is not created for the process until write access actually takes place. The sections below take a closer look at the implementation of the fault handler routines invoked dur- ing page fault correction. They do not cover how pages are swapped in from a swap area by means of do_swap_page, as this topic is discussed separately in Chapter 18 and requires additional knowledge of the structure and organization of the swap layer. 4.11.1 Demand Allocation/Paging Allocation of pages on demand is delegated to do_linear_fault, which is defined in mm/memory.c.After some parameter conversion, the work is delegated to __do_fault,andthecodeflowdiagramofthis function is shown in Figure 4-19. First of all, the kernel has to make sure that the required data are read into the faulting page. How this is handled depends on the file that is mapped into the faulting address space, and therefore a file-specific method is invoked to obtain the data. Usually, it is stored in vm->vm_ops->fault. Since earlier kernel versions used a method with a different calling convention, the kernel must account for the situation in which some code has not yet been updated to stick to the new convention. Therefore, the old variant vm->vm_ops->nopage is invoked if no fault method is registered. Most files use filemap_fault to read in the required data. The function not only reads in the required data, but also implements readahead functionality, which reads in pages ahead of time that will most likely be required in the future. The mechanisms needed to do this are introduced in Chapter 16, which discusses the function in greater length. At the moment, all we need to know is that the kernel reads the data from the backing store into a physical memory page using the information in the address_space object. 337 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 338 Chapter 4: Virtual Process Memory Write access on private page? Only vma->vm_ops->nopage available? Allocate page Write access? Anonymous page? Yes No Yes No _ _do_fault vma->vm_ops->nopage vma->vm_ops->fault anon_vma_prepare copy_user_highpage flush_icache_page mk_pte pte_mkwrite lru_cache_add_active page_add_new_anon_rmap page_add_file_rmap update_mmu_cache Figure 4-19: Code flow diagram for __do_fault. Given the vm_area_struct region involved, how can the kernel choose which method to use to read the page? 1. The mapped file object is found using vm_area_struct->vm_file. 2. A pointer to the mapping itself can be found in file->f_mapping. 3. Each address space has special address space operations from which the readpage method can be selected. The data are transferred from the file into RAM memory using mapping-> a_ops->readpage(file, page). If write access is required, the kernel has to distinguish between shared and private mappings. For private mappings, a copy of the page has to be prepared. mm/memory.c static int __do_fault(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, pmd_t *pmd, pgoff_t pgoff, unsigned int flags, pte_t orig_pte) { ... 338 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 339 Chapter 4: Virtual Process Memory /* * Should we do an early C-O-W break? */ if (flags & FAULT_FLAG_WRITE) { if (!(vma->vm_flags & VM_SHARED)) { anon = 1; if (unlikely(anon_vma_prepare(vma))) { ret = VM_FAULT_OOM; goto out; } page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, address); ... } copy_user_highpage(page, vmf.page, address, vma); } ... A new page must be allocated once a new anon_vma instance has been created for the region with anon_vma_prepare (the pointer to the old region is redirected to the new region in anon_vma_prepare). The high memory area is preferably used for this purposeasitpresentsnoproblemsforuserspacepages. copy_user_highpage then creates a copy of the data (routines for copying data between kernel and userspace are discussed in Section 4.13). Now that the position of the page is known, it must be added to the page table of the process and incor- porated in the reverse mapping data structures. Before this is done, a check is made to ensure that the page contents are visible in userspace by updating the caches with flush_icache_page. (Most processors don’t need to do this and define an empty operation.) A page table entry that normally points to a read-only page is generated using the mk_pte function discussed in Section 3.3.2. If a page with write access is created, the kernel must explicitly set write permission with pte_mkwrite. How pages are integrated into the reverse mapping depends on their type. If the page generated when handling the write access is anonymous, it is added to the active area of the LRU cache using lru_cache_ add_active (Chapter 16 examines the caching mechanisms used in more detail) and then integrated into the reverse mapping with page_add_new_anon_rmap. page_add_file_rmap is invoked for all other pages associated with a file-based mapping. Both functions are discussed in Section 4.8. Finally, the MMU cache of the processor has to be updated if required because the page tables have been modified. 4.11.2 Anonymous Pages do_anonymous_page is invoked to map pages not associated with a file as a backing store. Except that no data must be read into a page, the procedure hardly differs from the way in which file-based data are mapped. A new page is created in the highmem area, and all its contents are deleted. The page is then added to the page tables of the process, and the caches/MMU are updated. Notice that earlier kernels distinguished between read-only and write access to anonymous mappings: In the first case, a single, global page filled with zero bytes was used to satisfy read requests to anonymous regions. During the development of kernel 2.6.24, this behavior has, however, been dropped because measurements have shown that the performance gain is negligible, while larger systems can experience several problems with shared zero mappings, which I do not want to discuss in detail here. 339 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 340 Chapter 4: Virtual Process Memory 4.11.3 Copy on Write Copy on write is handled in do_wp_page, whose code flow diagram is shown in Figure 4-20. do_wp_page Insert page into page tables vm_normal_page anon_vma_prepare alloc_page_vma cow_user_vma page_remove_rmap lru_cache_add_active page_add_new_anon_rmap Figure 4-20: Code flow diagram for do_wp_page. Let’s examine a slightly simplified version in which I have omitted potential interference with the swap cache as well as some corner cases, since this would complicate the situation without revealing anything insightful about the mechanism itself. The kernel first invokes vm_normal_page to find the struct page instance of the page by reference to the page table entry — essentially, this function builds on pte_pfn and pfn_to_page, which must be defined on all architectures. The former finds the page number for an associated page table entry, and the latter determines the page instance associated with the page number. This is possible because the COW mechanism is invoked only for pages that actually reside in memory (otherwise, they are first automatically loaded by one of the other page fault mechanisms). After obtaining a reference on the page with page_cache_get, anon_vma_prepare then prepares the data structures of the reverse mapping mechanism to accept a new anonymous area. Since the fault originates from a page filled with useful data that must be copied to a new page, the kernel invokes alloc_page_vma to allocate a fresh page. cow_user_page then copies the data of the faulted page into the new page to which the process may subsequently write. The reverse mapping to the original read-only page is then removed using page_remove_rmap. The new page is added to the page tables, at which point the CPU caches must also be updated. The final actions involve placing the newly allocated pages on the active list of the LRU cache using lru_cache_add_active and inserting them in the reverse mapping data structures by means of page_add_anon_rmap. Thereafter, the userspace process can write to the page to its heart’s content. 340 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 341 Chapter 4: Virtual Process Memory 4.11.4 Getting Nonlinear Mappings Page fault handling for nonlinear mappings is much shorter than when the methods described above are used: mm/memory.c static int do_nonlinear_fault(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, pte_t *page_table, pmd_t *pmd, int write_access, pte_t orig_pte) { ... pgoff = pte_to_pgoff(orig_pte); return __do_fault(mm, vma, address, pmd, pgoff, flags, orig_pte); } Since the faulting address is not linearly associated with the contents of the mapped file, the desired posi- tion must be obtained from the information in the PTE that was previously encoded with pgoff_to_pte. Now comes the time to put this information to use: pte_to_pgoff analyzes the page table entry and obtains the desired page-sized offset into the file. Once the address within the file is known, reading in the required data can be pursued as for regular page faults. The kernel thus hands off the work to the previously discussed function __do_page_fault and is done. 4.12 Kernel Page Faults When kernel address space is accessed, page faults can be triggered by various conditions as described below. ❑ A programming error in the kernel has caused an incorrect address to be accessed — this is a genuine bug. Of course, this should never happen in stable versions20 but does occur occasion- ally in developer versions. ❑ The kernel accesses an invalid address passed as a system call parameter from userspace. ❑ The page fault was triggered by access to an area allocated using vmalloc. The first two conditions are genuine errors against which the kernel must guard by performing additional checks. The vmalloc situation is a legitimate reason for a page fault that must be corrected. Modifications in the vmalloc address space are not transmitted to the page tables of a process until a corresponding page fault has occurred; the appropriate access information must be copied from the master page table. Although this is not a difficult operation, it is strongly architecture-dependent, so I won’t discuss it here. The exception fixup mechanism is a last resort when handling page faults not due to accessing vmalloc area. At some points, the kernel makes preparations for intercepting incorrect accesses that are made for a legitimate reason — for example, when copying address data from userspace addresses passed as system call parameters. 20In fact, errors of this kind very rarely occur because — as you might have already noted — Linux is an extremely stable system ... 341 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 342 Chapter 4: Virtual Process Memory Copying is performed by various central functions, such as copy_from_user, which is discussed in the next section. At present, it is sufficient to know that access to the incorrect address may occur at only a few places in the kernel. When data are copied to or from userspace, page faults may occur if access is made to an address in virtual address space that is not associated with a physical page. We are already familiar with this situation in user mode. When an application accesses a virtual address, the kernel automatically and transparently returns a physical page using the demand paging mechanism discussed above. If access takes place in kernel mode, the fault must likewise be corrected, albeit using slightly different means. Each time a page fault occurs, the cause of the fault and the address in the code currently executing are output. This enables the kernel to compile a list of all risky chunks of code that may carry out unau- thorized memory access operations. This ‘‘exception table‘‘ is created when the kernel image is linked and is located between __start_exception_table and __end_exception_table in the binary file. Each entry corresponds to an instance of struct exception_table, which, although architecture-dependent, is almost always structured as follows: struct exception_table_entry { unsigned long insn, fixup; }; insn specifies the position in virtual address space at which the kernel expects the fault; fixup is the code address at which execution resumes when the fault occurs. fixup_exception is used to search the exception tables and is defined as follows on IA-32 systems: arch/x86/mm/extable_32.c int fixup_exception(struct pt_regs *regs) { const struct exception_table_entry *fixup; fixup = search_exception_tables(regs->eip); if (fixup) { regs->eip = fixup->fixup; return 1; } return 0; } regs->eip points to the EIP register that, on IA-32 processors, contains the address of the code segment where the fault was triggered. search_exception_tables scans the exception table for a suitable entry.21 21To be more accurate, several tables are scanned — the main kernel table and the tables registered by modules loaded dynamically at kernel run time. As the mechanisms used are practically the same, it’s not worth describing their minor differences. 342 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 343 Chapter 4: Virtual Process Memory When a fixup routine is found, the instruction pointer is set to the corresponding memory location. The kernel executes the routine found after fixup_exception returns with return. What happens if there is no fixup routine? This indicates a genuine kernel fault that is handled by the code in do_page_fault, which follows the (unsuccessful) call of search_exception_table and results in a kernel oops. It looks like this on IA-32 processors: arch/x86/mm/fault_32.c fastcall void __kprobes do_page_fault(struct pt_regs *regs, unsigned long error_code) { ... no_context: /* Are we prepared to handle this kernel fault? */ if (fixup_exception(regs)) return; ... /* * Oops. The kernel tried to access some bad page. We’ll have to * terminate things with extreme prejudice. */ ... if (address < PAGE_SIZE) printk(KERN_ALERT "BUG: unable to handle kernel NULL " "pointer dereference"); else printk(KERN_ALERT "BUG: unable to handle kernel paging" " request"); printk(" at virtual address %08lx\n",address); printk(KERN_ALERT "printing eip: %08lx ", regs->eip); page = read_cr3(); page = ((__typeof__(page) *) __va(page))[address >> PGDIR_SHIFT]; printk("*pde = %08lx ", page); ... tsk->thread.cr2 = address; tsk->thread.trap_no = 14; tsk->thread.error_code = error_code; die("Oops", regs, error_code); do_exit(SIGKILL); ... If a virtual address between 0 and PAGE_SIZE - 1 is accessed, the kernel reports an invalid NULL pointer de-reference. Otherwise, the user is informed that a paging request could not be satisfied in kernel memory — this is a kernel bug in both cases. Additional information is also output to help debug the fault and to supply hardware-specific data; die prints the contents of the current registers (among other things). Thereafter, the current process is forced to terminate with SIGKILL to save whatever can be saved (in many cases, the system is rendered unusable by a fault of this kind). 343 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 344 Chapter 4: Virtual Process Memory 4.13 Copying Data between Kernel and Userspace The kernel often needs to copy data from userspace to kernel space; for example, when lengthy data structures are passed indirectly in system calls by means of pointers. There is a similar need to write data in the reverse direction from kernel space to userspace. This cannot be done simply by passing and de-referencing pointers for two reasons. First, userspace programs must not access kernel addresses; and second, there is no guarantee that a virtual page belong- ing to a pointer from userspace is really associated with a physical page. The kernel therefore provides several standard functions to cater for these special situations when data are exchanged between kernel space and userspace. They are shown in summary form in Table 4-2. Table 4-2: Standard Functions for Exchanging Data between Userspace and Kernel Space Function Meaning copy_from_user(to, from, n) __copy_from_user Copies a string of n bytes from from (userspace) to to (kernel space). get_user(type *to, type* ptr) __get_user Reads a simple variable (char, long, ... )fromptr to to; depending on pointer type, the kernel decides automatically to transfer 1, 2, 4, or 8 bytes. strncopy_from_user(to, from, n) __strncopy_from_user Copies a null-terminated string with a maximum of n characters from from (userspace) to to (kernel space). put_user(type *from, type *to) __put_user Copies a simple value from from (kernel space) to to (userspace); the relevant value is determined automat- ically from the pointer type passed. copy_to_user(to, from, n) __copy_to_user Copies n bytes from from (kernel space) to to (userspace). Table 4-3 lists additional helper functions for working with strings from userspace. These functions are subject to the same restrictions as the functions for copying data. get_user and put_user function correctly only when applied to pointers to ‘‘simple‘‘ data types such as char, int, and so on. They do not function with compound data types or arrays because of the pointer arithmetic required (and owing to the necessary implementation optimizations). Before structscanbe exchanged between userspace and kernel space, it is necessary to copy the data and then convert it to the correct type by means of typecasts. 344 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 345 Chapter 4: Virtual Process Memory Table 4-3: Standard Functions for Working with Strings in Userspace Data Function Meaning clear_user(to, n) __clear_user Fills the next n bytes after to with zeros. strlen_user(s) __strlen_user Gets the size of a null-terminated string in userspace (includ- ing the terminating character). strnlen_user(s, n) __strnlen_user Gets the size of a null-terminated string but restricts the search to a maximum of n characters. As the tables show, there are two versions of most of the functions. In the versions without preceding underscores, access_user is also invoked to perform checks on the userspace address; the checks carried out differ from architecture to architecture. For example, one check ensures that a pointer really points to a position in the user segment; another invokes handle_mm_fault if pages are not found in memory to make sure that data are read in for processing. All functions also apply the fixup mechanism described above to detect and correct page faults. The functions are implemented mainly in assembler language. They are extremely performance-critical becausetheyareinvokedsofrequently.Theexception code must also be integrated using complicated GNU C constructions to embed assembler and linker directives in the code. It is not my intention to discuss the implementation of the individual functions in detail. A checker tool was added to the compilation process during the development of kernel 2.5. It analyzes the sources to check whether userspace pointers can be de-referenced directly without the need for the above functions. The pointers originating from userspace must be labeled with the keyword __user so that the tool knows which pointers to check. One particular example is the chroot system call, which expects a filename as argument. Many, many more places in the kernel contain similarly marked arguments from userspace. asmlinkage long sys_chroot(const char __user * filename) { ... } Address space randomization has been augmented further during the development of kernel 2.6.25. It is now possible to randomize the address of the heap, traditionally called brk address. The randomiza- tion is, however, only performed if the new configuration option COMPAT_BRK is not set because some ancient propgrams are not compatible with a randomized heap address. On the technical level, brk randomization works as all other randomization techniques introduced in this chapter. 4.14 Summary You have seen that handling the virtual address space of userland processes is a very important part of the Linux kernel. I have introduced you to the general structure of address spaces and how they are managed by the kernel, and you have learned how they are partitioned into regions. These allow for describing the contents of the virtual memory space of userland processes and form the backbone for 345 Mauerer runc04.tex V3 - 09/04/2008 4:55pm Page 346 Chapter 4: Virtual Process Memory linear and nonlinear memory mappings. Besides, they are connected with paging, which helps to manage the connection between physical and virtual memory. Since the virtual address space of each userland process is different but the virtual address space portion of the kernel always remains the same, some effort is required to exchange data between both, and I have introduced you to the mechanisms required for this purpose. 346 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 347 Locking and Interprocess Communication As a multitasking system, Linux is able to run several processes at the same time. Normally, the individual processes must be kept as separate as possible so that they do not interfere with each other. This is essential to protect data and to ensure system stability. However, there are situations in which applications must communicate with each other; for example, ❑ when data generated by one process are transferred to another. ❑ when data are shared. ❑ when processes are forced to wait for each other. ❑ when resource usage needs to be coordinated. These situations are handled using several classic techniques that were introduced in System V and have since proven their worth, so much so that they are now part and parcel of Linux. Because not only userspace applications but also the kernel itself are faced with such situations — particularly on multiprocessor systems — various kernel-internal mechanisms are in place to handle them. If several processes share a resource, they can easily interfere with each other — and this must be prevented. The kernel therefore provides mechanisms not only for sharing data but also for coordinating access to data. Again, the kernel employs mechanisms adopted from System V. Resources need to be protected not only in userspace applications but especially in the kernel itself. On SMP systems, the individual CPUs may be in kernel mode at the same time and, theoretically, may want to manipulate all existing data structures. To prevent the CPUs from getting into each other’s way, it is necessary to protect some kernel areas by means of locks; these ensure that access is restricted to one CPU at a time. Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 348 Chapter 5: Locking and Interprocess Communication 5.1 Control Mechanisms Before describing the various interprocess communication (IPC) and data synchronization options of the kernel, let’s briefly discuss the ways in which communicating processes can interfere with each other — and how this can be prevented. Our discussion is restricted to basic and central aspects. For detailed explanations and numerous examples of classic problems, see the general textbooks on operating systems that are available on the market. 5.1.1 Race Conditions Let us consider a system that reads data from an external device via two interfaces. Independent data packets arrive via both interfaces at irregular intervals and are saved in separate files. To log the order of arrival of the data packets, a number is added at the end of each filename to indicate the ‘‘serial number‘‘ of the packet. A typical sequence of filenames would be act1.fil, act2.fil, act3.fil,andsoon.A separate variable is used to simplify the work of both processes. This variable is held in a memory page shared by both processes and specifies the next unused serial number (for the sake of simplicity, I refer to this variable as counter below). When a packet arrives, the process must perform a few actions to save the data correctly: 1. It reads the data from the interface. 2. It opens a file with the serial number count. 3. It increments the serial number by 1. 4. It writes the data to the file and closes it. Why should errors occur with this system? If each process strictly observes the above procedure and increments the status variable at the appropriate places, the procedure should obviously function cor- rectly not just with two but with any number of processes. As a matter of fact, it will function correctly in most cases — and this is where the real difficulty lies with distributed programming — but it won’t in certain circumstances. Let us set a trap by calling the processes that read data from the interfaces process 1 and process 2: Our scenario begins with a number of files to which a serial number has been added, say, 12 files in all. The value of counter is therefore 13. Obviously a bad omen ... Process 1 receives data from the interface as a new block has just arrived. Dutifully it opens a file with the serial number 13 just at the moment when the scheduler is activated and decides that the process has had enough CPU time and must be replaced with another process — in this case, process 2. Note that at this time, process 1 has read but not yet incremented the value of counter. Once process 2 has started to run, it too receives data from its interface and begins to perform the neces- sary actions to save these data. It reads the value of counter, increments it to 14, opens a file with serial number 13, writes the data to the file, and terminates. Soon it’s the turn of process 1 again. It resumes where it left off and increments the value of counter by 1, from 14 to 15. Then it writes its data to the previously opened file with serial number 13 — and, in doing so, overwrites the existing data of process 2. This is a double mishap — a data record is lost, and serial number 14 is not used. 348 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 349 Chapter 5: Locking and Interprocess Communication The program sequence could be modified to prevent this error by changing the individual steps after data have been received. For example, processes could increment the value of counter immediately after reading its value and before opening a file. However, closer examination of suggestions of this kind quickly lead to the conclusion that it is always possible to devise situations that result in a fatal error. If we look at our suggestion, it soon becomes clear that an inconsistency is generated if the scheduler is invoked between reading counter and incrementing its value. Situations in which several processes interfere with each other when accessing resources are generally referred to as race conditions. Such conditions are a central problem in the programming of distributed applications because they cannot usually be detected by systematic trial and error. Instead, a thorough study of source code (coupled with intimate knowledge of the various paths that code can take) and a generous supply of intuition are needed to find and eliminate them. Situations leading to race conditions are few and far between, thus begging the question as to whether it’s worth making the — sometimes very considerable — effort to protect code against their occurrence. In some environments (electronic aircraft control, monitoring of vital machinery, or dangerous equip- ment), race conditions may prove to be fatal in the literal sense of the word. But even in routine software projects, protection against potential race conditions is an important contribution to program quality and user satisfaction. As part of improved multiprocessor support in the Linux kernel, much effort has been invested in pinpointing areas where dangers lurk and in providing suitable protection. Unexpected system crashes and mysterious errors owing to lack of protection are simply unacceptable. 5.1.2 Critical Sections The essence of the problem is as follows: Processes are interrupted at points where they shouldn’t be if they are to do their work correctly. Obviously, a potential solution is to mark the relevant code section so that it can no longer be interrupted by the scheduler. Although this approach would work in principle, there are several inherent problems. Under certain circumstances, a faulty program would be unable to find its way out of the marked code section and would fail to relinquish the CPU, thus making the computer unusable. We must therefore reject this solution out of hand.1 The solution to the problem does not necessarily require that the critical section not be interrupted. Processes may well be interrupted in a critical section as long as no other process enters the section.This strict prohibition ensures that values cannot be changed by several processes at the same time and is referred to as mutual exclusion. Only one process may enter a critical area of code at a given time. There are many ways of designing a mutual exclusion method of this kind (regardless of the technical implementation). However, all must ensure that the exclusion principle functions under all circum- stances. It must depend neither on the number nor on the speed of the processors involved. If this were the case (and the solution were therefore only available on a given computer system with a specific hardware configuration), the solution would be impracticable because it would not provide general protection — and this is exactly what is needed. Processes should not be allowed to block each other and come to a permanent stop. Although this is a desirable goal, it cannot always be achieved by technical means as you will see below. It is often up to the programmer to think ahead so that the problem does not occur. Which principles are applied to support a mutual exclusion method? A multitude of different solu- tions have been proposed in the history of multitasking and multiuser systems, all with their specific 1The kernel itself can (and must) reserve the right to disable interrupts at certain points to seal itself off completely from external or periodic events. This is not, however, possible for user processes. 349 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 350 Chapter 5: Locking and Interprocess Communication advantages and disadvantages. Some solutions are of an academic nature, and some have found their way into practice in various operating systems — one solution has been adopted in most systems and is therefore worth discussing in detail. Semaphores Semaphores were designed by E. W. Dijkstra in 1965. At first glance, they provide a surprisingly sim- ple solution to all kinds of interprocess communication problems — but their use calls for experience, intuition, and caution. Basically, semaphores are simply specially protected variables that can represent both positive and nega- tive integers; their initial value is 1. Two standard operations are defined to manipulate semaphores — up and down. They are used to control entry to and exit from critical code areas, where it is assumed that competing processes have equal access to the semaphores. When a process wants to enter critical code, it invokes the down function. This decrements the value of the semaphore by 1; that is, it sets it to 0 and executes the dangerous section of code. Once it has performed the programmed actions, the up function is invoked to increment the value of the semaphore by 1 — thus resetting it to its original value. Semaphores are characterized by two special features: 1. When a second process tries to enter the critical code section, it too must first perform a down operation on the semaphore. Because a first process has already entered the code section, the value of the semaphore is 0. This causes the second process to ‘‘sleep‘‘on the semaphore. In other words, it waits until the first process has exited the relevant code. It is of particular importance in the implementation of the down operation that it is handled as an elementary step from the perspective of the application. It cannot then be interrupted by a scheduler call, and this means that race conditions cannot occur. In the view of the kernel, querying the variable and modifying its value are two different actions but are seen by the user as an atomic unit. When a process sleeps on a semaphore, the kernel puts it in the blocked state and also places it on a wait list with all other processes waiting on the semaphore. 2. When a process exits the critical code section, it performs the up operation. This not only increments the value of the semaphore (to 1), but also selects a process sleeping on it. This process is now able to safely begin execution of the critical code after resuming and complet- ing its down operation to decrement the semaphore value to 0. This procedure would not be possible without the special support of the kernel because a userspace library cannot guarantee that the down operation will not be interrupted. Before describing the imple- mentation of the corresponding functions, it is first necessary to discuss the mechanisms that the kernel itself uses to protect critical code sections. These mechanisms are the basis for the protection facilities exported to user programs. Semaphores work well in userland, and could in principle also be used to solve all kinds of in-kernel locking problems. But they are not: Performance is one of the foremost goals of the kernel, and despite the fact that semaphores might seem simple to implement at a first glance, their overhead is usually too large for the kernel. This is why a plethora of different locking and synchronization mechanisms are available for use in the kernel, which I discuss in the following. 350 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 351 Chapter 5: Locking and Interprocess Communication 5.2 Kernel Locking Mechanisms The kernel requires no explicit mechanisms to facilitate the distributed use of memory areas because it has unrestricted access to the full address space. On multiprocessor systems (or similarly, on unipro- cessor systems with enabled kernel preemption; see Chapter 2), this gives rise to a few problems. If several processors are in kernel mode at the same time, they can access the same data structure simulta- neously — and this is exactly what causes the problem described in the previous sections. In the first SMP-capable version of the kernel, the solution to this problem was very simple. Only one processor at a time was ever allowed to be in kernel mode. Consequently, uncoordinated parallel access to data was automatically ruled out. Unfortunately, this method was obviously not very efficient and was quickly dropped. Nowadays, the kernel uses a fine-grained network of locks for the explicit protection of individual data structures. If processor A is manipulating data structure X, processor B may perform any other kernel actions — but it may not manipulate X. The kernel provides various locking options for this purpose, each optimized for different kernel data usage patterns. ❑ Atomic Operations — These are the simplest locking operations. They guarantee that simple operations, such as incrementing a counter, are performed atomically without interruption even if the operation is made up of several assembly language statements. ❑ Spinlocks — These are the most frequently used locking option. They are designed for the short- term protection of sections against access by other processors. While the kernel is waiting for a spinlock to be released, it repeatedly checks whether it can acquire the lock without going to sleep in the meantime (busy waiting). Of course, this is not very efficient if waits are long. ❑ Semaphores — These are implemented in the classical way. While waiting for a semaphore to be released, the kernel goes to sleep until it is woken. Only then does it attempt to acquire the semaphore. Mutexes are a special case of semaphores — only one user at a time can be in the critical region protected by them. ❑ Reader/Writer Locks — These are locks that distinguish between two types of access to data structures. Any number of processors may perform concurrent read access to a data structure, but write access is restricted to a single CPU. Naturally, the data structure cannot be read while it is being written. The sections below discuss the implementation and usage of these options in detail. Their deployment is omnipresent over all the kernel sources, and locking has become a very important aspect of kernel devel- opment, both for fundamental core kernel code as well as for device drivers. Nevertheless, when I discuss specific kernel code in this book, I will mostly omit locking operations except if locks are employed in an unusual way, or if special locking requirements must be fulfilled. But why do we omit this aspect of the kernel in other chapters if it is important? Most of you will certainly agree that this book does not belong to the thinner specimen on your bookshelves, and adding a detailed discussion of locking in all subsystems would certainly not be the ultimate diet for the book. More important, however, is that in most cases a discussion of locking would obstruct and complicate the view on the essential working of a particular mechanism. My main concern, however, is to provide exactly this to you. Fully understanding the use of locks requires line-by-line familiarity with all kernel code affected by the locks, and this is something a book cannot provide — in fact, something that it should not even try: 351 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 352 Chapter 5: Locking and Interprocess Communication The source code of Linux is readily available, and there’s no need to fill page after page with material that you cannot only easily inspect yourself on your computer, but that is additionally subject to perma- nent change in many details. So it is much better to equip you with a solid understanding of the concepts that are naturally much less likely to change rapidly. Nevertheless, this chapter will provide you with everything necessary to understand how protection against concurrency is implemented in specific sub- systems, and together with the explanations about the design and working of these, you will be well equipped to dive into the source code, read, and modify it. 5.2.1 Atomic Operations on Integers The kernel defines the atomic_t data type (in ) as the basis for atomic operations with integer counters. In the view of the kernel, these are operations that are performed as if they con- sisted of a single assembly language instruction. A short example, in which a counter is incremented by 1, is sufficient to demonstrate the need for operations of this kind. On the assembly language level, incrementation is usually performed in three steps: 1. The counter value is copied from memory into a processor register. 2. The value is incremented. 3. The register data are written back to memory. Problems may occur if this operation is performed on a second processor at the same time. Both proces- sors read the value in memory simultaneously (e.g., 4), increment it to 5, and write the new value back to memory. However, the correct value in memory should be 6 because the counter was incremented twice. All processors supported by the kernel provide means of performing operations of this kind atomically. In general, special lock instructions are used to prevent the other processors in the system from working until the current processor has completed the next action. Equivalent mechanisms that produce the same effect may also be used.2 To enable platform-independent kernel parts to use atomic operations, the architecture-specific code must provide the operations listed in Table 5-1 that manipulate variables of type atomic_t.Because, on some systems, these operations are much slower than normal C operations, they should not be used unless really necessary. As an understanding of operation implementation presupposes a deep knowledge of the assembler facilities of the individual CPUs, I do not deal with this topic here (each processor architecture provides special functions to implement operations). It is not possible to mix classic and atomic operations. The operations listed in Table 5-1 do not function with normal data types such as int or long,and conversely standard operators such as ++ do not work with atomic_t variables. It should also be noted that atomic variables may be initialized only with the help of the ATOMIC_INIT macro. Because the atomic data types are ultimately implemented with normal C types, the kernel encapsulates standard variables in a structure that can no longer be processed with normal operators such as ++. 2The required instruction is actually called lock on IA-32 systems. 352 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 353 Chapter 5: Locking and Interprocess Communication typedef struct { volatile int counter; } atomic_t; Table 5-1: Atomic Operations Operation Effect atomic_read(atomic_t *v) Reads the value of the atomic variable. atomic_set(atomic_t *v, int i) Sets v to i. atomic_add(int i, atomic_t *v) Adds i to v. atomic_add_return(int i, atomic_t *v) Adds i to v and returns the result. atomic_sub(int i, atomic_t *v) Subtracts i from v. atomic_sub_return(int i, atomic_t *v) Subtracts i from v and returns the result. atomic_sub_and_test(int i, atomic_t *v) Subtracts i from v. Returns a true value if the result is 0, otherwise false. atomic_inc(atomic_t *v) Subtracts 1 from v. atomic_inc_and_test(atomic_t *v) Adds 1 to v.Returnstrue if the result is 0, otherwise false. atomic_dec(atomic_t *v) Subtracts 1 from v. atomic_dec_and_test(atomic_t *v) Subtracts 1 from v.Returnstrue if the result is 0, other- wise false. atomic_add_negative(int i, atomic_t *v) Adds i to v.Returnstrue if the result is less than 0, otherwise false. atomic_add_negative(int i, atomic_t *v) Adds i to v and returns true if the result is negative, otherwise false. If the kernel was compiled without SMP support, the operations described are implemented in the same way as for normal variables (only atomic_t encapsulation is observed) because there is no interference from other processors. The kernel provides the local_t data type for SMP systems. This permits atomic operations on a single CPU. To modify variables of this kind, the kernel basically makes the same functions available as for the atomic_t data type, but it is then necessary to replace atomic with local. Notice that atomic variables are well suited for integer operations, but not so for bit operations. Each architecture therefore has to define a set of bit manipulation operations, and these also work atomically to provide coherence across processors on SMP systems. The available operations are summarized in Section A.8. 353 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 354 Chapter 5: Locking and Interprocess Communication 5.2.2 Spinlocks Spinlocks are used to protect short code sections that comprise just a few C statements and are therefore quickly executed and exited. Most kernel data structures have their own spinlock that must be acquired when critical elements in the structure are processed. Although spinlocks are omnipresent in the kernel sources, I omit them in most pieces of code shown in this book. They do not provide any valuable insight into how the kernel functions but make the code more difficult to read, as explained above. Nevertheless, it is important that code is equipped with appropriate locks! Data Structures and Usage Spinlocks are implemented by means of the spinlock_t data structure, which is manipulated essentially using spin_lock and spin_unlock. There are a few other spinlock operations: spin_lock_irqsave not only acquires the spinlock but also disables the interrupts on the local CPU, and spin_lock_bh also disables softIRQs (see Chapter 14). Spinlocks acquired with these operations must be released by means of their counterpart; spin_unlock_bh and spin_unlock_irqsave, respectively. Once again, implementation is almost fully in (strongly architecture-dependent) assembly language and is therefore not discussed here. Spinlocks are used as follows: spinlock_t lock = SPIN_LOCK_UNLOCKED; ... spin_lock(&lock); /* Critical section */ spin_unlock(&lock); SPIN_LOCK_UNLOCKED must be used in its unlocked state to initialize the spinlock. spin_lock takes account of two situations: 1. If lock is not yet acquired from another place in the kernel, it is reserved for the current processor. Other processors may no longer enter the following area. 2. If lock is already acquired by another processor, spin_lock goes into an endless loop to repeatedly check whether lock has been released by spin_unlock (thus the name spinlock). Once this is the case, lock is acquired, and the critical section of code is entered. spin_lock is defined as an atomic operation to prevent race conditions arising when spinlocks are acquired. Additionally, the kernel provides the two methods spin_trylock and spin_trylock_bh.Theytryto obtain a lock, but will not block if it cannot be immediately acquired. When the locking operation has succeeded, they return a nonzero value (and the code is protected by the spinlock), but otherwise they return 0. In this case, the code is not protected by the lock. Two points must be noted when using spinlocks: 1. If a lock is acquired but no longer released, the system is rendered unusable. All processors — including the one that acquired the lock — sooner or later arrive at a point where they must enter the critical region. They go into the endless loop to wait for lock release, but this never happens. This produces a deadlock, and the grim name suggests that it’s something that should be avoided. 354 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 355 Chapter 5: Locking and Interprocess Communication 2. On no account should spinlocks be acquired for a longer period because all processors wait- ing for lock release are no longer available for other productive tasks (the situation with semaphores is different, as you will see shortly). Code that is protected by spinlocks must not gotosleep.Thisruleisnotsosimple to obey as it seems: It is not complicated to avoid going to sleep actively, but it must also be ensured that none of the functions that are called inside a spinlocked region can go to sleep! One particular example is the kmalloc function: Usually the requested memory will be returned straight away, but when the kernel is short on memory, the function can go to sleep, as discussed in Chapter 3. Code that makes the mistake of allocating memory inside a spinlocked region will thus work perfectly fine most of the time, but sometimes cause a failure. Naturally, such problems are very hard to reproduce and debug. Therefore, you should pay great attention to which functions you call inside a spinlocked region, and make sure that they cannot go to sleep in any circumstance. On uniprocessor systems, spinlocks are defined as empty operations because critical sections cannot be entered by several CPUs at the same time. However, this does not apply if kernel preemption is enabled. If the kernel is interrupted in a critical region and this region is then entered by another process, this has exactly the same effect as if the region were actually being executed by two processors on SMP systems. This is prevented by a simple trick — kernel preemption is disabled when the kernel is in a critical region protected by a spinlock. When a uniprocessor kernel is compiled with enabled kernel preemption, spin_lock is (basically) equivalent to preempt_disable and spin_unlock to preempt_enable. Spinlocks cannot be acquired more than once from the current holder! This is especially important when functions that call other functions that each operate with the same lock. If a lock has already been acquired and a function is called that tries to again acquire it although the current code path is already holding the lock, a deadlock will occur — the processor will wait for itself to release the lock, and this mighttakeawhile... Finally, notice that the kernel itself also provides some notes on how to use spinlocks in Documentation/spinlocks.txt. 5.2.3 Semaphores Semaphores that are used in the kernel are defined by the structure below. Userspace semaphores are implemented differently, as described in Section 5.3.2. struct semaphore { atomic_t count; int sleepers; wait_queue_head_t wait; }; 355 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 356 Chapter 5: Locking and Interprocess Communication Although the structure is defined in an architecture-dependent header file, most architectures use the structure shown. ❑ count specifies how many processes may be in the critical region protected by the semaphore at the same time. count == 1 is used in most cases (semaphores of this kind are also known as mutex semaphores because they are used to implement mutual exclusion). ❑ sleepers specifies the number of processes waiting to be allowed to enter the critical region. Unlike spinlocks, waiting processes go to sleep and are not woken until the semaphore is free; this means that the relevant CPU can perform other tasks in the meantime. ❑ wait is used to implement a queue to hold the task structures of all processes sleeping on the semaphore (Chapter 14 describes the underlying mechanisms). In contrast to spinlocks, semaphores are suitable for protecting longer critical sections against parallel access. However, they should not be used to protect shorter sections because it is very costly to put processes to sleep and wake them up again — as happens when the semaphore is contended. In most cases, the full potential of semaphores is not required, but they are used in the form of mutexes, which are nothing other than binary semaphores. To simplify this case, the kernel provides the macros DECLARE_MUTEX, which declare a binary semaphore that starts out unlocked with count = 1.3 DECLARE_MUTEX(mutex) ... down(&mutex); /* Critical section */ up(&mutex); The usage counter is decremented with down when the critical section is entered. When the counter has reached 0, no other process may enter the section. When an attempt is made to acquire a reserved semaphore with down, the current process is put to sleep and placed on the wait queue associated with the semaphore. At the same time, the process is placed in the TASK_UNINTERRUPTIBLE state and cannot receive signals while waiting to enter the critical region. If the semaphore is not reserved, the process may immediately continue without being put to sleep and enters the critical region, but not without reserving the semaphore first. up must be called when the critical region is exited. The routine is responsible for waking one of the processes sleeping on the semaphore — this process is then allowed to enter the critical section, and all other processes continue to sleep. In addition to down, two other operations are used to reserve a semaphore (unlike spinlocks, only one up function is available and is used to exit the section protected by a semaphore): ❑ down_interruptible works in the same way as down but places the task in the TASK_INTERRUPTIBLE state if the semaphore could not be acquired. As a result, the process can be woken by signals while it is sleeping.4 3Note that earlier kernel versions also provided the macro DECLARE_MUTEX_LOCKED to initialize a locked semaphore, but this variant has been removed during the development of kernel 2.6.24 because it was only required for operations that can be better implemented by completions, as discussed in Section 14.4. 4If the semaphore is acquired, the function returns 0. If the process is interrupted by a signal without acquiring the semaphore, -EINTR is returned. 356 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 357 Chapter 5: Locking and Interprocess Communication ❑ down_trylock attempts to acquire a semaphore. If it fails, the process does not go to sleep to wait for the semaphore but continues execution normally. If the semaphore is acquired, the function returns a false value, otherwise a true value. In addition to mutex variables that can be used in the kernel only, Linux also features so-called futexes (fast userspace mutex) that consist of a combination of kernel and user mode. These provide mutex functionality for userspace processes. However, it must be ensured that they are used and manipulated as quickly and efficiently as possible. For reasons of space, I dispense with a description of their implementation, particularly as they are not especially important for the kernel itself. See the manual page futex(2) for more information. 5.2.4 The Read-Copy-Update Mechanism Read-copy-update (RCU) is a rather new synchronization mechanism that was added during the devel- opment of kernel 2.5, but has been very favorably accepted by the kernel community. It is by now used in numerous places all over the kernel. RCU performs very well in terms of performance impact, if at a slight cost in memory requirements, which is, however, mostly negligible. This is a good thing, but good things are always accompanied by a number of not-so-good things. This time, it’s the constraints that RCU places on potential users: ❑ Accesses to the shared resource should be Read Only most of the time, and writes should be correspondingly rare. ❑ The kernel cannot go to sleep within a region protected by RCU. ❑ The protected resource must be accessed via a pointer. The principle of RCU is simple: The mechanism keeps track of all users of the pointer to the shared data structure. When the structure is supposed to change, a copy (or a new instance that is filled in appropriately, this does not make any difference) is first created and the change is performed there. After all previous readers have finished their reading work on the old copy, the pointer can be replaced by a pointer to the new, modified copy. Notice that this allows read access to happen concurrently with write updates! Core API Suppose that a pointer ptr points to a data structure that is protected by RCU. It is forbidden to simply de-reference the pointer, but rcu_dereference(ptr) must be invoked before and the result be de- referenced. Additionally, the code that de-references the pointer and uses the result needs to be embraced by calls to rcu_read_lock and rcu_read_unlock: rcu_read_lock(); p = rcu_dereference(ptr); if (p != NULL) { awesome_function(p); } rcu_read_unlock(); The de-referenced pointer may not be used outside the region protected by rcu_read_lock() ... rcu_read_unlock(), nor may it be used for write access! 357 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 358 Chapter 5: Locking and Interprocess Communication If the object pointed at by ptr has to be modified, this must be done with rcu_assign_pointer: struct super_duper *new_ptr = kmalloc(...); new_ptr->meaning = xyz; new_ptr->of = 42; new_ptr->life = 23; rcu_assign_pointer(ptr, new_ptr); In RCU terminology, this publishes the pointer, and subsequent read operations will see the new structure instead of the old one. If updates can come from many places in the kernel, protection against concurrent write operations must be provided using regular synchronization primitives, for instance, spinlocks. While RCU protects readers from writers, it does not protect writers against writers! What happens to the old structure once the new value has been published? After all readers are gone, the kernel can get rid of the memory — but it needs to know when this is safe to do. RCU provides two more functions for this purpose: ❑ synchronize_rcu() waits until all existing readers have finished their work. After the function returns, it is safe to free the memory associated with the old pointer. ❑ call_rcu can be used to register a function that is called after all existing readers to a shared resource are gone. This requires that an instance of rcu_head is embedded — and not just acces- sible via a pointer — into the data stucture protected by RCU: struct super_duper { struct rcu_head head; int meaning, of, life; }; The callback gets the rcu_head of the object passed as parameter and can use the container_of mechanism to access the object. kernel/rcupdate.c void fastcall call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu)) List Operations Generic pointers are not the only objects that can be protected by RCU. The kernel also provides stan- dard functions that allow for protecting doubly linked lists by the RCU mechanism, and this is the most prominent application within the kernel. Additionally, hash lists that consist of struct hlist_head and struct hlist_node pairs can also be protected by RCU. The nice thing about list protection by RCU is that the standard list elements can still be used — it is only necessary to invoke the RCU variants of standard functions to iterate over lists and change and delete list elements. The names of the functions are easy to remember: Just append _rcu to the standard functions. 358 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 359 Chapter 5: Locking and Interprocess Communication static inline void list_add_rcu(struct list_head *new, struct list_head *head) static inline void list_add_tail_rcu(struct list_head *new, struct list_head *head) static inline void list_del_rcu(struct list_head *entry) static inline void list_replace_rcu(struct list_head *old, struct list_head *new) ❑ list_add_rcu adds a new element new to the beginning of a list headed by head, while list_add_tail_rcu adds it to the end. ❑ list_replace_rcu replaces the list element old with new. ❑ list_del_rcu removes the element entry from its list. Most importantly, list_for_each_rcu allows for iterating over all elements of a list. The variant list_for_each_rcu_safe is even safe against element removal. Both operations must be enclosed in a rcu_read_lock() ... rcu_read_unlock() pair. Notice that the kernel provides a large amount of documentation about RCU by the creator of the mecha- nism. It is located in Documentation/RCU and makes for a very interesting read — especially because it is not outdated like many other texts included in the kernel. The documents not only provide information about how RCU is implemented, but additionally introduce some further standard functions not covered here because their use in the kernel is not so common. 5.2.5 Memory and Optimization Barriers Modern compilers and processors will try to squeeze every bit of performance out of code, and readers will certainly agree that this is a good thing. However, as with every good thing, there is also a drawback to consider (maybe you’ve heard this before in this chapter). One particular technique to achieve better performance is to reorder instructions. This can be perfectly fine, as long as the result is identical. How- ever, it can be hard to decide for a compiler or processor if the result of a reordering will really match the intended purpose, especially if side effects need to be considered — a thing at which machines are naturally suboptimal compared to humans. Side effects are, however, a common and necessary effect when data are written to I/O registers. While locks are sufficient to ensure atomicity, they cannot always guarantee time ordering of code that is subjected to optimizations by compilers and processors. And, in contrast to race conditions, this problem not only affects SMP systems, but also uniprocessor machines. The kernel provides several functions to prevent both the processor and the compiler from reordering code: ❑ mb(), rmb(),andwmb() insert hardware memory barriers into the code flow. rmb() is a read mem- ory barrier. It guarantees that all read operations issued before the barrier are completed before any read operations after the barrier are carried out. wmb does the same thing, but this time for write accesses. And, as you have guessed, mb() combines both effects. ❑ barrier inserts an optimization barrier. This instructs the compiler to assume that all memory locations in RAM stored in CPU registers that were valid before the barrier are invalid after the barrier. Essentially, this means that the compiler does not process any read or write 359 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 360 Chapter 5: Locking and Interprocess Communication requests following the barrier before read or write requests issued before the barrier have been completed. The CPU can still reorder the time flow! ❑ smb_mb(), smp_rmb(),andsmp_wmb() act as the hardware memory barriers described above, but only when they are used on SMP systems. They generate a software barrier on uniprocessor systems instead. ❑ read_barrier_depends() is a special form of a read barrier that takes dependencies among read operations into account. If a read request after the barrier depends on data for which a read request is performed before the barrier, then both compiler and hardware must not reorder these requests. Notice that all commands presented above will have an impact on run-time performance. This is only natural because if optimizations are disabled, things tend to run slower than with optimizations, which is the whole purpose of optimizing code. Most of you will agree, though, the code that runs a little slower but does the right thing is preferable to code that is fast — and wrong. One particular application for optimization barriers is the kernel preemption mechanism. Note that the preempt_disable increments the preemption counter and thus disables preemption, while preempt_enable reenables preemption again by decreasing the preemption counter. Code inside a region embraced by these commands is protected against preemption. Consider the following code: preempt_disable(); function_which_must_not_be_preempted(); preempt_enable(); It would be quite inconvenient if the compiler would decide to rearrange the code as follows: function_which_must_not_be_preempted(); preempt_disable(); preempt_enable(); Another possible reordering would be likewise suboptimal: preempt_disable(); preempt_enable(); function_which_must_not_be_preempted(); In both cases, the non-preemptible part could be preempted. Therefore, preempt_disable inserts a mem- ory barrier after the preemption counter has been incremented: #define preempt_disable() \ do { \ inc_preempt_count(); \ barrier(); \ } while (0) This prevents the compiler from swapping inc_preempt_count() with any of the following statements. Likewise, preempt_enable has to insert an optimization barrier before preemption is enabled again: #define preempt_enable() \ do { \ 360 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 361 Chapter 5: Locking and Interprocess Communication ... barrier(); \ preempt_check_resched(); \ } while (0) This measure protects against the second erroneous reordering shown above. All barrier commands discussed so far are made available by including . You might have got- ten the impression that memory barriers are notoriously complicated to use, and your perception serves you well, indeed — getting memory and optimization barriers right can be a very tricky business. It should therefore be noted that memory barriers are not particularly favored by some kernel maintainers, and code using them will have a hard time finding its way into mainline. So it’s always worth a try to see if things cannot be done differently without barriers. This is possible because locking instructions will on many architectures also act as memory barriers. However, this needs to be checked for the specific cases that require memory barriers, and general advice is hard to give. 5.2.6 Reader/Writer Locks The mechanisms described above have one disadvantage. They do not differentiate between situations in which data structures are simply read and those in which they are actively manipulated. Usually, any number of processes are granted concurrent read access to data structures, whereas write access must be restricted exclusively to a single task. The kernel therefore provides additional semaphore and spinlock versions to cater for the above — these are known accordingly as Reader/Writer semaphores and Reader/Writer spinlocks. The rwlock_t data type is defined for Reader/Writer spinlocks. Locks must be acquired in different ways in order to differentiate between read and write access. ❑ read_lock and read_unlock must be executed before and after a critical region to which a pro- cess requires read access. The kernel grants any number of read processes concurrent access to the critical region. ❑ write_lock and write_unlock are used for write access. The kernel ensures that only one writer (and no readers) is in the region. An _irq _irqsave variant is also available and functions in the same way as normal spinlocks. Variants ending in _bh are also available. They disable software interrupts, but leave hardware interrupts still enabled. Read/write semaphores are used in a similar way. The equivalent data structure is struct rw_semaphore, and down_read and up_read are used to obtain read access to the critical region. Write access is per- formed with the help of down_write and up_write.The_trylock variants are also available for all commands — they also function as described above. 5.2.7 The Big Kernel Lock A relic of earlier days is the option of locking the entire kernel to ensure that no processors run in par- allel in kernel mode. This lock is known as the big kernel lock but is most frequently referred to by its abbreviation, BKL. Thecompletekernelislockedusinglock_kernel; its unlocking counterpart is unlock_kernel. 361 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 362 Chapter 5: Locking and Interprocess Communication A special feature of the BKL is that its lock depth is also counted. This means that lock_kernel can be invoked even when the kernel has already been locked. The matching unlocking operation (unlock_kernel) must then be invoked the same number of times to unlock the kernel so that other processors can enter it. Although the BKL is still present at more than 1,000 points in the kernel, it is an obsolete concept whose use is deprecated by kernel developers because it is a catastrophe in terms of performance and scalability. New code should on no account use the lock but should instead adopt the finer-grained options described above. Nevertheless, it will be a few more years before the BKL finally disappears — if ever at all.5 The kernel sources summarize the situation well: lib/kernel_lock.c /* * lib/kernel_lock.c * * This is the traditional BKL - big kernel lock. Largely * relegated to obsolescence, but used by various less * important (or lazy) subsystems. */ Notice that SMP systems and UP systems with kernel preemption allow for preempting the big kernel lock if the configuration option PREEMPT_BKL is set, although I will not discuss this mechanism further. While this helps to decrease kernel latency, it is not a cure for the problems created by the BKL and should only be seen as an emergency measure that serves as good as possible as an interim solution. 5.2.8 Mutexes Although semaphores can be used to implement the functionality of mutexes, the overhead imposed by the generality of semaphores is often not necessary. Because of this, the kernel contains a separate implementation of special-purpose mutexes that are not based on semaphores. Or, to be precise, the kernel contains two implementations of mutexes: A classical variant, and real-time mutexes that allow for solving priority inversion problems. I discuss both approaches in the following. Classical Mutexes The basic data structure for classical mutexes is defined as follows: struct mutex { /* 1: unlocked, 0: locked, negative: locked, possible waiters */ atomic_t count; spinlock_t wait_lock; struct list_head wait_list; }; The concept is rather simple: count is 1 if the mutex is unlocked. The locked case distinguishes between two situations. If only a single process is using the mutex, then count is set to 0. If the mutex is locked and any processes are waiting on the mutex to be unlocked (and need be awoken when this happens), count is negative. This special casing helps to speed up the code because in the usual case, no one will be waiting on the mutex. 5During the development of kernel 2.6.26, a special kernel tree whose purpose is to speed up BKL removal was created, and hope- fully progress will be accelerated by this measure. 362 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 363 Chapter 5: Locking and Interprocess Communication There are two ways to define new mutexes: 1. Static mutexes can be generated at compile time by using DEFINE_MUTEX (be sure not to con- fuse this with DECLARE_MUTEX from the semaphore-based mutexes!). 2. mutex_init dynamically initializes a new mutex at run time. mutex_lock and mutex_unlock are used to lock and unlock a mutex, respectively. In addition, the kernel also provides the function mutex_trylock, which tries to obtain the mutex, but will return immediately if this fails because the mutex is already locked. Finally, mutex_trylock can be used to check if a given mutex is locked or not. Real-Time Mutexes Real-time mutexes (RT-mutexes) are another form of mutex supported by the kernel. They need to be explicitly enabled at compile time by selecting the configuration option CONFIG_RT_MUTEX. In contrast to regular mutexes, they implement priority inheritance, which, in turn, allows for solving (or, at least, attenuating) the effects of priority inversion. Both are well-known effects, respectively, methods and are discussed in most operating systems textbooks. Consider a situation in which two processes run on a system: A has high priority, and C has low priority. Assume the C has acquired a mutex and is happily operating in the protected region, with no intention of leaving it sometime in the near future. However, shortly after C has entered the protected region, A also tries to obtain the mutex that protects it — and has to wait, because C has already acquired the mutex. This causes the higher-priority process A to wait for the lower-priority process C. Things can get worse when a third process B with a priority between A and C enters the field. Suppose that C still holds the lock, and A is waiting for it. Now B gets ready to run. Because it has a higher priority than C, it can preempt C. But it has also effectively preempted A, although this process has a higher priority than B. If B continues to stay runnable, it can prevent A from running for a long time, because C will finish its operation only slowly. B therefore acts as if it had a higher priority than A. This unfortunate situation is referred to as unbounded priority inversion. This problem can be solved by priority inheritance: When a high-priority process blocks on a mutex that is currently held by a low-priority process, the priority of C (in our example) is temporarily boosted to the priority of A. If B starts to run now, it will only get as much CPU time as if it were competing with A, thus setting priorities straight again. The definition of a RT-mutex is tantalizingly close to the definition of a regular mutex: struct rt_mutex { spinlock_t wait_lock; struct plist_head wait_list; struct task_struct *owner; }; The mutex owner is denoted by owner,andwait_lock provides the actual protection. All waiting pro- cesses are enqueued on wait_list. The decisive change in contrast to regular mutexes is that the tasks on the waiter lists are sorted by priority. Whenever the waiter list changes, the kernel can consequently adjust the priority of the owner up or down. This requires an interface to the scheduler that is provided by the function rt_mutex_setprio. The function updates the dynamic priority task_struct->prio,but 363 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 364 Chapter 5: Locking and Interprocess Communication leaves the normal priority task_struct->normal_priority untouched. If you are confused by these terms, it might be a good idea to refresh yourself with the discussion of the scheduler in Chapter 2. Besides, the kernel provides several standard functions (rt_mutex_init, rt_mutex_lock, rt_mutex_unlock, rt_mutex_trylock) that work exactly as for regular mutexes and thus need not be discussed any further. 5.2.9 Approximate Per-CPU Counters Counters can become a bottleneck if a system is equipped with a large number of CPUs: Only one CPU at a time may modify the value; all other CPUs need to wait for the operation to finish until they can access the counter again. If a counter is frequently visited, this has a severe impact on system performance. For some counters, it is not necessary to know the exact value at all times. An approximation of the value serves quite as well as the proper value would do. This insight can be used to accelerate counter manipulation on SMP systems by introducing per-CPU counters. The basic idea is depicted in Figure 5-1: The proper counter value is stored at a certain place in memory, and an array with one entry for every CPU in the system is kept below the memory location of the proper value. value Per-CPU difference Figure 5-1: Data stucture for approximate per-CPU counters. If a processor wants to modify the value of the counter by adding or subtracting a number n,itdoes not perform this modification by directly changing the counter value because this would require locking out other CPUs from accessing the counter, a potentially time-consuming operation. Instead, the desired modification is stored in the CPU-specific entry of the array associated with the counter. If, for instance, the value 3 was supposed to be added to the counter, the entry +3 would be stored in the array. If the same CPU wants to substract a number (say, 5) from the counter at some other time, it also does not perform the operation directly on the counter, but on the value in the CPU-specific array: 5 is subtracted from 2, and the new value is thus −2. If any processor reads the counter value, it is not entirely accurate. If the original value was 15, then it would be 13 after the previously mentioned modifications, but is still 15. If one wants to know the value only approximately, 13 is still a good approximation to 15. If the changes in one of the CPU-specific array elements sum up to a value above or below a threshold that is considered to be large, the proper counter value is changed. In this case, the kernel needs to make sure that the access is protected by appropriate locking. But since this change now occurs only seldom, the cost of the locking operation is not so important anymore. 364 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 365 Chapter 5: Locking and Interprocess Communication As long as the counter changes only moderately, the mean value received from read operations in this scheme is quite close to the proper value of the counter. The kernel implements per-CPU counters with the help of the following data structure: struct percpu_counter { spinlock_t lock; long count; long *counters; }; count is the proper value of the counter, and lock is a spinlock to protect the counter when the exact value is required. The CPU-specific array buffering counter manipulations is given by counters. The threshold value that triggers the modification of the proper counter depends on the number of CPUs found in the system: #if NR_CPUS >= 16 #define FBC_BATCH (NR_CPUS*2) #else #define FBC_BATCH (NR_CPUS*4) #endif The following functions are available to modify approximate per-CPU counters: static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount) static inline void percpu_counter_dec(struct percpu_counter *fbc) static inline s64 percpu_counter_sum(struct percpu_counter *fbc) static inline void percpu_counter_set(struct percpu_counter *fbc, s64 amount) static inline void percpu_counter_inc(struct percpu_counter *fbc) static inline void percpu_counter_dev(struct percpu_counter *fbc) ❑ percpu_counter_add and percpu_counter_dec modify the counter by a given increment or decrement. The change is propagated to the proper counter if the accumulated changes surpass the threshold as given by FBC_BATCH. ❑ percpu_counter_read reads the current value of the counter without considering changes made by the individual CPUs. ❑ percpu_counter_inc and percpu_counter_inc are shortcuts to, respectively, increment and decrement an approximate counter by 1. ❑ percpu_counter_set sets the counter to a specific value. ❑ percpu_counter_sum computes the exact value. 5.2.10 Lock Contention and Fine-Grained Locking After having discussed the numerous locking primitives provided by the kernel, let us briefly address some of the problems related to locking and kernel scalability. While multiprocessor systems were nearly completely unknown to the average user only a decade ago, they are present on nearly every desktop 365 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 366 Chapter 5: Locking and Interprocess Communication today. Scalability of Linux on systems with more than a single CPU has therefore become a very impor- tant goal. This needs especially to be taken into account when locking rules are designed for a piece of kernel code. Locking needs to fulfill two purposes that are often hard to achieve simultaneously: 1. Code must be protected against concurrent access, which would lead to failures. 2. The impact on performance must be as little as possible. Having both things at the same time is especially complicated when data are heavily used by the kernel. Consider a really important data structure that is accessed very often — the memory management sub- system contains such structures, for instance, but also the networking code and many other components of the kernel. If the whole data structure (or, even worse, multiple data structures, or a whole driver, or awholesubsystem6) is protected by only a single lock, than chances are high that the lock is acquired by some other part of the system when one part want to get hold of it. Lock contention is said to be high in this case, and the lock becomes a hotspot of the kernel. To remedy this situation, it is customary to iden- tify independent parts of a data structure and use multiple locks to protect the elements. This solution is known as fine-grained locking. While the approach is beneficial for scalability on really big machines, it raises two other problems: 1. Taking many locks increases the overhead of an operation, especially on smaller SMP machines. 2. When a data structure is protected by multiple locks, then cases naturally arise that an oper- ation needs to access two protected regions simultaneously, and multiple locks must be held at the same time. This makes it necessary to obey a certain lock ordering, which mandates in which order locks are to be acquired and released. If not, then again deadlocks will be the result! Since the code paths through the kernel can be complicated and interwoven, it is especially hard to ensure that all cases are right. Achieving fine-grained locking for good scalability while making sure to avoid deadlocks is therefore currently among the kernel’s foremost challenges. 5.3 System V Interprocess Communication Linux uses mechanisms introduced in System V (SysV) to support interprocess communication and syn- chronization for user processes. System calls provide various routines to enable user libraries (typically the C standard library) to implement the required operations. In addition to semaphores, the SysV scheme of interprocess communication includes an option for exchanging messages and sharing memory areas between processes in accordance with a controlled pattern as described below.7 5.3.1 System V Mechanisms The three IPC mechanisms of System V Unix (semaphores, message queues, and shared memory) reflect three very different concepts but have one thing in common. They all make use of system-wide resources 6This is not so ridiculous as it may first sound, but the initial SMP-capable kernels even went one step further: After all, the big kernel lock protects all of the kernel! 7The POSIX standard has now introduced similar structures in a more modern form. I do not discuss these because most applications still use SysV mechanisms. 366 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 367 Chapter 5: Locking and Interprocess Communication that can be shared by several processes at the same time. This would seem to be logical for IPC mecha- nisms but nevertheless should not be taken for granted. For example, the mechanisms could have been designed in such a way as to ensure that only the threads of a program or a structure generated by a fork are able to access shared SysV objects. Before individual SysV elements can be accessed by various independent processes, they must be uniquely identifiable in the system. To this end, each IPC structure is assigned a number when it is created. Each program with knowledge of this magic number is able to access the corresponding structure. If independent applications are to communicate with each other, this number is usually permanently compiled into the individual programs. An alternative is to dynamically generate a magic number that is guaranteed to be unique (statically assigned numbers cannot be guaranteed to be unique). The standard library provides several functions to do this (see the relevant system programming manuals for detailed information). A privilege system based on the system adopted for file access permissions is used to access IPC objects. Each object has a user ID and a group ID that depend on the UID/GID under which the program that generated the IPC object is running. Read and write permissions are assigned at initialization. As with normal files, these govern access for three different user classes — owner, group, and others. Detailed information on how this is done is provided in the corresponding system programming manuals. The flag 0666 must be specified to create a semaphore that grants all possible access permissions (owner, group, and all other users have read and write permissions). 5.3.2 Semaphores System V semaphores are implemented using sem/sem.c in conjunction with the header file . These semaphores are not related in any way to the kernel semaphores described above. Using System V Semaphores The System V interface for using semaphores is anything but intuitive because the concept of a semaphore has been expanded well beyond its actual definition. Semaphores are no longer treated as simple variables to support atomic execution of predefined operations. Instead, a System V semaphore now refers to a whole set of semaphores, which allows not just one but several operations to be performed at the same time (although they appear to be atomic to users). It is, of course, possible to request a semaphore set with just a single semaphore and to define functions that simulate the behavior of simple operations. The following sample program shows how semaphores are used: #include #include #include #include #define SEMKEY 1234L /* Identifier */ #define PERMS 0666 /* Access permission: rwrwrw */ struct sembuf op_down[1] = { 0, -1 , 0 }; struct sembuf op_up[1] = { 0, 1 , 0 }; int semid = -1; /* Semaphore identifier */ int res; /* Result of semaphore operations */ 367 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 368 Chapter 5: Locking and Interprocess Communication void init_sem() { /* Test whether semaphore already exists */ semid = semget(SEMKEY, 0, IPC_CREAT | PERMS); if (semid < 0) { printf("Create the semaphore\n"); semid = semget(SEMKEY, 1, IPC_CREAT | PERMS); if (semid < 0) { printf("Couldn’t create semaphore!\n"); exit(-1); } /* Initialize with 1 */ res = semctl(semid, 0, SETVAL, 1); } } void down() { /* Perform down operation */ res = semop(semid, &op_down[0], 1); } void up() { /* Perform up operation */ res = semop(semid, &op_up[0], 1); } int main(){ init_sem(); /* Normal program code. */ printf("Before critical code\n"); down(); /* Critical code */ printf("In critical code\n"); sleep(10); up(); /* Remaing program code */ return 0; } A new semaphore with a permanently defined magic number (1234) is first created in main for purposes of identification within the system. Because several copies of the program are to run in parallel, it is necessary to test whether a corresponding semaphore already exists. If not, one is created. This is done using the semget system call to reserve a semaphore set. It requires the following parameters: the magic number (SEMKEY), the number of semaphores in the set (1), and the desired access permissions. The above sample program creates a semaphore set with just a single semaphore. The access permissions indicate that all users have both read and write access to the semaphore.8 Then the value of the single semaphore in the semaphore set is initialized to 1 using the semctl system call. The semid variable identifies the semaphore in the kernel (it can be obtained with the help of the magic number of any other program). 8IPC_CREAT is a system constant that must be ‘‘ORed‘‘with the access number to specify that a new semaphore is to be created. 368 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 369 Chapter 5: Locking and Interprocess Communication 0 specifies that we want to manipulate the semaphore with identifier 0 in our semaphore set (this is the only semaphore in our set). The meaning of SETVAL, 1 is obvious — the semaphore value is to be set to 1.9 The familiar up and down operations are implemented by procedures of the same name. How the semaphore value is modified in the SysV scheme is interesting. Operations are performed using the semop system call, and, as usual, the semid variable is used to identify the desired semaphore. Of particular note are the last two arguments. One is a pointer to an array with sembuf elements, each of which represents a semaphore operation. The number of operations in the array is defined by an integer argument because the kernel cannot otherwise identify the operations. Entries in the sembuf array consist of three elements with the following meanings: 1. The first entry serves to select the semaphore in the semaphore set. 2. The second entry specifies the desired operation. 0 waits until the value of the semaphore reaches 0; a positive number is added to the value of the semaphore (and corresponds to releasing a resource; the process cannot go to sleep with this action); a negative number is used to request resources. If the absolute value is less than the value of the semaphore, its (absolute) value is subtracted from the current semaphore value without sleeping on the semaphore; otherwise, the process blocks until the semaphore value reaches a value that allows the operation to be performed. 3. The third entry is a flag used for fine control of the operation. The behavior of a semaphore can be simulated by using 1 and -1 as numeric arguments. down tries to subtract 1 from the semaphore counter (and goes to sleep when the semaphore value reaches 0), while up adds 1 to the semaphore value and therefore corresponds to releasing a resource. The code yields the following result: wolfgang@meitner> ./sema Create the semaphore Before the critical code In the critical code The program creates the semaphore, enters the critical code, and waits there for 10 seconds. Before the code is entered, a down operation is performed to decrement the semaphore value to 0. A second process started during the wait period is not allowed to enter critical code. wolfgang@meitner> ./sema Before the critical code Any attempt to enter critical code triggers a down operation, which tries to subtract 1 from the semaphore value. This fails because the current value is 0. The process goes to sleep on the semaphore. It is not woken until the first process has released the resource by means of an up operation (and the semaphore value has reverted to 1). It can then decrement the semaphore value and enter the critical code. Data Structures The kernel uses several data structures to describe the current status of all registered semaphores and to build a kind of network. They are responsible not only for managing the semaphores and their 9For the sake of simplicity, we do not query for errors in our sample program. 369 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 370 Chapter 5: Locking and Interprocess Communication characteristics (value, read, and write permissions, etc.), but also for associating semaphores with waiting processes by means of a waiting list. Starting with kernel 2.6.19, the IPC mechanism is aware of namespaces (see Chapter 2 for more informa- tion about this concept). Managing IPC namespaces is, however, simple because they are not hierarchi- cally related. A given task belongs to the namespace pointed at by task_struct->nsproxy->ipc_ns,and the initial default namespace is implemented by the static ipc_namespace instance init_ipc_ns.Each namespace carries essentially the following information: struct ipc_namespace { ... struct ipc_ids *ids[3]; /* Resource limits */ ... } I have omitted a large number of elements devoted to observing resource consumption and setting resource limits. The kernel, for instance, restricts the maximum number of shared memory pages, the maximum size for a shared memory segment, the maximum number of message queues, and so on. All restrictions apply on a per-namespace basis and are documented in the manual pages msgget(2), shmget(2),andsemget(2), so I will not discuss them further here. All are implemented by simple counters. More interesting is the array ids. One array position per IPC mechanism — shared memory, semaphores, and messages — exists, and each points to an instance of struct ipc_ids that is the basis to keep track of the existing IPC objects per category. To prevent getting lost in search of the proper array index per category, the kernel provides the auxiliary functions msg_ids, shm_ids,andsem_ids. But just in case you were wondering, semaphores live in position 0, followed by message queues and then by shared memory. struct ipc_ids is defined as follows: ipc/util.h struct ipc_ids { int in_use; unsigned short seq; unsigned short seq_max; struct rw_semaphore rw_mutex; struct idr ipcs_idr; }; The first elements hold general information on the status of the IPC objects: ❑ in_use holds the number of IPC objects currently in use. ❑ seq and seq_id allow generating userspace IPC identifiers sequentially. Note that the identifiers are not identical with the sequence numbers, though. The kernel identifies an IPC object inter- nally with an identifier managed per resource type, that is, one identifier for message queues, one for semaphores, and one for shared memory objects. Each time a new IPC object is created, the sequence number is incremented by 1 (wrapping is handled automatically). 370 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 371 Chapter 5: Locking and Interprocess Communication The identifier visible to userland is given by s*SEQ_MULTIPLIER+i,wheres is the current sequence number and i is the kernel-internal identifier. The sequence multiplier is set to the upper limit for IPC objects. If an internal ID is reused, a different userspace identifier is generated this way because the sequence number is reused. This minimizes the risk of using a wrong resource when a stale ID is passed from userland. ❑ rw_mutex is a kernel semaphore. It is used to implement semaphore operations and safeguards against race conditions in userspace. The mutex appropriately protects the data structures that contain, for instance, the semaphore value. Each IPC object is represented by an instance of kern_ipc_perm towhichwecomeinamoment.Each object has a kernel-internal ID, and ipcs_idr is used to associate an ID with a pointer to the correspond- ing kern_ipc_perm instance. Since the number of used IPC objects can grow and shrink dynamically, a static array would not serve well to manage the information, but the kernel provides a radix-tree-like (see Appendix C) standard data structure in lib/idr.c for this purpose. How the entries are managed in detail is not relevant for our purposes; it suffices to know that each internal ID can be associated with the respective kern_ipc_perm instance without problems. The elements of kern_ipc_perm hold information on semaphore ‘‘owners‘‘ and on access permissions. struct kern_ipc_perm { int id; key_t key; uid_t uid; gid_t gid; uid_t cuid; gid_t cgid; mode_t mode; unsigned long seq; }; The structure can be used not only for semaphores but also for other IPC mechanisms. You will come across it frequently in this chapter. ❑ key holds the magic number used by user programs to identify the semaphore, and id is the kernel-internal identifier. ❑ uid and gid specify the user and group ID of the owner. cuid and cgid hold the same data for the process that generated the semaphore. ❑ seq is a sequence number that was used when the object was reserved. ❑ mode holds the bitmask, which specifies access permissions in accordance with the owner, group, others scheme. The above data structures are not sufficient to keep all information required for semaphores. A special per-task element is required: struct task_struct { ... #ifdef CONFIG_SYSVIPC 371 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 372 Chapter 5: Locking and Interprocess Communication /* ipc stuff */ struct sysv_sem sysvsem; #endif ... }; Note that the SysV code is only compiled into the kernel if the configuration option CONFIG_SYSVIPC is set. The sysv_sem data structure is used to encapsulate a further element. sem.h struct sysv_sem { struct sem_undo_list *undo_list; }; The only member, undo_list, is used to permit semaphore manipulations that can be undone. If a pro- cess crashes after modifying a semaphore, the information held in the list is used to return the semaphore to its state prior to modification. The mechanism is useful when the crashed process has made changes after which processes waiting on the semaphore can no longer be woken. By undoing these actions (using the information in the undo list), the semaphore can be returned to a consistent state, thus preventing deadlocks. I won’t bother with the details here, however. sem_queue is another data structure that is used to associate a semaphore with a sleeping process that wants to perform a semaphore operation but is not allowed to do so at the moment. In other words, each instance of the data structure is an entry in the list of pending operations. struct sem_queue { struct sem_queue * next; /* next entry in the queue */ struct sem_queue ** prev; /* previous entry in the queue, *(q->prev) == q */ struct task_struct* sleeper; /* this process */ struct sem_undo * undo; /* undo structure */ int pid; /* process id of requesting process */ int status; /* completion status of operation */ struct sem_array * sma; /* semaphore array for operations */ int id; /* internal sem id */ struct sembuf * sops; /* array of pending operations */ int nsops; /* number of operations */ int alter; /* does the operation alter the array? */ }; For each semaphore, there is exactly one queue that manages all sleeping processes associated with the semaphore. The queue is not implemented using standard kernel facilities but manually by means of next and prev pointers. ❑ sleeper is a pointer to the task structure of the process waiting for permission to perform a semaphore operation. ❑ pid specifies the PID of the waiting process. ❑ id holds the kernel-internal semaphore identifier. 372 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 373 Chapter 5: Locking and Interprocess Communication ❑ sops is a pointer to an array that holds the pending semaphore operations (a further data struc- ture discussed below is used to describe the operations themselves). The number of operations (i.e., the size of the array) is defined in sops. ❑ alter indicates whether the operations alter the value of the semaphore (e.g., a status query leaves the value unchanged). sma holds a pointer to an instance of the data structure used to manage the semaphore status. struct sem_array { struct kern_ipc_perm sem_perm; /* permissions .. see ipc.h */ time_t sem_otime; /* last semop time */ time_t sem_ctime; /* last change time */ struct sem *sem_base; /* ptr to first semaphore in array */ struct sem_queue *sem_pending; /* pending operations to be processed */ struct sem_queue **sem_pending_last; /* last pending operation */ struct sem_undo *undo; /* undo requests on this array */ unsigned long sem_nsems; /* no. of semaphores in array */ }; There is exactly one instance of this data structure in the system for each semaphore set. The instance is used to manage all the semaphores that make up the set. ❑ Semaphore access permissions are held in sem_perm of the familiar kern_ipc_perm type. This must be located at the beginning of the structure so that a trick can be used involving the ipc_ids->entries arrays employed to manage all semaphore sets. Because the individual elements point to areas in which sufficient memory is reserved not only for kern_ipc_perm but also for sem_array, the kernel can switch between both representations by means of typecasts. This trick is also used for other SysV-IPC objects, as you will see further below. ❑ sem_nsems specifies the number of semaphores in a user semaphore. ❑ sem_base is an array, each of whose entries describes a semaphore in the set. It holds the current semaphore value and the PID of the process that last accessed it. struct sem { int semval; /* current value */ int sempid; /* pid of last operation */ }; ❑ sem_otime specifies the time of the last access to the semaphore in jiffies (including, e.g., infor- mation queries). sem_ctime specifies the time the semaphore value was last changed. ❑ sem_pending points to a linked list of pending semaphore operations. The list consists of sem_queue instances. sem_pending_last is used to quickly access the last element in the list, whereas sem_pending points to the start of the list. Figure 5-2 shows the interplay between the data structures involved. 373 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 374 Chapter 5: Locking and Interprocess Communication ipcs_idr struct sem_array sem_pending _last sem_base sem_ pending struct kern_ipc_perm ID to pointer mapping sma sma sma sma struct task_struct struct sem_queue sembuf->sleeper struct sem[] struct ipc_ids Figure 5-2: Interplay between the semaphore data structures. Starting from the sem_ids instance obtained from the current namespace, the kernel travels via ipcs_idr to the ID-to-pointer database and looks up the required instance of kern_ipc_perm.Thekern_ipc_perm entry can be type-cast into an instance of type sem_array. The current status of the semaphore is indicated by linking with two further structures. ❑ The pending operations are managed in a linked list of sem_queue instances. Processes that are sleeping while waiting for operation execution can also be determined from this list. ❑ An array of struct sem instances is used to hold the values of the individual semaphores of the set. Not shown is the information for managing undo operations because it is not particularly interesting and would complicate matters unnecessarily. kern_ipc_perm is the first element of the data structure for managing IPC objects not only for semaphores but also for message queues and shared memory objects. It enables the kernel to use the same code to check access permissions in all three cases. Each sem_queue element contains a pointer to an array of sem_ops instances that describe in detail the operations to be performed on the semaphore. (An array of sem_ops instances is used because several operations can be performed on the semaphores in a semaphore set using a single semctl call.) struct sembuf { unsigned short sem_num; /* semaphore index in array */ short sem_op; /* semaphore operation */ short sem_flg; /* operation flags */ }; 374 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 375 Chapter 5: Locking and Interprocess Communication This definition brings to mind the sample code shown in Section 5.3.2. It is exactly the same data struc- ture used by the program to describe an operation on a semaphore. It holds not only the number of the semaphore in the semaphore set (sem_num) but also the desired operation (sem_op)andanumberof operation flags (sem_flg). Implementing System Calls All operations on semaphores are performed using a single system call named ipc.Thiscallisusednot only for semaphores but also to manipulate message queues and shared memory. Its first parameter delegates the actual multiplex work to other functions. These functions are as follows for semaphores: ❑ SEMCTL performs a semaphore operation and is implemented in sys_semctl. ❑ SEMGET reads the semaphore identifier; sys_semget is the associated implementation. ❑ SEMOP and SEMTIMEDOP are responsible for incrementing and decrementing the semaphore value; the latter enables a time-out to be specified. The use of a single system call to delegate work to multiple other functions is a relic of early days.10 Some architectures to which the kernel was later ported (e.g., IA-64 and AMD64) dispense with the implementation of the ipc multiplexer and use the above ‘‘subfunctions‘‘ directly as system calls. Older architectures like IA-32 still provide the multiplexer, but individual system calls for the variants have been added during the development of kernel 2.5. Since the implementation is generic, all architectures benefit from this. sys_semtimedop offers the functionality of sys_ipc for SEMOP and SEMTIMEDOP,and sys_semctl and sys_semget are direct implementations of SEMCTL and SEMGET, respectively. Notice that the operations that get an IPC object are, however, quickly reunited, as Figure 5-3 illustrates. This is possible because the data structures to manage IPC objects are generic and not dependent on a particular IPC type as described above. ipcget Other operations select ipd_ids instance select ipd_ids instance select ipd_ids instance Yes sys_ipc sys_shmget sys_msgget sys_semget IPC_PRIVATE? ipc_get_public ipcget_new Figure 5-3: The system calls to obtain IPC objects can be unified by a common helper function. 10The kernel comment ‘‘This is really horribly ugly‘‘ for sys_ipc does have a cause. 375 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 376 Chapter 5: Locking and Interprocess Communication Permission Checks IPC objects are protected by the same mechanisms that apply to regular file-based objects. Access rights can be separately specified for the owner of an object, the group, and all other users. Furthermore, the possible rights are reading, writing, and executing. ipcperms is responsible for checking if permissions are given for a certain operation on any of the possible IPC objects. It is defined as follows: ipc/util.c int ipcperms (struct kern_ipc_perm *ipcp, short flag) { /* flag will most probably be 0 or S_...UGO from */ int requested_mode, granted_mode, err; ... requested_mode = (flag >> 6) | (flag >> 3) | flag; granted_mode = ipcp->mode; if (current->euid == ipcp->cuid || current->euid == ipcp->uid) granted_mode >>= 6; else if (in_group_p(ipcp->cgid) || in_group_p(ipcp->gid)) granted_mode >>= 3; /* is there some bit set in requested_mode but not in granted_mode? */ if ((requested_mode & ~granted_mode & 0007) && !capable(CAP_IPC_OWNER)) return -1; return security_ipc_permission(ipcp, flag); } The requested mode (request_mode) contains the requested flags bit-triples as a threefold copy. granted_mode initially holds the mode bits of the IPC object. Depending on whether the user himself, a member of the group, or someone else wants to perform a specific operation, the contents of granted_mode are shifted to the right such that the appropriate bit-triple resides in the low three bits. If the last three bits of requested_mode and granted_mode disagree, permission is denied accordingly. securit_ipc_permission hooks into other security frameworks like SELinux, which are potentially active but need not concern us here. 5.3.3 Message Queues Another way of communicating between processes is to exchange messages. This is done using the message queue mechanism, whose implementation is based on the System V model. There are some commonalities between message queues and semaphores as far as data structures are concerned. The functional principle of messages queues is relatively simple, as Figure 5-4 shows. A C B 1 2 3 4 Figure 5-4: Functional principle of System V message queues. 376 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 377 Chapter 5: Locking and Interprocess Communication A process usually referred to as the sender generates messages and writes them to a queue, while one or more other processes (logically known as receivers) retrieve them from the queue. The individual message elements contain the message text and a (positive) number to implement several types within a message queue. Receivers can retrieve messages on the basis of the number (e.g., they can specify that they will accept only messages with the number 1 or messages up to the number 5). Once a message has been read, the kernel deletes it from the queue. Even if several processes are listening in on a channel, each message can be read by one process only. Messages with the same number are processed in FIFO order (first in, first out). Messages placed on the queue first are retrieved first. However, if messages are read selectively, the FIFO order no longer applies. Sender and receiver need not be running at the same time in order to communicate via message queues. For example, a sender process can open a queue, write messages on it, and terminate its work. A receiver process started after the sender has terminated can still access the queue and (by reference to the message number) retrieve the messages intended for it. The messages are held by the kernel in the intervening period. Message queues are also implemented using a network of data structures similar to those already dis- cussed. The starting point is the appropriate ipc_ids instance of the current namespace. Again, the internal ID numbers are formally associated with kern_ipc_perm instances, but as in the semaphore case, a different data type (struct msg_queue) is obtained as a result of type conversion. The structure is defined as follows: struct msg_queue { struct kern_ipc_perm q_perm; time_t q_stime; /* last msgsnd time */ time_t q_rtime; /* last msgrcv time */ time_t q_ctime; /* last change time */ unsigned long q_cbytes; /* current number of bytes on queue */ unsigned long q_qnum; /* number of messages in queue */ unsigned long q_qbytes; /* max number of bytes on queue */ pid_t q_lspid; /* pid of last msgsnd */ pid_t q_lrpid; /* last receive pid */ struct list_head q_messages; struct list_head q_receivers; struct list_head q_senders; }; The structure includes status information as well as queue access permissions. ❑ q_stime, q_rtime and q_ctime specify the last send, receive, and change time (if queue proper- ties are changed). ❑ q_cbytes specifies the number of bytes currently used by the messages in the queue. 377 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 378 Chapter 5: Locking and Interprocess Communication ❑ q_qbytes specifies the maximum number of bytes that may be used by the messages in the queue. ❑ q_num specifies the number of messages in the queue. ❑ q_lspid is the PID of the last sender process; q_lrpid is the PID of the last receiver process. Three standard lists of the kernel are used to manage sleeping senders (q_senders), sleeping receivers (q_receivers), and the messages themselves (q_messages). Each uses its own separate data structures as list elements. Each message in q_messages is encapsulated in an instance of msg_msg. ipc/msg.c struct msg_msg { struct list_head m_list; long m_type; int m_ts; /* message text size */ struct msg_msgseg* next; /* the actual message follows immediately */ }; m_list is used as a list element to link the individual messages; the other elements are used to manage the message itself. ❑ m_type specifies the message type and is used as described to support several types per queue. ❑ m_ts specifies the message text size in bytes. ❑ next is needed to hold long messages requiring more than one memory page. There is no explicit field in which the message itself is stored. Because (at least) one page is always reserved for each message and the msg_msg instance is held at the beginning of this page, the remaining space can be used to store the message text, as shown in Figure 5-5. Page frame struct msg_msg next nextstruct msg_msg_seq struct msg_msg_seq Message text Figure 5-5: Managing an IPC message in memory. The maximum number of bytes available for the message text in a msg_msg page is calculated by sub- tracting the size of the structure from the size of a memory page. ipc/msgutils.c #define DATALEN_MSG (PAGE_SIZE-sizeof(struct msg_msg)) 378 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 379 Chapter 5: Locking and Interprocess Communication Longer messages must be spread over several pages with the help of the next pointer. This points to an instance of msg_msgseq that is also situated at the beginning of a page, as shown in Figure 5-5. It is defined as follows: ipc/msgutils.c struct msg_msgseg { struct msg_msgseg* next; /* the next part of the message follows immediately */ }; Again, the message text immediately follows the data structure instance. next is used to enable the message to be spread over any number of pages. Both sender and receiver processes can go to sleep when communicating via message queues — senders while they are attempting to write a message on a queue whose maximum capacity has already been reached; receivers when they want to retrieve a message from a queue although none has arrived. Sleeping senders are placed on the q_senders list of msg_queue using the following data structure: ipc/msg.c struct msg_sender { struct list_head list; struct task_struct* tsk; }; list is a list element, and tsk is a pointer to the corresponding task structure. No additional information is required because the sender process goes to sleep during the sys_msgsnd system call — which can also be activated via sys_ipc — used to send the message and automatically repeats the send operation when it is woken. The data structure to hold the receiver process in the q_receivers list is a little longer. ipc/msg.c struct msg_receiver { struct list_head r_list; struct task_struct *r_tsk; int r_mode; long r_msgtype; long r_maxsize; struct msg_msg *volatile r_msg; }; It holds not only a pointer to the corresponding task structure but also the descriptors of the expected message (above all, the message type r_msgtype) and a pointer to a msg_msg instance into which the data are copied if available. Figure 5-6 illustrates the interplay of the message queue data structures (for the sake of clarity, the list of sleeping senders is not shown). 379 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 380 Chapter 5: Locking and Interprocess Communication q_messages q_receivers q_senders struct msg_receiver struct task_struct struct msg_msgipcs_idr struct msg_queue struct kern_ipc_perm ID to pointer mapping struct ipc_ids Figure 5-6: Data structures for System V message queues. 5.3.4 Shared Memory From the user and kernel perspective, shared memory — the third and final SysV concept for interprocess communication — uses similar structures for its implementation to the first two mechanisms described above. Its essential aspects do not differ from those of semaphores and message queues. ❑ Applications request an IPC object that can be accessed via a common magic number and a kernel-internal identifier via the current namespace. ❑ Access to memory can be restricted by means of a system of privileges. ❑ System calls are used to allocate memory that is associated with the IPC object and that can be accessed by all processes with the appropriate authorization. Kernel-side implementation also adopts very similar concepts to those described above. I will therefore make do with a brief description of the data structures shown in Figure 5-7. shm_file shmid_kernel struct address_ space struct file ipcs_idr ID to pointer mapping struct ipc_ids kern_ipc_perm Figure 5-7: Data structures for System V shared memory. Once again, a combination of kern_ipc_perm and shmid_kernel held in the entries array of the smd_ids global variable is used to facilitate management of the access permissions to the IPC object. A dummy 380 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 381 Chapter 5: Locking and Interprocess Communication file linked with the corresponding instance of shmid_kernel via shm_file is created for each shared memory object. The kernel uses the pointer smh_file->f_mapping to access the address space object (struct address_space) used to create anonymous mappings as described in Chapter 4. The page tables of the processes involved are set up so that each process is able to access the areas linked with the region. 5.4 Other IPC Mechanisms There are other traditional ways of exchanging messages and data between processes in addition to the IPC mechanisms adopted from System V Unix. Whereas the SysV options are primarily of interest to application programmers, practically all users who have ever worked with the shell will know signals and pipes. 5.4.1 Signals Signals are an older form of communication than the SysV mechanisms. Although they provide fewer options, they are generally very suited to their purpose. The underlying concept is very simple — the kill command sends a signal to a process identified by its PID. The number of the signal is specified using -s sig and is a positive integer whose maximum size varies depending on processor type. The two most frequently used variants of the command are kill without a signal number, which politely requests the process to terminate (the process is free to ignore the signal), and kill -9, which is the equivalent of a signature on an execution warrant (and results in certain death). In the past, 32-bit systems supported a maximum of 32 signals, but this restriction has now been lifted, and all signals listed on the kill manual page are now supported. Nevertheless, all the ‘‘classic‘‘ sig- nals occupy the first 32 positions on the list. They are followed by new signals introduced for real-time processes. Processes must install handler routines to process signals. These are invoked when signals are sent to the processes (but there are several signals such as SIGKILL whose behavior cannot be overridden). If no explicit handler routine is installed, the kernel uses a default handler implementation. Signals introduce several special features that must always be kept in mind. A process can decide to block specific signals (sometimes referred to as the masking of signals). If this happens, the signal is ignored until the process decides to remove the block. There is therefore no guarantee that a process will be aware that a signal has been sent to it. When a signal is blocked, the kernel places it on a pending list. If the same signal is blocked more than once, only a single occurrence of the signal is placed on the pending list. No matter how many identical signals are sent, the process receives just one occurrence of the signal when it removes the block. The SIGKILL signal cannot be blocked and cannot be handled by a process-specific handler function. It cannot be overridden because it is the last resort to remove an out-of-control process from the system. This contrasts with the SIGTERM signal, which can be dealt with by a user-defined signal handler — after all, the signal is just a polite request to the process to stop work as soon as possible. If a handler is installed for this signal, the program is, for example, given the opportunity to save data or to ask users whether they really want to exit the program. SIGKILL does not provide such opportunities because the kernel brings the process to an immediate and abrupt end. The init process is granted a special status. The kernel ignores any SIGKILL signals sent to it. Because this process is of particular importance to the entire system, it may not be forced to terminate — not even unintentionally. 381 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 382 Chapter 5: Locking and Interprocess Communication Implementing Signal Handlers The sigaction system call is used to install a new handler function. #include #include /* Handler function */ void handler(int sig) { printf("Receive signal: %u\n", sig); }; int main(void) { struct sigaction sa; int count; /* Initialize the signal handler structure */ sa.sa_handler = handler; sigemptyset(&sa.sa_mask); sa.sa_flags = 0; /* Assign a new handler function to the SIGTERM signal */ sigaction(SIGTERM, &sa, NULL); sigprocmask(&sa.sa_mask); /* Accept all signals */ /* Block and wait until a signal arrives */ while (1) { sigsuspend(&sa.sa_mask); printf("loop\n"); } return 0; }; If no user-specific handler function is assigned to a signal, the kernel automatically installs predefined functions to provide reasonable standard operations and to deal with the specific situation. The definition of the field of type sigaction used to describe the new handler is platform-specific but has practically the same contents on all architectures. struct sigaction { __sighandler_t sa_handler; unsigned long sa_flags; ... sigset_t sa_mask; /* mask last for extensibility */ }; ❑ sa_handler is a pointer to the handler function invoked by the kernel when a signal arrives. ❑ sa_mask contains a bitmask with exactly one bit for each signal available in the system. It is used to block other signals during execution of the handler routine. On completion of the routine, the kernel resets the list of blocked signals to its value prior to signal handling. ❑ sa_flags contains additional flags to specify how the signal must be handled; these are docu- mented in various system programming manuals. 382 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 383 Chapter 5: Locking and Interprocess Communication The prototype of functions that act as signal handlers is as follows: typedef void __signalfn_t(int); typedef __signalfn_t __user *__sighandler_t; The parameter accepts the number of the signal received so that the same handler function can be installed for different signals.11 The signal handler is installed using the sigaction system call, which (in our example) replaces the default handler for SIGTERM with the user-defined handler function. Processes can set a global mask to specify which signals are to be blocked while the handler is running. A bit chain is used to indicate that a signal is either blocked (bit value 1) or not blocked (bit value 0). The sample program sets all bit positions to 0 so that all signals sent to the process from the outside can be received while the handler is running. The last step in the program is to wait for a signal using the sigsuspend system call. The process is placed in the blocked state (see Chapter 2) and sleeps until woken by the arrival of a signal; it is then immediately put to sleep again (by the while loop). The main code need not concern itself with signal handling because this is done automatically by the kernel in conjunction with the handler function. The approach shown is a good example of how to avoid the deprecated practice of busy waiting.12 If the SIGTERM signal is sent to the process using kill,theprocessisnotterminatedasitnormallywould be; instead, it outputs the number of the received signal (15) and continues to run because, as desired, the signal was forwarded to the user-defined handler routine and not to the default implementation of the kernel. Implementing Signal Handling All signal-related data are managed with the help of a linked data structure consisting of several C structures. Its entry point is the task_struct task structure, which includes various signal-relevant fields. struct task_struct { ... /* signal handlers */ struct signal_struct *signal; struct sighand_struct *sighand; sigset_t blocked; struct sigpending pending; unsigned long sas_ss_sp; size_t sas_ss_size; ... }; 11Another version that passes more information exists for handler functions used with POSIX real-time signals. 12Instead of repeatedly running through an empty loop to wait for a signal (a senseless waste of CPU time because the process is always running in this approach), the program can happily devote itself to doing nothing without burdening the CPU — the kernel automatically wakes the process when the signal arrives and can use CPU time more profitably in the meantime. 383 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 384 Chapter 5: Locking and Interprocess Communication Although signal handling takes place in the kernel, the installed signal handlers run in user mode — otherwise, it would be very easy to introduce malicious or faulty code into the kernel and undermine the system security mechanisms. Generally, signal handlers use the user mode stack of the process in question. However, POSIX mandates the option of running signal handlers on a stack set up specifically for this purpose (using the sigaltstack system call). The address and size of this additional stack (which must be explicitly allocated by the user application) are held in sas_ss_sp and sas_ss_size, respectively.13 The sighand element with the following structure is used to manage information on the signal handlers installed. The underlying structure is essentially defined as follows: struct sighand_struct { atomic_t count; struct k_sigaction action[_NSIG]; }; count holds the number of processes that share the instance.AsdescribedinChapter2,itispossibleto specify in the clone operation that parent and child process share the same signal handler so that there is no need to copy the data structure. The installed signal handlers are held in the action array that has _NSIG elements. _NSIG specifies the number of different signals that can be handled. This figure is 64 on most platforms, but there are exceptions — Mips, for instance, which supports 128 signals. Each element contains an instance of the k_sigaction structure to specify the properties of a signal as seen by the kernel. On some platforms, the kernel has more information on signal handlers than is available for userspace applications. Normally, k_sigaction has a single element that includes the familiar sigaction structure. struct k_sigaction { struct sigaction sa; }; If no user-defined handler routine is installed for a signal (this means that the default routine is used instead), the sa.sa_handler flag is set to SIG_DFL. In this case, the kernel performs one of four standard actions depending on the signal type: ❑ Ignore — Nothing happens. ❑ Terminate — Terminates the process or process group. ❑ Stop — Places the process in the TASK_STOPPED state. ❑ Core Dump — Creates a core dump of the address space and writes it to a core file for process- ing, for example, by a debugger. 13Signal handlers that use this stack must be installed using the SA_ONSTACK flag. Since this mechanism is rarely used, I will not bother discussing it here. 384 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 385 Chapter 5: Locking and Interprocess Communication Table 5-2 shows which signals are assigned to which default handler. The corresponding infor- mation can be obtained from the macros SIG_KERNEL_ONLY_MASK, SIG_KERNEL_COREDUMP_ MASK, SIG_KERNEL_IGNORE_MASK,andSIG_KERNEL_STOP_MASK in . Table 5-2: Default Actions for Standard Signals Action Signals Ignore SIGCONT, SIGCHLD, SIGWINCH, SIGURG Terminate SIGHUP, SIGINT, SIGKILL, SIGUSR1, SIGUSR2, SIGALRM, SIGTERM, SIGVTALRM, SIGPROF, SIGPOLL, SIGIO, SIGPWR and all real-time signals. Stop SIGSTOP, SIGTSTP, SIGTTIN, SIGTTOU Core dump SIGQUIT, SIGILL, SIGTRAP, SIGABRT, SIGBUS, SIGFPE, SIGSEGV, SIGXCPU, SIGXFSZ, SIGSYS, SIGXCPU, SIGEMT All blocked signals are defined by the blocked element of the task structure. The sigset_t data type used is a bitmask that must contain (at least) as many positions as the number of signals supported. For this purpose, the kernel uses an array of unsigned longs whose size is calculated on the basis of _NSIG and _NSIG_BPW (bits per word). #define _NSIG 64 #define _NSIG_BPW 32 #define _NSIG_WORDS (_NSIG / _NSIG_BPW) typedef struct { unsigned long sig[_NSIG_WORDS]; } sigset_t; pending is the final task structure element of relevance for signal handling. It creates a linked list of all signals raised and still to be handled by the kernel. The following data structure is used: struct sigpending { struct list_head list; sigset_t signal; }; list manages all pending signals in a doubly linked list, while signal, with the bitmask described above, specifies the numbers of all signals still to be handled. The list elements are instances of type sigqueue, which is essentially defined as follows: struct sigqueue { struct list_head list; siginfo_t info; }; The individual entries are linked by means of list.Thesiginfo_t data structure contains more detailed information on the pending signals. 385 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 386 Chapter 5: Locking and Interprocess Communication typedef struct siginfo { int si_signo; int si_errno; int si_code; union { /* Signal-specific information */ struct { ... } _kill; struct { ... } _timer; /* POSIX.1b timers */ struct { ... } _rt; /* POSIX.1b signals */ struct { ... } _sigchld; struct { ... } _sigfault; /* SIGILL, SIGFPE, SIGSEGV, SIGBUS */ struct { ... } _sigpoll; } _sifields; } siginfo_t; ❑ si_signo holds the signal number. ❑ si_errno has a non-zero value if the signal was raised as a result of an error; otherwise, its value is 0. ❑ si_code returns detailed information on the origin of the signal; we are interested only in the distinction between user signal (SI_USER) and kernel-generated signal (SI_KERNEL). ❑ Additional information needed by the kernel to handle some signals is held in the _sifield union. For example, _sigfault contains the userspace address of the instruction that raised the signal. Because a very large number of data structures are used, Figure 5-8 gives an overview of how they are interlinked. pending sighand task_struct count k_sigaction[_NSIG] list_head list sigset_t signal Figure 5-8: Data structures used in signal management. Implementing Signal Handling Table 5-3 shows an overview of the most important system calls used by the kernel to implement signal handling. In practice, there are a few more, some for historical reasons, some to ensure compatibility with various standards — above all, POSIX. Although the signal mechanism appears to be very simple, its implementation is made more complicated by the many subtleties and details that must be taken into account. Because these reveal no significant 386 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 387 Chapter 5: Locking and Interprocess Communication information on the implementation structure, I will not discuss specific cases but restrict myself to a description of the key mechanisms. Table 5-3: Some System Calls Relating to Signals System call Function kill Sends a signal to all processes of a process group. tkill Sends a signal to a single process. sigpending Checks whether there are pending signals. sigprocmask Manipulates a bitmask of blocked signals. sigsuspend Sleeps until a specific signal is received. Sending Signals Despite their names, kill and tkill send any signal to a process group or to a single process, respec- tively. As both functions are basically identical,14 I discuss only sys_tkill, whose code flow diagram is shown in Figure 5-9. sys_tkill Cancel processing do_tkill find_task_by_vpid check_kill_permission specific_send_sig_info sig_ignored? send_signal signal_wake_up Figure 5-9: Code flow diagram for sys_tkill. 14sys_kill sends the signal to several processes that are interpreted according to the form of the PID passed. ❑ pid > 0 sends the signal to the process with the specified PID. ❑ pid = 0 sends the signal to all members of the process group of the task that sent the signal. ❑ pid =−1 sends the signal to all processes with pid > 1. ❑ pid =−pgrp < −1 sends the signal to all members of the pgrp process group. 387 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 388 Chapter 5: Locking and Interprocess Communication Once find_task_by_vpid has found the task structure of the target process, the kernel dele- gates the job of checking whether the process has the permissions needed to send the signal to check_kill_permission, which uses the following query: kernel/signal.c static int check_kill_permission(int sig, struct siginfo *info, struct task_struct *t) { ... if ((info == SEND_SIG_NOINFO || (!is_si_special(info) && SI_FROMUSER(info))) && ((sig != SIGCONT) || (task_session_nr(current) != task_session_nr(t))) && (current->euid ^ t->suid) && (current->euid ^ t->uid) && (current->uid ^ t->suid) && (current->uid ^ t->uid) && !capable(CAP_KILL)) return -EPERM; ... } It could be helpful to remember that the ^ operator implements an XOR operation, but otherwise the checks are rather straightforward. The remaining signal handling work is passed on to specific_send_sig_info. ❑ If the signal is blocked (this can be checked with sig_ignored), handling is aborted immediately to prevent further waste of time. ❑ send_signal generates a new sigqueue instance (using the cache sigqueue_cachep), which is filled with the signal data and added to the sigpending list of the target process. ❑ If the signal is delivered successfully and is not blocked, the process is woken with signal_wake_up so that it is available for selection by the scheduler. The TIF_SIGPENDING flag is also set to indicate to the kernel that it must deliver signals to the process. Although the signal is sent after these actions, it does not trigger the signal handler. How this is done is described below. Processing the Signal Queue Signal queue processing is not triggered by a system call but is initiated by the kernel each time a switch is made from kernel mode to user mode, as mentioned in Chapter 14. Implementation is naturally very architecture-specific because processing is initiated in the assembly language code of entry.S. Regardless of the particular architecture, the ultimate effect of the actions performed is to invoke the do_signal function, which, although also platform-specific, behaves in much the same way on all systems. ❑ get_signal_to_deliver gathers all information on the next signal to be delivered. It also removes the signal from the process-specific pending list. ❑ handle_signal manipulates the user mode stack of the process so that the signal handler is run and not the normal program code after switching from kernel to user mode. This complicated approach is necessary because the handler function may not be executed in kernel mode. The stack is also modified so that the sigreturn system call is invoked when the handler function terminates. How this is done depends on the particular architecture, but the kernel 388 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 389 Chapter 5: Locking and Interprocess Communication either writes the required machine code instructions to execute the system call directly onto the stack, or uses some glue that is available in userspace.15 This routine is responsible for restoring the process context so that the application can continue to run when the next switch is made to user mode. Figure 5-10 illustrates the chronological flow and the various switches between user and kernel mode during signal handler execution. Kernel User Handler Function Continue program execution do_signal handle_signal sys_sigreturn Figure 5-10: Signal handler execution. 5.4.2 Pipes and Sockets Pipes and sockets are popular interprocess communication mechanisms. I provide only an overview of how both concepts work because both make intense use of other kernel subsystems. Pipes use objects of the virtual filesystem, while sockets use various network functions and also the virtual filesystem. Shell users are familiar with pipes in command lines such as wolfgang@meitner> prog | ghostscript | lpr - which use the output of one process as input for another process, the pipe being responsible for data transport. As the name suggests, a pipe is a connection used to exchange data. One process feeds data into one end of the pipe, and another takes the data out at the other end for further processing. Several processes can be joined together by a series of individual pipes. When pipes are generated via the shell, there is always one read and one write process. Applications must invoke the pipe system call to generate pipes. The call returns two file descriptors — one for the read end and one for the write end of the pipe. Because both descriptors exist in the same process, the process can initially only send messages to itself, and this is not very practical. Pipes exist as a data object in process address space — and are retained when a process is duplicated with fork or clone. It is this very characteristic that programs exploit. Once the exec system call has replaced the child process with another program, there is a communication link between two different applications (the pipe descriptors must be directed to standard input and output or the dup system call must be invoked to ensure that the file descriptors are not closed when exec is called). Sockets are also objects that return a file descriptor when initialized in the kernel and can then be handled as normal files. However, unlike pipes, they can be used bidirectionally and to communicate with remote 15On IA-32 machines, this glue code can, for instance, be found in the vsyscall page, which is mapped into every user address space. It assists the C standard library in finding the fastest way to perform system calls on a given machine by providing the required machine code instructions. The kernel decides at boot time which method is used best, and maps the page into the address space of each userland process. The page also includes the required code to execute the aforementioned sigreturn system call. 389 Mauerer runc05.tex V3 - 09/04/2008 4:59pm Page 390 Chapter 5: Locking and Interprocess Communication systems connected via a network (this does not mean that they cannot be used to support communication between two processes located on the same system). Socket implementation is one of the rather more complex parts of the kernel because extensive abstraction mechanisms are needed to hide the details of communication. From a user point of view, there is no great difference whether communication is between two local processes on the same system or between applications running on computers located in different continents. The implementation of this amazing mechanism is discussed in depth in Chapter 12. During the development of kernel 2.6.26, the architecture-specific implementation of semaphores has been replaced with a generic variant. Naturally, the generic implementation performs slightly less efficient than optimized code, but since semaphores are not in very widespread use across the kernel (mutexes are much more common), this does not really matter. The definition of struct semaphore has been moved to include/linux/semaphore.h, and all operations are implemented in kernel/semaphore.c. Most importantly, the semaphore API has not changed, so existing code will run without modifications. Another change introduced during the development of kernel 2.6.26 concerns the implementation of spinlocks. Because these locks are by definition supposed to be uncontended in the average case, the kernel did not make any efforts to achieve fairness among multiple waiters, i.e., the order in which tasks waiting for a spinlock to become unlocked are allowed to run after the lock is released by the current holder was undefined. Measurements have, however, shown that this can lead to unfairness problems on machines with a larger number of processors, e.g., 8-CPU systems. Since machines of this kind are not uncommon anymore nowadays, the implementation of spinlocks has been changed such that the order in which multiple waiters are allowed to obtain the lock is the same order in which they arrived. The API was also left unchanged in this case, so existing code will again run without modifications. 5.5 Summary While systems with more than one CPU were oddities only a few years ago, recent achievements of semiconductor engineering have changed this drastically. Thanks to multi-core CPUs, SMP computers are not only found in specialized niches like number crunching and supercomputing, but on the average desktop. This creates some unique challenges for the kernel: More than one instance of the kernel can run simultaneously, and this requires coordinated manipulation of shared data structures. The kernel provides a whole set of possibilities for this purpose, which I have discussed in this chapter. They range from simple and fast spinlocks to the powerful read-copy-update mechanism, and allow for ensuring correctness of parallel operations while preserving performance. Choosing the proper solution is impor- tant, and I have also discussed the need to select an appropriate design that ensures performance by fine-grained locking, but does not lead to too much overhead on smaller machines. Similar problems as in the kernel arise when userland tasks communicate with each other. Besides pro- viding means that allow otherwise separated processes to communicate, the kernel must also make means of synchronization available to them. I have discussed how the mechanisms originally invented in System V Unix are implemented in the Linux kernel. 390 Mauerer runc06.tex V2 - 09/04/2008 5:03pm Page 391 Device Drivers Device drivers are a key area of the kernel as many users judge operating system performance primarily by the number of peripherals for which drivers are available and how effectively they are supported. Consequently, large parts of the kernel sources are devoted to the implementation of device drivers. Device drivers build on many different mechanisms provided by the central kernel (this is why drivers are sometimes referred to as kernel ‘‘applications‘‘). The immense number of drivers in the Linux kernel means that it is impossible to discuss all (or even a few) in detail. Fortunately, this is not necessary. The structures of the drivers are generally very similar — regardless of device — so that in this chapter we need only discuss a few key aspects common to all drivers. Since the objective of this book is to cover all important parts of the kernel, this chapter omits some of the more specific points of driver writing which would require a book of its own. However, two books that focus solely on driver writing are currently available. The classic text in this area is Linux Device Drivers by Corbet et al. [CRKH05]. We can recommend it wholeheartedly to anyone interested in or charged with writing a device driver. A recent addition to kernel hackers’ bookshelves is Essential Linux Device Drivers by Venkateswaran [Ven08]. Developers who are able to read German will certainly also enjoy Linux Ger¨atetreiber by Quade and Kunst [QK06]. The quoted references are complementary to this book. Here, we document how the kernel sets up and manages data structures and generic infrastructure for device drivers. Also, we discuss routines that are provided to support device drivers. Device driver books, on the other hand, focus on how to use these routines to actually create new drivers, but are not so much interested in how the underlying foundations are implemented. 6.1 I/O Architecture Communication with peripherals is usually referred to as input and output, abbreviated I/O in the literature. The kernel must deal with three problem areas when implementing I/O for peripherals. Firstly, the hardware must be addressed using a variety of methods depending on the specific Mauerer runc06.tex V2 - 09/04/2008 5:03pm Page 392 Chapter 6: Device Drivers device type and model. Secondly, the kernel must provide user applications and system tools with ways of accessing the various devices. Wherever possible, a uniform scheme should be adopted to keep programming effort within limits and to ensure that applications are able to interoperate regardless of the particular hardware approach. Thirdly, userspace needs to know which devices are available in the kernel. Communication with peripherals is layered as illustrated in Figure 6-1. User Kern VFS Driver Hardware Application /dev/xyz Device special file open, read, write etc. Figure 6-1: Layer model for addressing peripherals. Access to each individual device is performed via abstraction layers arranged hierarchically. At the bottom of the hierarchy is the device itself, which is attached to other devices and the system CPU by means of a bus system. Communication with the kernel follows this path. Before we examine the relevant algorithms and structures in the Linux kernel, it is worthwhile taking a brief look at how expansion hardware generally works. For detailed descriptions, readers are referred to hardware-specific publications such as [MD03]. 6.1.1 Expansion Hardware Hardware devices may be attache