Perl 自动化系统管理

Automating System Administration with Perl SECOND EDITION Automating System Administration with Perl David N. Blank-Edelman Beijing • Cambridge • Farnham • Köln • Sebastopol • Taipei • Tokyo Automating System Administration with Perl, Second Edition by David N. Blank-Edelman Copyright © 2009 O’Reilly Media, Inc. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles ( For more information, contact our corporate/institutional sales department: 800-998-9938 or Editor: Andy Oram Production Editor: Sarah Schneider Copyeditor: Rachel Head Proofreader: Kiel Van Horn Indexer: Lucie Haskins Cover Designer: Karen Montgomery Interior Designer: David Futato Illustrator: Robert Romano Printing History: May 2009: Second Edition. O’Reilly and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Automating System Administration with Perl, the image of a sea otter, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions, or for damages resulting from the use of the information con- tained herein. ISBN: 978-0-596-00639-6 [M] 1241809111 To Cindy, ever the love of my life, and to Elijah, a true blessing. Table of Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Automation Is a Must 1 How Perl Can Help You 2 This Book Will Show You How 3 What You Need 5 Some Notes About the Perl Versions Used for This Book 6 What About Perl 5.10? 6 What About Strawberry Perl? 6 What About Perl 6? 6 Some Notes About Using Vista with the Code in This Book 7 Locating and Installing Modules 8 Installing Modules on Unix 9 Installing Modules on Win32 9 It’s Not Easy Being Omnipotent 10 Don’t Do It 10 Drop Your Privileges As Soon As Possible 10 Be Careful When Reading Data 11 Be Careful When Writing Data 12 Avoid Race Conditions 12 Enjoy 13 References for More Information 13 2. Filesystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Perl to the Rescue 15 Filesystem Differences 16 Unix 16 Windows-Based Operating Systems 16 Mac OS X 18 Filesystem Differences Summary 19 Dealing with Filesystem Differences from Perl 19 vii Walking or Traversing the Filesystem by Hand 21 Walking the Filesystem Using the File::Find Module 26 Walking the Filesystem Using the File::Find::Rule Module 36 Manipulating Disk Quotas 38 Editing Quotas with edquota Trickery 40 Editing Quotas Using the Quota Module 44 Editing NTFS Quotas Under Windows 45 Querying Filesystem Usage 46 Module Information for This Chapter 48 References for More Information 48 3. User Accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Unix User Identities 50 The Classic Unix Password File 50 Changes to the Password File in BSD 4.4 Systems 57 Shadow Passwords 58 Windows-Based Operating System User Identities 59 Windows User Identity Storage and Access 59 Windows User ID Numbers 61 Windows Passwords Don’t Play Nice with Unix Passwords 63 Windows Groups 63 Windows User Rights 68 Building an Account System to Manage Users 71 The Backend Database 73 The Low-Level Component Library 78 The Process Scripts 89 Account System Wrap-Up 94 Module Information for This Chapter 97 References for More Information 97 Unix Password Files 97 Windows User Administration 98 4. User Activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Process Management 100 Windows-Based Operating System Process Control 100 Unix Process Control 119 File and Network Operations 125 Tracking File Operations on Windows 125 Tracking Network Operations on Windows 128 Tracking File and Network Operations in Unix 129 Module Information for This Chapter 135 Installing Win32::Setupsup 135 References for More Information 136 viii | Table of Contents 5. TCP/IP Name and Configuration Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 Host Files 137 Generating Host Files 140 Error-Checking the Host File Generation Process 143 Improving the Host File Output 144 Incorporating a Source Code Control System 148 NIS, NIS+, and WINS 151 NIS+ 154 Windows Internet Name Server (WINS) 154 Domain Name Service (DNS) 155 Generating DNS (BIND) Configuration Files 156 DNS Checking: An Iterative Approach 165 DHCP 174 Active Probing for Rogue DHCP Servers 176 Monitoring Legitimate DHCP Servers 181 Module Information for This Chapter 183 References for More Information 184 6. Working with Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 Configuration File Formats 188 Binary 188 Naked Delimited Data 189 Key/Value Pairs 190 Markup Languages 192 All-in-One Modules 235 Advanced Configuration Storage Mechanisms 236 Module Information for This Chapter 236 References for More Information 237 XML and YAML 237 7. SQL Database Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 Interacting with a SQL Server from Perl 240 Using the DBI Framework 243 Using ODBC from Within DBI 249 Server Documentation 251 MySQL Server via DBI 252 Oracle Server via DBI 254 Microsoft SQL Server via ODBC 255 Database Logins 258 Monitoring Space Usage on a Database Server 260 Module Information for This Chapter 263 References for More Information 263 DBI 263 Table of Contents | ix Microsoft SQL Server 264 ODBC 264 Oracle 264 8. Email . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 Sending Mail 265 Getting sendmail (or a Similar Mail Transport Agent) 266 Using the OS-Specific IPC Framework to Drive a Mail Client 266 Speaking the Mail Protocols Directly 268 Common Mistakes in Sending Email 273 Overzealous Message Sending 273 Subject Line Waste 282 Insufficient Information in the Message Body 282 Fetching Mail 285 Talking POP3 to Fetch Mail 285 Talking IMAP4rev1 to Fetch Mail 287 Processing Mail 291 Dissecting a Single Message 291 Dissecting a Whole Mailbox 296 Dealing with Spam 297 Support Mail Augmentation 305 Module Information for This Chapter 310 References for More Information 311 9. Directory Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313 What’s a Directory? 313 Finger: A Simple Directory Service 314 The WHOIS Directory Service 318 LDAP: A Sophisticated Directory Service 321 LDAP Programming with Perl 322 The Initial LDAP Connection 323 Performing LDAP Searches 325 Entry Representation in Perl 329 Adding Entries with LDIF 331 Adding Entries with Standard LDAP Operations 333 Deleting Entries 334 Modifying Entry Names 335 Modifying Entry Attributes 337 Deeper LDAP Topics 339 Putting It All Together 348 Active Directory Service Interfaces 354 ADSI Basics 355 Using ADSI from Perl 357 x | Table of Contents Dealing with Container/Collection Objects 359 Identifying a Container Object 360 So How Do You Know Anything About an Object? 360 Searching 363 Performing Common Tasks Using the WinNT and LDAP Namespaces 366 Working with Users via ADSI 367 Working with Groups via ADSI 369 Working with File Shares via ADSI 369 Working with Print Queues and Print Jobs via ADSI 370 Working with Windows-Based Operating System Services via ADSI 371 Module Information for This Chapter 373 References for More Information 373 LDAP 373 ADSI 374 10. Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377 Reading Text Logs 377 Reading Binary Log Files 378 Using unpack() 378 Calling an OS (or Someone Else’s) Binary 383 Using the OS’s Logging API 384 Structure of Log File Data 385 Dealing with Log File Information 388 Space Management of Logging Information 388 Log Parsing and Analysis 395 Writing Your Own Log Files 425 Logging Shortcuts and Formatting Help 425 Basic/Intermediate Logging Frameworks 426 Advanced Logging Framework 428 Module Information for This Chapter 429 References for More Information 430 11. Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 Noticing Unexpected or Unauthorized Changes 434 Local Filesystem Changes 434 Changes in Data Served Over the Network 440 Noticing Suspicious Activities 442 Local Signs of Peril 442 Finding Problematic Patterns 444 Danger on the Wire, or “Perl Saves the Day” 449 Preventing Suspicious Activities 460 Suggest Better Passwords 460 Reject Bad Passwords 461 Table of Contents | xi Module Information for This Chapter 466 References for More Information 467 12. SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469 Using SNMP from Perl 469 Sending and Receiving SNMP Traps, Notifications, and Informs 480 Alternative SNMP Programming Interfaces 484 Module Information for This Chapter 486 References for More Information 486 13. Network Mapping and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489 Network Mapping 489 Discovering Hosts 490 Discovering Network Services 499 Physical Location 501 Presenting the Information 503 Textual Presentation Tools 503 Graphical Presentation Tools 507 Monitoring Frameworks 522 Extending Existing Monitoring Packages 524 What’s Left? 526 Module Information for This Chapter 527 References for More Information 527 14. Experiential Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529 Playing with Timelines 530 Task One: Parsing crontab Files 530 Task Two: Displaying the Timeline 531 Task Three: Writing Out the Correct XML File 533 Putting It All Together 534 Summary: What Can We Learn from This? 536 Playing with Geocoding 537 Geocoding from Postal Addresses 537 Geocoding from IP Addresses 541 Summary: What Can We Learn from This? 544 Playing with an MP3 Collection 544 Summary: What Can We Learn from This? 546 One Final Exploration 546 Part One: Retrieving the Wiki Page with WWW::Mechanize 547 Part Two: Extracting the Data 550 Part Three: Geocoding and Mapping the Data 551 Summary: What Can We Learn from This? 554 Remember to Play 555 xii | Table of Contents Module Information for This Chapter 555 Source Material for This Chapter 556 A. The Eight-Minute XML Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557 B. The 10-Minute XPath Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563 C. The 10-Minute LDAP Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573 D. The 15-Minute SQL Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 579 E. The Five-Minute RCS Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593 F. The Two-Minute VBScript-to-Perl Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597 G. The 20-Minute SNMP Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617 Table of Contents | xiii Preface Do you need tools for making your system administration work easier and more effi- cient? You’ve come to the right place. Perl is a powerful programming language that grew out of the traditional system administration toolbox. Over the years it has adapted and expanded to meet the chal- lenges of new operating systems and new tasks. If you know a little Perl, and you need to perform system administration tasks, this is the right book for you. Readers with varying levels of both Perl programming experience and system administration expe- rience will all find something of use within these pages. What’s New in This Edition? A tremendous amount of work went into updating this book so it could be even better than the first edition. Here’s some of what has been improved in the second edition: New title My editors and I realized that the material in this book was more about how to automate your system administration work in ways that would make your working life more efficient and pleasant than it was about Perl. While Perl is still the toolshed that makes all this possible, it isn’t the main focus of the book. New material It’s hard to know where to begin on this one. The new edition is four chapters and two appendixes bigger (with a total page count that is 50% greater) than the last one. Included in this edition are a cornucopia of new tools and techniques that you are going to love. I tried to add material on the things I wished I had sysadmin- targeted material on, including: XML and YAML best practices (using XML::LibXML, XML::Twig, and XPath); dealing with config files; more advanced LDAP topics (including updated Net::LDAP information); email-related topics (including POP3/ IMAP, MIME, and spam); new ways of dealing with filesystems; more advanced log file creation and parsing tools; DHCP; mapping/monitoring a network using Nmap and other tools; packet creation and sniffing; information reporting using tools like GraphViz, RRDtool, and Timeline; using SHA-2 instead of MD5; xv SNMPv3; Mac OS X; converting VBScript code to Perl; geocoding; MP3 file manipulation; using Google Maps; and so on. New advice Part of the value of this book is the advice you can pick up from an experienced system administrator like me who has been doing this stuff for a long time and has compared notes with many other seasoned veterans. This new edition is packed with more sidebars to explain not only the what, but also the why behind the material. Operating system and software information updates All of the text and code has been updated and augmented to work with the latest versions of Unix- (including Linux and Mac OS X) and Windows-based operating systems. Module and code updates/improvements The descriptions and code in this book match the latest versions of the modules mentioned in the first edition. In cases where a module is no longer available or a better alternative has emerged, the appropriate replacement modules have been substituted. Also, all example code is now “use strict” friendly. Errata corrected I have attempted to address all of the errata I received from all of the printings of the first edition. I appreciate the time readers took to report errors to O’Reilly and me so I could fix them at each printing and in this edition. Special thanks go to Andreas Karrer, the German translator for the first edition. Andi pored over every single byte of the original text and submitted almost 200 (mostly layout-related) corrections, all with good cheer. How This Book Is Structured Each chapter in this book addresses a different system administration domain and ends with a list of the Perl modules used in that chapter and references to facilitate deeper exploration of the information presented. The chapters are as follows: Chapter 1, Introduction This introductory chapter describes the material covered in the book in more detail, explaining how it will serve you and what you need to get the most from it. The material in this book is powerful and is meant to be used by powerful people (e.g., Unix superusers and Windows-based operating system administrators). The introduction provides some important guidelines to help you write more secure Perl programs. Chapter 2, Filesystems This chapter is about keeping multiplatform filesystems tidy and ensuring that they are used properly. We’ll start by looking at the salient differences between the native filesystems for each operating system. We’ll then explore the process of xvi | Preface intelligently walking or traversing filesystems from Perl and how that can be useful. Finally, we’ll look at manipulating disk quotas from Perl. Chapter 3, User Accounts This chapter discusses how user accounts manifest themselves on two different operating systems, including what is stored for each user and how to manipulate the information from Perl. That leads into a discussion of a rudimentary account system written in Perl. In the process of building this system, we’ll examine the mechanisms necessary for recording accounts in a simple database, creating these accounts, and deleting them. Chapter 4, User Activity Chapter 4 explores ways to automate tasks centered around user activity, intro- ducing a number of ways to track and control process, file, and network operations initiated by users. This chapter also presents various operating system-specific frameworks and tools (e.g., Windows Management Instrumentation, GUI setup tools, lsof, etc.) that are helpful for user-oriented tasks on different platforms. Chapter 5, TCP/IP Name and Configuration Services Name and configuration services allow hosts on a TCP/IP network to communicate with each other amicably and to self-configure. This chapter takes a historical per- spective by starting with host files, then moving on to the Network Information Service (NIS) and finally to the glue of the Internet, the Domain Name Service (DNS). Each step of the way, it shows how Perl can make professional management of these services easier. We’ll also explore how to work with the Dynamic Host Configuration Protocol (DHCP) from Perl in this chapter. Chapter 6, Working with Configuration Files Almost every system or software package we touch relies heavily on configuration files to be useful in our environment. This chapter explores the tools that make writing and reading those files from Perl easy. We’ll look at various formats, with special attention paid to XML and the current best practices for working with it using Perl. Chapter 7, SQL Database Administration Over time, more uses for relational databases are being found in the system ad- ministration realm. As a result, system administrators need to become familiar with SQL database administration. This chapter explains DBI, the preeminent SQL database framework for Perl, and provides examples of it in action for database administration. Chapter 8, Email This chapter demonstrates how Perl can make better use of email as a system ad- ministration tool. After discussing sending via SMTP (including MIME-based HTML messages), receiving via POP3/IMAP, and parsing via Perl, we’ll explore several interesting applications, including tools for analyzing unsolicited commer- cial email (a.k.a. spam) and managing tech support emails. Preface | xvii Chapter 9, Directory Services As the complexity of the information we deal with increases over time, so does the importance of the directory services we use to access that information. System administrators are increasingly being called upon not only to use these services, but also to build tools for their management. This chapter discusses some of the more popular directory service protocols/frameworks, such as LDAP and ADSI, and shows you how to work with them from Perl. Chapter 10, Log Files System administrators are often awash in a sea of log files. Every machine, operating system, and program can (and often does) log information. This chapter looks at the logging systems offered by Unix- and Windows-based operating systems and discusses approaches for analyzing logging information so it can work for you. Chapter 11, Security This chapter heads right into the maelstrom called “security,” demonstrating how Perl can make hosts and networks more secure. Chapter 12, SNMP This chapter is devoted to the Simple Network Management Protocol (SNMP). It illustrates how to use this protocol to communicate with network devices (both to poll and to receive trap information). Chapter 13, Network Mapping and Monitoring Perl offers some excellent tools for the mapping and monitoring of networks. In this chapter, we’ll look at several ways to discover the hosts on the network and the services they offer. We’ll then explore helpful graphical and textual ways to present the information collected, including some of the best tools for graphing and charting the data (such as GraphViz and RRDtool). Chapter 14, Experiential Learning This is the chapter you don’t want your boss to catch you reading. Appendixes Some of the chapters assume basic knowledge about topics with which you may not be familiar. For those who are new to these subjects, this book includes several mini-tutorials to bring you up to speed quickly. The appendixes provide introduc- tions to the eXtensible Markup Language (XML), the XML Path Language (XPath), the Lightweight Directory Access Protocol (LDAP), the Structured Query Lan- guage (SQL), the Revision Control System (RCS), translating VBScript to Perl, and SNMP. xviii | Preface Typographical Conventions This book uses the following typographical conventions: Italic Used for file- and pathnames, usernames, directories, program names, hostnames, URLs, and new terms where they are first introduced. Constant width Used for Perl module and function names, namespaces, libraries, commands, methods, and variables, and when showing code and computer output. Constant width bold Used to indicate user input and for emphasis in code examples. Constant width italic Used to indicate parts of a command line that are user-replaceable, and for code annotations. This icon signifies a tip, suggestion, or general note. This icon indicates a warning or caution. Operating System Naming Conventions This book is steadfastly multiplatform in its thinking. However, reading about “a Microsoft Vista/Microsoft Windows Server 2008/Microsoft Windows Server 2003/ Microsoft XP script” or a “Linux/Solaris/Irix/HPUX/Mac OS X/etc. script” gets old fast. Having consulted some style guides, here’s how I’ve chosen to handle discussing the operating system collectives: • When writing about the Microsoft products—Microsoft Vista, Microsoft Win- dows Server 2008, Microsoft Windows Server 2003, and Microsoft XP (on which, by and large, all scripts were tested)—I refer to them collectively as “Windows- based operating systems,” at least first time they show up in a chapter or heading. From that point on in the chapter I shorten this to simply “Windows.” If something is particular to a specific Windows-based operating system, I will mention it by name. Preface | xix • When writing about any of the members of the Unix family (in which I include both Linux and Mac OS X), I refer to them collectively as just “Unix,” “the Unix family,” or sometimes “Unix variants.” If something is particular to a specific Unix vendor or release, I will mention it by name. Coding Conventions There are a few points I want to mention about the code in this book: • All code examples were written and tested with use strict; as the first line (I highly recommend you do the same). However, given the number of examples in this book, the repetition would have taken up a significant amount of space, so to save trees and wasted bits I did not include that line in any of the examples. Please just assume that every example uses this convention. • Almost all of the code is formatted using Steve Hancock’s fabulous perltidy util- ity ( to improve readability. • Although these examples don’t reach anything like that level of perfection, much of the code has been rewritten with the advice in Damian Conway’s book Perl Best Practices ( (O’Reilly) in mind. I highly recommend reading Conway’s book to improve your code and generally reinvigo- rate your Perl programming. The automated source code analyzer Perl::Critic that Perl Best Practices inspired was still in heavy development for much of the writing of this book, so I did not use it. You should, though, as it’s another great tool. Using Code Examples This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Automating System Administration with Perl, Second Edition, by David N. Blank-Edelman. Copyright 2009 O’Reilly Media, Inc., 978-0-596-00639-6.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at xx | Preface How to Contact Us We have tested and verified the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know of any errors you find, as well as your suggestions for future editions, by writing to: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the U.S. or Canada) 707-829-0515 (international/local) 707-829-0104 (fax) We have a website for the book, where we’ll list examples, errata, and any plans for future editions. You can access this page at: The author has set up a personal website for this book. Please visit it at: To ask technical questions or comment on the book, send email to: For more information about our books, conferences, software, Resource Centers, and the O’Reilly Network, see the O’Reilly website: Safari® Books Online When you see a Safari® Books Online icon on the cover of your favorite technology book, that means the book is available online through the O’Reilly Network Safari Bookshelf. Safari offers a solution that’s better than e-books. It’s a virtual library that lets you easily search thousands of top tech books, cut and paste code samples, download chapters, and find quick answers when you need the most accurate, current information. Try it for free at Acknowledgments from the First Edition To keep the preface from becoming too much like an Oscar acceptance speech, here’s a condensed version of the acknowledgments from the first edition. Preface | xxi Thanks to the Perl Community, especially Larry Wall, Tom Christiansen, and the ker- jillions of programmers and hackers who poured countless hours and energy into the language and then chose to share their work with me and the rest of the Perl community. Thanks to the SysAdmin community: members of Usenix, SAGE, and the people who have contributed to the LISA conferences over the years. Thanks to Rémy Evard for being such a great influence on my professional and personal understanding of this field as a friend, mentor, and role model. He is still one of the system administrators I want to be when I grow up. Thanks to the reviewers of the first edition: Jerry Carter, Toby Everett, Æleen Frisch, Joe Johnston, Tom Limoncelli, John A. Montgomery, Jr., Chris Nandor, Michael Peppler, Michael Stok, and Nathan Torkington. Thanks to the O’Reilly staff: to Rhon Porter for his illustrations, to Hanna Dyer and Lorrie LeJeune for the most amazing cover animal, and to the O’Reilly production staff. I am still thankful to Linda Mui, my editor for the first edition, whose incredible skill, finesse, and care allowed me to birth this book and raise it in a good home. Thanks to my spiritual community: Havurat Shalom in Somerville. Thank you, M’ kor HaChayim, for this book and all of the many blessings in my life. Thanks to the Shona people of Zimbabwe for their incredible mbira music. Thanks to my friends (Avner, Ellen, Phil Shapiro, Alex Skovronek, Jon Orwant, and Joel Segel), the faculty and staff at the Northeastern University College of Computer and Information Science (especially the folks in the CCIS Systems group), and my boss Larry Finkelstein, the Dean of the College of Computer Science. Thanks to my nuclear family (Myra, Jason, and Steven Edelman-Blank), my cats Shim- mer and Bendir (bye-bye, Bendir, I’ll miss you), and my TCM pit crew (Kristen Porter and Thom Donovan). The first edition was dedicated to Cindy, love of my life. Acknowledgments for the Second Edition One of the only things better than having all of these great people and things in your life is to have them remain in your life. I’m still thankful for all of the above from the first edition. Here are some of the changes and additions: This edition had a much expanded and tremendous group of technical reviewers. I’m very grateful to Æleen Frisch, Aaron Crane, Aleksey Tsalolikhin, Andrew Langmead, Bill Cole, Cat Okita, Chaos Golubitsky, Charles Richmond, Chris Grau, Clifton Roy- ston, Dan Wilson, Dean Wilson, Denny Allain, Derek J. Balling, Earl Gay, Eric Soren- son, Eric Toczek, Federico Lucifredi, Gordon “Fyodor” Lyon, Graham Barr, Grant McLean, Hugh Brown, James Keating, Jan Dubois, Jennifer Davis, Jerry Carter, Jesse Vincent, Joe Morri, John Levine, John Tsangaris, Josh Roberts, Justin Mason, Mark xxii | Preface Bergman, Michel Rodriguez, Mike DeGraw-Bertsch, Mike Stok, Neil Neely, Petr Pajas, Philip J. Hollenback, Randy Dees, Scott Murphy, Shlomi Fish, Stephen Potter, Steve Atkins, Steven Tylock, Terry Zink, Thomas Leyer, Tim Bunce, Tobias Oetiker, Toby Ovod-Everett, and Tom Regner for the time and energy they spent on making this book better. I continue to be amazed by the generosity and kindness shown by the members of the SysAdmin and Perl communities. The editorial chain was a bit longer than usual on this book, so thanks to all of the editors. Starting from the first edition in chronological order: Linda Mui, Paula Fergu- son, Nathan Torkington, Allison Randal, Colleen Gorman, Tatiana Apandi, Isabel Kunkle, and Andy Oram. I’m also thankful to the other O’Reilly people who have had a hand in bringing this book to fruition, including Mike Hendrickson, Rachel Head, Sarah Schneider, Rob Romano, Sanders Kleinfeld, and all the others. I was taken with sea otters even before the first edition was published with one on the front cover, but since then my appreciation for them keeps on growing. They are an amazing species in so many ways. Unfortunately, humans historically haven’t been particularly kind to the sea otters. They are still classified as an endangered species, and some of our activities actively threaten their survival. I believe they deserve our protection and our support. One organization that works toward this end is Friends of the Sea Otter (, based in Monterey, California. I’m a member, and I encourage you to join, too. Mbira kept me sane through the arduous process of writing the first edition of this book. For this edition, I have yoga to thank for my current health and sanity. I’d like to express my profound gratitude to my teacher, Karin Stephan, and her teacher, B.K.S. Iyengar, for sharing such a wonderful union of mind and body with me. I’ve tried to cut down the florid prose of the first edition’s acknowledgments, but I hope you’ll indulge me just one more time. The biggest change for me between these editions was the birth of our first child, Elijah. He’s been a constant blessing to us, both in the noun and verb senses of the word. Preface | xxiii CHAPTER 1 Introduction In my town, several of our local bus lines are powered by cables strung high above the street. One day, when going to an unfamiliar destination, I asked the driver to let me know when a particular street was approaching. He said, “I’m sorry, I can’t. I just follow the wires.” These are words you will never hear from good system administrators asked to describe their jobs. System administration is a craft. It’s not about following wires. System and network administration is about deciding what wires to put in place and where to put them, getting them deployed, keeping watch over them, and then eventually ripping them out and starting all over again. Good system administration is hardly ever rote, especially in multiplatform environments where the challenges come fast and furious. As in any other craft, there are better and worse ways to meet these challenges. Whether you’re a full-time system administrator or a part-time tinkerer, this book will help you along that path. Automation Is a Must Any solution that involves fiddling with every one of your machines by hand is almost always the wrong one. This book will make that approach a thing of the past for you. Even in the best of economic climates, system administrators always have too much to do. This is true both for the people who do this work by choice and for those who had a boss walk into their office and say, “Hey, you know about computers. We can’t hire anyone else. Why don’t you be in charge of the servers?” When hiring gets frozen, existing employees (including those not trained for the task) may well be asked to take on added system administration responsibilities. Automation, when applied intelligently, is one of the few things that can actually make a difference under these circumstances. It can help people work more efficiently, often freeing up time previously spent on sysadmin scut work for more interesting things. This can improve both productivity and morale. 1 My editors and I changed the title of this edition of the book because we realized that the real value of the material was its ability to make your life better through automation. In this book, I’ll try very hard to give you the tools you need (including the mental ones—new ways to think about your problems, for example) to improve your time at work (and, as you’ll see in the last chapter, your time at play). Related Topic: Configuration Management Before we get started, a quick note about what this book is not is in order. It’s not a book about configuration management, and it doesn’t cover the popular tools that support configuration management, such as cfengine, puppet, and bcfg2. Most environments can benefit both from having the configuration of their machines/ networks managed and from automating their everyday processes. This book focuses strictly on the second topic, but I strongly encourage you to look at the tools I mentioned in the first paragraph if you are not already using some sort of configuration manage- ment system. Once you adopt a configuration tool, you can integrate it with the scripts you’ll learn to write using this book. How Perl Can Help You System administrators should use any and every computer language available when appropriate. So why single out Perl for a book? The answer to this question harks back to the very nature of system administration. Rémy Evard, a colleague and friend, once described the job of a system administrator as follows: On one side, you have a set of resources: computers, networks, software, etc. On the other side, you have a set of users with needs and projects—people who want to get work done. Our job is to bring these two sets together in the most optimal way possible, translating between the world of vague human needs and the technical world when necessary. System administration is often a glue job, and Perl is one of the best glue languages. Perl was being used for system administration work well before the World Wide Web came along with its voracious need for glue mechanisms. Conversations I’ve had with numerous system administrators at Large Installation System Administration (LISA) ( conferences and other venues have indicated that Perl is still the dominant language in use for the field. Perl has several other things going for it from a system administration perspective: • It has visible origins in the various Unix shells and the C language, which are tools many system administrators are comfortable using. 2 | Chapter 1: Introduction • It is available on almost all modern operating systems and does its best to present a consistent interface on each. This is important for multiplatform system administration. • It has excellent tools for text manipulation, database access, and network pro- gramming, which are three of the mainstays of the profession. • The core language can easily be extended through a carefully constructed module mechanism. • A large and dedicated community of users has poured countless hours into creating modules for virtually every task. Most of these modules are collected in an organ- ized fashion (more on these collections in a moment). This community support can be very empowering. • It is just plain fun to program. In the interest of full disclosure, it is important to note that Perl is not the answer to all of the world’s problems. Sometimes it is not even the appropriate tool for system ad- ministration programming. There are a few things going against it: • Perl has a somewhat dicey object-oriented programming mechanism grafted on top of it. Python or Ruby is much better in this regard. • Perl is not always simple or internally self-consistent and is chock-full of arcane invocations. Other languages have far fewer surprises. • Perl is powerful and esoteric enough to shoot you in the foot. The moral here is to choose the appropriate tool. More often than not, Perl has been that tool for me, and hence it’s the focus of this book. This Book Will Show You How In the 1966–68 Batman television show, the dynamic duo wore utility belts. If Batman and Robin had to scale a building, Batman would say, “Quick Robin, the Bat Grappling Hook!” or “Quick Robin, the Bat Knockout Gas!” and they’d both have the right tool at hand to subdue the bad guys. This book aims to equip you with the utility belt you need to do good system administration work. Every chapter attempts to provide you with three things: Clear and concise information about a system administration domain In each chapter, we discuss in depth one domain of the system administration world. The number of possible domains in multiplatform system administration is huge; there are far too many to be included in a single book. The best survey books on just Unix system administration—Essential System Administration by Æleen Frisch (O’Reilly), and Unix System Administration Handbook, by Evi Nem- eth, Garth Snyder, Scott Seebass, and Trent H. Hein (Prentice Hall)—are two and three times, respectively, the size of this book, and we’ll be looking at topics related This Book Will Show You How | 3 to three different operating systems: Unix (including variants like Linux), Windows-based operating systems, and Mac OS X. The list of topics covered is necessarily incomplete, but I’ve tried to put together a good stew of system and network administration information for people with varying levels of experience in the field. Seasoned veterans and new recruits may come away from this book having learned completely different material, but ev- eryone should find something of interest to chew on. Each chapter ends with a list of references that can help you get deeper into a topic should you so choose. For each domain or topic—especially those that have a considerable learning curve—I’ve included appendixes that will give you all the information you need to get up to speed quickly. Even if you’re familiar with a topic, you may find that these appendixes can round out your knowledge (e.g., showing how something is implemented on a different operating system). Perl techniques and approaches that can be used in system administration To get the most out of this book, you’ll need some initial background in Perl. Every chapter is full of Perl code that ranges in complexity from beginner to advanced levels. Whenever we encounter an intermediate-to-advanced technique, data struc- ture, or idiom, I’ll take the time to carefully step through it, piece by piece. In the process, you should be able to pick up some interesting Perl techniques to add to your programming repertoire. My hope is that Perl programmers of all levels will be able to learn something from the examples presented in this book. And as your Perl skills improve over time, you should be able to come back to the book again and again, learning new things each time. To further enhance the learning experience, I will often present more than one way to accomplish the same task using Perl, rather than showing a single, limited an- swer. Remember the Perl motto, “There’s more than one way to do it.” These multiple-approach examples are designed to better equip your Perl utility belt: the more tools you have at hand, the better the choices you can make when approach- ing a new task. Sometimes it will be obvious that one technique is superior to the others. But this book addresses only a certain subset of situations you may find yourself in, and a solution that is woefully crude for one problem may be just the ticket for another. So bear with me. For each example, I’ll try to show you both the advantages and the drawbacks of each approach (and often tell you which method I prefer). System administration best practices and deep principles As I mentioned at the start of this chapter, there are better and worse ways to do system administration. I’ve been a system and network administrator for the past 25 years in some pretty demanding multiplatform environments. In each chapter I try to bring this experience to bear as I offer you some of the best practices I’ve learned and the deeper principles behind them. Occasionally I’ll use a personal “war story from the front lines” as the starting point for these discussions. 4 | Chapter 1: Introduction Hopefully the depth of the craft in system administration will become apparent as you read along. What You Need To get the most out of this book, you will need some technical background and some resources at hand. Let’s start with the background first: You’ll need to know some Perl There isn’t enough room in this book to teach you the basics of the Perl language, so you’ll need to seek that information elsewhere before working through this ma- terial. A book like Learning Perl (, by Randal L. Schwartz et al. (O’Reilly), can get you in good shape to approach the code in this book. You’ll need to know the basics of your operating system(s) This book assumes that you have some facility with the operating system or systems you plan to administer. You’ll need to know how to get around in that OS (run commands, find documentation, etc.). Background information on the more com- plex frameworks built into the OS (e.g., WMI on Windows or SNMP) is provided. You may need to know the specifics of your operating system(s) I’ll attempt to describe the differences between the major operating systems as we encounter them, but I can’t cover all of the intra-OS differences. In particular, every variant of Unix is a little different. As a result, you may need to track down OS- specific information and roll with the punches should that information be different from what is described here. For technical resources, you will need just two things: Perl You will need a copy of Perl installed on or available to every system you wish to administer. The downloads section of the Perl website ( will help you find either the source code or the binary distribution for your particular operating system. The code in this book was developed and tested under Perl 5.8.8 and ActivePerl (5.8.8) 822. See the next section for more information about these versions. The ability to find and install Perl modules A later section of this chapter is devoted to the location and installation of Perl modules, an extremely important skill for our purposes. This book assumes you have the knowledge and necessary permissions to install any modules you need. At the end of each chapter is a list of the version numbers for all of the modules used by the code in that chapter. The version information is provided because modules are updated all the time, and not all updates retain backward What You Need | 5 compatibility. If you run into problems, this information can help you determine whether there has been a module change since this book was published. Some Notes About the Perl Versions Used for This Book I chose to develop and test the code in this book under Perl 5.8.8 and ActivePerl (5.8.8) 822. These choices might lead you to ask a few questions. What About Perl 5.10? The Perl 5 development team has done some fabulous work to produce 5.10. They’ve added some great features to the language that I encourage you to explore. However, 5.10 wasn’t released until well after this edition was under way, and at the time of this writing no major OS distribution has shipped with it as its default version of Perl. Because I know the adoption of new versions takes a while, I didn’t want to include code in the book that depended on features in the language most people couldn’t use out of the box. All of the code here should work just fine on Perl 5.10, and in the interest of making this code useful to as many readers as possible, I deliberately chose to target the previous stable release. What About Strawberry Perl? Strawberry Perl ( is an effort to bring a more “generic” and self-sufficient version of Perl to the Win32 platform. ActiveState’s Perl distribution ships with a packaging system (PPM) so users don’t have to compile modules or update them via the Comprehensive Perl Archive Network (CPAN). Strawberry Perl aims to provide an environment where compilation and CPAN use are easy (or at least possible) and are the norm. I think this is an excellent project because it is helping to push some portability back into the non-Win32 Perl community. Some great progress has been made so far, but the project is still fairly young as of this writing and it does not yet have a sufficiently large ecosystem of available modules (e.g., lots of the Win32:: modules are missing). That ruled it out for this edition, but it is definitely something to watch. What About Perl 6? Ah, that’s the big question, isn’t it? I have the pleasure of occasionally bumping into Jesse Vincent, the current Perl 6 project manager (and author of the fabulous RT trouble ticketing system). Here’s what he had to say when I asked about Perl 6: Perl 5 is a mature, widely deployed, production-ready language. Perl 6 is maturing rapidly, but isn’t yet ready for production deployment. 6 | Chapter 1: Introduction There are some Perl 5 modules that let you get a taste of some planned Perl 6 features (some of which have found their way into Perl 5.10). I encourage you to try modules like Perl6::Slurp and Perl6::Form. But at this point in time, there just isn’t a language implementation ready for production use, and hence there is no Perl 6 in this book. Furthermore, once Perl 6 is ready for widespread use, it will take considerable time for the necessary ecosystem of modules to be developed to replace the many, many mod- ules we leverage in this book. I look forward to that time; perhaps you’ll see a Perl 6 edition of this book some day. Some Notes About Using Vista with the Code in This Book The code in this book has been tested under Microsoft Vista, but there is one twist you will need to know about if you plan to use it on that platform: some of the examples in this book must be run using elevated privileges for this to work. Which things require this and which don’t is somewhat idiosyncratic. For example, part of the Windows quota example in Chapter 2 works without elevated privileges and part (the important part) fails with an unhelpful error if it doesn’t have them. Under Vista’s User Account Control (UAC), it is not enough to be running the code as an Administrator; you must have explicitly requested it to run at an elevated privilege level. Here are the ways I know to run Perl scripts at that privilege level (since you can’t by default right-click and use “Run as administrator”). You should choose the method or methods that make the most sense in your environment: • Use the runas.exe command-line utility. • Designate that the perl.exe binary itself be run as an Administrator (right-click on the binary name, choose Properties, switch to the Compatibility tab, and select “Run this program as administrator.” • Use one of the Elevation Power Toys described at -us/magazine/2008.06.elevation.aspx and zine/2007.06.utilityspotlight.aspx to allow Perl scripts to be Run as administrator. • Use the command-line utility pl2bat to convert your Perl script into a batch file and then permit that batch file to run as Administrator. Batch files don’t require any special magic (like the previous option) for this to happen. You may be wondering if it is possible to add something to your Perl script to have it request elevated privileges as needed. Unfortunately, according to Jan Dubois (one of the top Windows Perl luminaries in the field), the answer is no. He notes that there is no way to elevate an already running process; it must be created with elevated privileges. The closest you could come would be to check whether the process was already running in this fashion (e.g., by using the Win32 module’s IsAdminUser() function), and if not invoke another copy of the script using something like runas.exe. Some Notes About Using Vista with the Code in This Book | 7 One last note in a similar vein: in several of the chapters I recommend using the Mi- crosoft Scriptomatic tool to become familiar with WMI. By default this won’t run under Vista because it needs elevated privileges to function, but it is an “HTML Application” (.hta) file. Like Perl scripts, .hta files can’t easily be Run as administrator. Here’s a recipe for getting around this limitation so you can use this excellent tool: 1. Right-click on the Internet Explorer icon in the taskbar (the “E”) and choose “Run as administrator” to run it using elevated privileges. (Warning: don’t use this run- ning copy of IE to browse to any website or load anything but the Scriptomatic file, to be on the safe side.) 2. Press the Alt key to display the IE File menu. Choose “Open…” and then press the “Browse…” button. Change the dialog filter to display “All Files” and then browse to the location of the Scriptomatic .hta file. Open that file and you should be all set. Locating and Installing Modules Much of the benefit of using Perl for system administration work comes from all of the free code available in module form. The modules mentioned in this book can be found in one of three places: The Comprehensive Perl Archive Network CPAN is a huge archive of Perl source code, documentation, scripts, and modules that is replicated at over a hundred sites around the world. Information on CPAN can be found at The easiest way to find the modules in CPAN is to use the search engine at The “CPAN Search” box makes it simple to find the right modules for the job. Individual repositories for prebuilt packages In a moment we’ll encounter the Perl Package Manager (PPM), an especially im- portant tool for Win32 Perl users. This tool connects to repositories (the most famous one is housed at ActiveState) to retrieve prebuilt module packages. A good list of these repositories can be found in the wiki at If a Win32 package we use comes from a repository other than ActiveState’s, I’ll be sure to point you to it. Individual websites Some modules are not published to CPAN or any of the PPM repositories. I really try to avoid them if possible, but in those rare cases where they fill a critical gap, I’ll tell you where to get them. How do you install one of these modules when you find it? The answer depends on the operating system you are running. Perl now ships with documentation on this process in a file called perlmodinstall.pod (type perldoc perlmodinstall to read it). The next sections provide brief summaries of the steps required for each operating system used in this book. 8 | Chapter 1: Introduction Installing Modules on Unix In most cases, the process goes like this: 1. Download the module and unpack it. 2. Run perl Makefile.PL to create the necessary Makefile. 3. Run make to build the package. 4. Run make test to run any test suites included with the module by the author. 5. Run make install to install it in the usual place for modules on your system. If you want to save yourself the trouble of performing all these steps by hand, you can use the CPAN module by Andreas J. König (shipped with Perl), or the CPANPLUS module by Jos Boumans. CPAN allows you to perform all of those steps by typing: % cpan cpan[1]> install modulename and CPANPLUS does the same with: % cpanp CPAN Terminal> i modulename Both modules are smart enough to handle module dependencies (i.e., if one module requires another module to run, it will install both modules for you automatically). They also each have a built-in search function for finding related modules and packages. I recommend typing perldoc CPAN or perldoc CPANPLUS on your system to find out more about all of the handy features of these modules. Installing Modules on Win32 The process for installing modules on Win32 platforms using the ActiveState distribu- tion mirrors that for Unix, with one additional step: the Perl Package Manager (PPM). If you are comfortable installing modules by hand using the Unix instructions in the previous section, you can use a program like WinZip ( to un- pack a distribution and use nmake ( .exe) instead of make to build and install a module. Some modules require compilation of C files as part of their build process. A large portion of the Perl users in the Win32 world do not have the necessary software installed on their computers for this compilation, so ActiveState created PPM to handle prebuilt module distribution. The PPM system is similar to that of the CPAN module. It uses a Perl program called to handle the download and installation of special archive files from PPM repositories. You can start the program either by typing ppm or by running ppm-shell from within the Perl bin directory: Locating and Installing Modules | 9 C:\Perl\bin> ppm-shell ppm 4.03 ppm> install module-name PPM, like CPAN, can search the list of available and installed modules for you. Type help at the ppm> command prompt for more information on how to use these commands. It’s Not Easy Being Omnipotent Before we continue with the book, let’s take a few minutes for some cautionary words. Programs written for system administration have a twist that makes them different from most other programs: on Unix and Windows they are often run with elevated privileges (i.e., as root or Administrator). With this power comes responsibility. There is an extra onus on us as programmers to write secure code. We write code that can and will bypass the security restrictions placed on mere mortals. Tiny mistakes can lead to severe dis- ruptions for our users or damage to key system files. And, if we are not careful, less “ethical” users may use flaws in our code for nefarious purposes. Here are some of the issues you should consider when you use Perl under these circumstances. Don’t Do It By all means, use Perl. But if you can, avoid having your code run in a privileged context. Most tasks do not require root or Administrator privileges. For example, your log analysis program probably does not need to run as root. Create another, less privileged user for this sort of automation. Have a small, dedicated, priv- ileged program hand the data to that user if necessary, and then perform the analysis as the unprivileged user. Drop Your Privileges As Soon As Possible Sometimes you can’t avoid running a script as root or Administrator. For instance, a mail delivery program you create may need to be able to write to a file as any user on the system. However, programs like these should shed their omnipotence as soon as possible during their run. Perl programs running under Unix can set the $< and $> variables: # permanently drops privs ($<,$>) = (getpwnam('nobody'),getpwnam('nobody')); This sets the real and effective user IDs to nobody, which exists on most Unix/Linux systems as an underprivileged user (you can create the user yourself if need be). To be even more thorough, you may wish to use $( and $) to change the real and effective group IDs as well. 10 | Chapter 1: Introduction Windows does not have user IDs per se, but there are similar processes for dropping privileges, and you can use runas.exe to run processes as a different user. Be Careful When Reading Data When reading important data like configuration files, test for unsafe conditions first. For instance, you may wish to check that the file and all of the directories in its path are not writable (since that would make it possible for someone to tamper with them). There’s a good recipe for testing this in Chapter 8 of the Perl Cookbook (http://oreilly .com/catalog/9780596003135/), by Tom Christiansen and Nathan Torkington (O’Reilly). The other concern is user input. Never trust that input from a user is palatable. Even if you explicitly print Please answer Y or N:, there is nothing to prevent the users from answering with 2,049 random characters (either out of malice or because they stepped away from the computer and a two-year-old came over to the keyboard instead). User input can be the cause of even more subtle trouble. My favorite example is the “poison NULL byte” exploit reported in an article on Perl CGI problems (cited in the references section at the end of this chapter—be sure to read the whole article!). This particular exploit takes advantage of the difference between Perl’s handling of a NULL (\000) byte in a string and the handling done by the C libraries on a system. To Perl, there is nothing special about this character, but to the libraries it indicates the end of a string. In practical terms, this means it is possible for a user to evade simple security tests. One example given in the article is that of a password-changing program whose code looks like this: if ($user ne "root"){ } If a malicious user manages to set $user to root\000 (i.e., root followed by a NULL byte), the test will think that the name is not root and will allow the Perl script to continue. But when that string is passed to the underlying C library, the string will be treated as just root, and the user will have walked right past the security check. If not caught, this same exploit will allow access to random files and other resources on the system. The easiest way to avoid being caught by this exploit is to sanitize your input with something like this: $input =~ tr/\000//d; or better yet, only use valid data that you’ve explicitly extracted from the user’s input (e.g., with a regular expression). This is just one example of how user input can get programs into trouble. Because user input can be so problematic, Perl has a security precaution called taint mode. See the perlsec manpage that ships with Perl for an excellent discussion of “taintedness” and other security precautions. It’s Not Easy Being Omnipotent | 11 Be Careful When Writing Data If your program can write or append to every single file on the local filesystem, you need to take special care with how, where, and when it writes data. On Unix systems, this is especially important because symbolic links make file switching and redirection easy. Unless your program is diligent, it may find itself writing to the wrong file or device. There are two classes of programs where this concern comes especially into play. Programs that append data to a file fall into the first class. The steps your program should take before appending to a file are: 1. Check the file’s attributes before opening it, using stat() and the normal file test operators. Make sure that it is not a hard or soft link, that it has the appropriate permissions and ownership, etc. 2. Open the file for appending. 3. stat() the open filehandle. 4. Compare the values from steps 1 and 3 to be sure that you have an open handle to the file you intended. The bigbuffy program in Chapter 10 illustrates this procedure. Programs that use temporary files or directories are in the second class. Chances are you’ve often seen code like this: open(TEMPFILE,">/tmp/temp.$$") or die "unable to write /tmp/temp.$$:$!\n"; Unfortunately, that’s not sufficiently secure on a multiuser machine. The process ID ($$) sequence on most machines is easily predictable, which means the next temporary filename your script will use is equally predictable. If others can predict that name they may be able to get there first, and that’s usually bad news. The easiest way to avoid this conundrum is to use Tim Jenness’s File::Temp module, which has shipped with Perl since version 5.6. Here’s how it is used: use File::Temp qw(tempfile); # returns both an open filehandle and the name of that file my ($fh, $filename) = tempfile(); print $fh "Writing to the temp file now...\n"; File::Temp can also remove the temporary file for you automatically if desired. See the module’s documentation for more details. Avoid Race Conditions Whenever possible, avoid writing code that is susceptible to race condition exploits. The traditional race condition starts with the assumption that the following sequence is valid: 12 | Chapter 1: Introduction 1. Your program will amass some data. 2. Your program can then act on that data. Here’s a simple example: 1. Your program checks the timestamp on a file of bug submissions to make sure nothing has been added since you last read the file. 2. Your program modifies the contents of the file. If users can break into this sequence at a point we’ll call “step 1.5” and make some key substitutions, they may cause trouble. If they can get your program in step 2 to naively act upon different data from what it found in step 1, they have effectively exploited a race condition (i.e., their program won the race to get at the data in question). Other race conditions occur if you do not handle file locking properly. Race conditions often show up in system administration programs that scan the file- system as a first pass and then change things in a second pass. Nefarious users may be able to make changes to the filesystem right after the scanner pass so that changes are made to the wrong file. Make sure your code does not leave such gaps open. Enjoy It is important to remember that system administration is fun. Not all the time, and not when you have to deal with the most frustrating of problems, but there’s definitely enjoyment to be found. There is a real pleasure in supporting other people and building the infrastructures that make users’ lives better. When the collection of Perl programs you’ve just written brings other people together for a common purpose, there is joy. So, now that you’re ready, let’s get to work on those wires. References for More Information is a HOWTO document written by David A. Wheeler for secure programming under Linux and Unix. The concepts and techni- ques Wheeler describes are applicable to other situations as well. contains more good secure programming re- sources from security expert Matt Bishop. lists security code review guidelines by Adam Shostack. is an old but good paper on how to find security holes (especially in your own code) by Kragen Sitaker. “Perl CGI Problems,” by rain.forest.puppy (Phrack Magazine, 1999), describes CGI security vulnerabilities. It can be found online at .txt or in the Phrack archives at References for More Information | 13 Perl Cookbook, Second Edition (, by Tom Christiansen and Nathan Torkington (O’Reilly), contains many good tips on coding securely. 14 | Chapter 1: Introduction CHAPTER 2 Filesystems Perl to the Rescue Laptops fall in slow motion. Or at least that’s the way it looked when the laptop I was using to write the first edition of this book fell off a table onto a hardwood floor. The machine was still in one piece and running when I picked it up, but as I checked to see whether anything was damaged, it started to run slower and slower. Then it began to make sporadic and disturbing humming-buzzing sounds during disk access. Figuring the software slowdown was caused by a software problem, I shut down the laptop. It did not go gently into the night, refusing to shut down cleanly. This was a bad sign. Even worse was its reluctance to boot again. Each time I tried, it began the Windows NT booting process and then failed with a “file not found” error. By now it was clear that the fall had caused some serious physical damage to the hard drive. The heads had probably skidded over the platter surface, destroying files and directory entries in their wake. Now the question was, “Did any of my files survive? Did the files for this book survive?” I first tried booting into Linux, the other operating system installed on the laptop. Linux booted fine, an encouraging sign. The files for this book, however, resided on the Win- dows NT NTFS partition that did not boot. Using Martin von Löwis’s Linux NTFS driver, available at (now shipping with the Linux kernels), I mounted the partition and was greeted with what looked like all of my files, intact. My ensuing attempts to copy those files off that partition would proceed fine for a while, until I reached a certain file. At that point the drive would make those ominous sounds again and the backup would fail. It was clear that if I wanted to rescue my data I was going to have to skip all the damaged files on the disk. The program I was using to copy the data (gnutar) had the ability to skip a list of files, but here was the problem: which files? There were over sixteen thousand* files on the filesystem at the time of impact. How was I going to figure out which files were damaged and which were fine? * At the time, 16,000 files seemed like a lot. My current laptop has 1,096,010 files on it as I write this. I imagine if this story had happened today it would have been even more fun. 15 Clearly running gnutar again and again was not a reasonable strategy. This was a job for Perl! I’ll show you the code I used to solve this problem a little later in this chapter. For that code to make sense, we’ll first need to place it into context by looking at filesystems in general and how we operate on them using Perl. Filesystem Differences We’ll start with a quick review of the native filesystems for each of our target operating systems. Some of this may be old news to you, especially if you have significant expe- rience with a particular operating system. Still, it’s worth your while to pay careful attention to the differences between the filesystems (especially the ones you don’t know) if you intend to write Perl code that will work on multiple platforms. Unix All modern Unix variants ship with a native filesystem whose semantics resemble those of their common ancestor, the Berkeley Fast File System (FFS). Different vendors have extended their filesystem implementations in different ways: some filesystems support POSIX access control lists (ACLs) for better security, some support journaling for better recovery, others include the ability to set special file-based attributes, and so on. We’ll be writing code aimed at the lowest common denominator to allow it to work across different Unix platforms. The top, or root, of a Unix filesystem is indicated by a forward slash (/). To uniquely identify a file or directory in a Unix filesystem, we construct a path starting with a slash and then add directories, separating them with forward slashes, as we descend deeper into the filesystem. The final component of this path is the desired directory or filename. Directory and filenames in modern Unix variants are case-sensitive. Almost all ASCII characters can be used in these names if you are crafty enough, but sticking to alpha- numeric characters and some limited punctuation will save you hassle later. Windows-Based Operating Systems All current Windows-based operating systems ship with three supported filesystems: File Allocation Table (FAT), NT FileSystem (NTFS), and FAT32 (an improved version of FAT that allows for larger partitions and smaller cluster sizes). The FAT filesystem found in these operating systems uses an extended version of the basic FAT filesystems found in DOS. Before we look at the extended version, it is im- portant to understand the foibles of the basic FAT filesystem. In basic or real-mode FAT filesystems, filenames conform to the 8.3 specification. This means that file and directory names can consist of a maximum of eight characters, followed by a period (or dot as it is spoken) and a suffix of up to three characters in length. Unlike in Unix, 16 | Chapter 2: Filesystems where a period in a filename has no special meaning, in basic FAT filesystems a filename can contain only a single period as an enforced separator between the name and its extension or suffix. Real-mode FAT was later enhanced in a version called VFAT or protected-mode FAT. This is roughly the version that current operating systems support when they say they use FAT. VFAT hides all of the name restrictions from the user. Longer filenames without separators are supported by a very creative hack: VFAT uses a chain of standard file/directory name slots to transparently shoehorn extended filename support into the basic FAT filesystem structure. For compatibility, every file and directory name can still be accessed using a special 8.3-conforming DOS alias. For instance, the directory called Downloaded Program Files is also available as DOWNLO~1. There are four key differences between a VFAT and a Unix filesystem: • FAT filesystems are case-insensitive. In Unix, an attempt to open a file using the wrong case (i.e., MYFAVORITEFILE versus myfavoritefile) will fail, but with FAT or VFAT, this will succeed with no problem. • Instead of a forward slash, FAT uses the backward slash (\) as its path separator. This has a direct ramification for the Perl programmer, because the backslash is a quoting character in Perl. Paths written in single quotes with only single separators (e.g., $path='\dir\dir\filename') are just fine. However, situations in which you need to place multiple backslashes next to each other (e.g., \\server\dir\file) are potential trouble. In those cases, you have to be vigilant in doubling any multiple backslashes. Some Perl functions and some Perl modules will accept paths with forward slashes, but you shouldn’t count on this convention when programming. It is better to bite the bullet and write \\\\winnt\\temp\\ than to learn that your code breaks because the conversion hasn’t been done for you. • FAT files and directories have special flags associated with them that are called attributes. Example attributes include “Read-only” and “System.” • The root of a FAT filesystem is specified starting with the drive letter on which the filesystem resides. For instance, the absolute path for a file might be specified as c:\home\cindy\docs\resume\current.doc. FAT32 and NTFS filesystems have the same semantics as VFAT filesystems. They share the same support for long filenames and use the same root designator. NTFS is more sophisticated in its name support, however, because it allows these names to be speci- fied using Unicode. Unicode is a multibyte character encoding scheme that can be used to represent all of the characters of all of the written languages on the planet. NTFS also has some functional differences that distinguish it from the other Windows and basic Unix filesystems. Later in this chapter, we will write some code to take ad- vantage of some of these differences, such as filesystem quotas. NTFS supports ACLs, which provide a fine-grained permission mechanism for file and directory access. It also adds some functionality that we won’t touch on, including file encryption and file Filesystem Differences | 17 compression. As a related aside, Vista will only install on an NTFS-formatted filesystem. Before we move on to another operating system, it is important to at least mention the universal naming convention (UNC). UNC is a convention for locating things (files and directories, in our case) in a networked environment. In UNC names, the drive letter and colon preceding the absolute path are replaced with \\server\sharename. This convention suffers from the same Perl backslash syntax clash we saw a moment ago, though, so it is not uncommon to see a set of leaning toothpicks like this: $path = '\\\\server\\sharename\\directory\\file'; Mac OS X At the time the previous edition of this book was written, OS X had just recently appeared on the horizon. Classic Mac OS used a filesystem (Mac OS Hierarchical File System, or HFS) that was a very different beast from any of the filesystems described earlier. It had very different file semantics and required special handling from Perl. Mac OS 8.1 introduced an improved variant of HFS called HFS+, which became the default filesystem format for OS X.† New releases of OS X saw continued development of the filesystem and its capabilities. It has taken some time and a number of releases to get to this point, but the current HFS+ filesystem semantics don’t look very different from any other Unix filesystem at this point. Files and paths are specified the same way, and HFS+ supports BSD extended attributes in the usual way (e.g., ACLs are available). If you stick to the standard Perl mechanisms for interacting with filesystems, you can generally treat HFS+ like any other Unix filesystem. If you do need to muck with an HFS+ filesystem in a nongeneric fashion, as I’ve cavalierly suggested here (i.e., if you really need to get your hands dirty and twiddle bits that are specific to HFS+), you have at least a couple of options: • Call the Mac OS X command-line utilities directly (e.g., using chmod +a..., once fsaclctl has been used to turn on ACLs). • Use Dan Kogai’s MacOSX::File modules. These modules will also give you access to the “legacy” extended attributes (type, creator, etc.) that played a larger role in pre-OS X filesystem use. † As an aside, you can create UFS-formatted filesystems under OS X. Full ZFS support is also on the way as of this writing. 18 | Chapter 2: Filesystems There is one important difference between a standard UFS and a standard HFS+ file- system. By default,‡ HFS+ is case-insensitive (albeit case-preserving): it will treat BillyJoeBob and billyJoebob exactly the same (i.e., if you try to open() the first but the second one is the real name of the file, you will still get a filehandle that points at the file’s data). There’s nothing special you have to do about this difference from a Perl perspective except be very careful about your assumptions. Be especially careful when removing files, because you can sometimes wind up targeting the wrong one. Filesystem Differences Summary Table 2-1 summarizes all of the differences we just discussed, along with a few more items of interest. Table 2-1. Filesystem comparison OS and filesystem Path separator Filename spec. length Absolute path format Relative path format Unique features Unix (Berkeley Fast File System and others) / OS-dependent number of chars /dir/file dir/file OS-variant- dependent additions Mac OS (HFS+) / 255 Unicode chars /dir/file dir/file Mac OS legacy support (e.g., creator/type attributes), BSD extended attributes Windows-based operating systems (NTFS) \ 255 Unicode chars Drive:\dir\file dir\file File encryption and compression DOS (basic FAT) \ 8.3 Drive:\dir\file dir\file Attributes Dealing with Filesystem Differences from Perl Perl can help you write code that takes most of these filesystem quirks into account. It ships with a module called File::Spec that hides some of the differences between the filesystems. For instance, if we pass in the components of a path to the catfile method: use File::Spec; my $path = File::Spec->catfile(qw{home cindy docs resume.doc}); $path is set to home\cindy\docs\resume.doc on a Windows system, while on a Unix or OS X system it becomes home/cindy/docs/resume.doc, and so on. File::Spec also has methods like curdir and updir that return the punctuation necessary to describe the current and parent directories (e.g., “.” and “..”). The methods in this module give you ‡ It is possible to create a case-sensitive HFS+ volume in current versions of OS X, but doing so can be fraught with peril. This practice has been known to break (albeit naively written) applications that did not expect anything but the default semantics. Don’t do this unless you have a really good reason. Filesystem Differences | 19 an abstract way to construct and manipulate your path specifications. If you would prefer not to have to write your code using an object-oriented syntax, the module File::Spec::Functions provides a more direct route to the methods found in File::Spec. If you find File::Spec’s interface to be a little peculiar (e.g., the name catfile() makes sense only if you know enough Unix to understand that the cat command is used to concatenate parts of its input together), there’s a much nicer wrapper by Ken Williams called Path::Class. It doesn’t ship with Perl like File::Spec does, but it is probably worth the extra installation step. Here’s how it works. First, you create either a Path::Class::File or a Path::Class::Dir object using a natural syntax that specifies the path components: use Path::Class; my $pcfile = file(qw{home cindy docs resume.doc}); my $pcdir = dir(qw{home cindy docs}); $pcfile and $pcdir are now both magic. If you use them as you would any other scalar variable (in a case where you “stringify” them), they turn into a path constructed to match the current operating system. For example: print $pcfile; print $pcdir; would yield home/cindy/docs/resume.doc and home/cindy/docs or home\cindy\docs \resume.doc and home\cindy\docs, as we saw earlier with File::Spec. Even though $pcfile and $pcdir stringify into paths that look like strings, they are still objects. And like most other objects, there are methods that can be called on them. These methods include those found in File::Spec and more. Here are some examples: my $absfile = $pcfile->absolute; # returns the absolute path for $pcfile my @contents = $pcfile->slurp; # slurps in the contents of that file $pcfile->remove(); # actually deletes the file There are two more tricks Path::Class can do that are worth mentioning before we move on. First, it can parse existing paths: use Path::Class; # handing it a full path (a string) instead of components my $pcfile = file('/home/cindy/docs/resume.doc'); print $pcfile->dir(); # note: this returns a Path::Class::Dir, # which we're stringify-ing print $pcfile->parent(); # same as dir(), but can make code read better print $pcfile->basename(); # removes the directory part of the name The second trick comes in handy when you want to write code on one operating system that understands the filesystem semantics of another. For example, you may need a web application running on your Linux box to be able to instruct its users on how to 20 | Chapter 2: Filesystems find a file on their local Windows machines. To ask Path::Class to consider the semantics of a different operating system, you need to explicitly import two different methods: foreign_file() and foreign_dir(). These two methods each take the target operating system type as their first argument: use Path::Class qw(foreign_file foreign_dir); my $fpcfile = foreign_file('Win32', qw{home cindy docs resume.doc}); my $fpcdir = foreign_dir('Win32', qw{home cindy}); Now, $fpcfile will yield home\cindy\docs\resume.doc even if the code is run from a Mac. This probably won’t come up often, but it’s very handy when you need it. Walking or Traversing the Filesystem by Hand By now, you’re probably itching to get to some practical applications of Perl. We’ll begin by examining the process of “walking the filesystem,” one of the most common system administration tasks associated with filesystems. Typically this entails searching an entire set of directory trees and taking action based on the files or directories found. Each OS provides a tool for this task: under Unix it’s the find command, under Win- dows it’s Search, and in Mac OS it’s Spotlight or the search box in the Finder (if you aren’t going to run find from a Terminal window). All of these are useful for searching, but they lack the power to perform arbitrary and complex operations by themselves. In this section we’ll explore how Perl allows us to write more sophisticated file-walking code, beginning with the very basics and ratcheting up the complexity as we go on. To get started, let’s take a common scenario that provides a clear problem for us to solve. In this scenario, we’re Unix system administrators with overflowing user file- systems and empty budgets. (We’re picking on Unix first, but the other operating systems will get their turns in a moment.) We can’t add more disk space without money, so we have to make better use of our existing resources. Our first step is to remove all the files on our filesystems that can be eliminated. Under Unix, good candidates for elimination are the core files left around by programs that have died nasty deaths. Most users either do not notice these files or just ignore them, leaving large amounts of disk space claimed for no reason. We need a way to search through a filesystem and delete these varmints. To walk a filesystem by hand, we start by reading the contents of a single directory and work our way down from there. Let’s ease into the process and begin with code that examines the contents of the current directory and reports if it finds either a core file or another directory to be searched. First, we open the directory using roughly the same syntax used for opening a file. If the open fails, we exit the program and print the error message set by the opendir() call ($!): opendir my $DIR, '.' or die "Can't open the current directory: $!\n"; Walking or Traversing the Filesystem by Hand | 21 This provides us with a directory handle, $DIR in this case, which we can pass to readdir() to get a list of all the files and directories in the current directory. If readdir() can’t read that directory, our code prints an error message (which hopefully explains why it failed) and the program exits: # read file/directory names in that directory into @names my @names = readdir $DIR or die "Unable to read current dir:$!\n"; We then close the open directory handle: closedir $DIR; Now we can work with those names: foreach my $name (@names) { next if ($name eq '.'); # skip the current directory entry next if ($name eq '..'); # skip the parent directory entry if (-d $name) { # is this a directory? print "found a directory: $name\n"; next; # can skip to the next name in the for loop } if ($name eq 'core') { # is this a file named "core"? print "found one!\n"; } } That’s all it takes to write some very simple code that scans a single directory. This isn’t even “crawling” a filesystem, though, never mind walking it. To walk the filesystem we’ll have to enter all of the directories we find in the scan and look at their contents as well. If those subdirectories have subdirectories, we’ll need to check them out too. Whenever you have a hierarchy of containers and an operation that gets performed the exact same way on every container and subcontainer in that hierarchy, the situation calls out for a recursive solution (at least to computer science majors). As long as the hierarchy is not too deep and doesn’t loop back on itself (i.e., all containers hold only their immediate children and do not reference other parts of the hierarchy), recursive solutions tend to make the most sense. This is the case with our example; we’re going to be scanning a directory, all of its subdirectories, all of their subdirectories, and so on. If you’ve never seen recursive code (i.e., code that calls itself), you may find it a bit strange at first. Writing recursive code is a bit like painting a set of matryoshka nesting Russian dolls, the largest of which contains a slightly smaller doll of the exact same shape, which contains another doll, and so on until you get to a very small doll in the center. A recipe for painting these dolls might go something like this: 1. Examine the doll in front of you. Does it contain a smaller doll? If so, remove the contents and set aside the outer doll. 2. Repeat step 1 with the contents you just removed until you reach the center. 3. Paint the center doll. When it is dry, put it back in its container doll. 22 | Chapter 2: Filesystems 4. Repeat step 3 with the next-smallest doll until they’re all back in their containers and you’ve painted the last one. The process is the same every step of the way. If the thing in your hand has sub-things, put off dealing with it and deal with the sub-things first. If the thing you have in your hand doesn’t have sub-things, do something with it, and then return to the last thing you put off and work your way back up the chain. In coding terms, this process is typically handled by a subroutine that deals with con- tainers. The routine first looks to see whether the current container has subcontainers. If it does, it calls itself again and again to deal with all of these subcontainers. If it doesn’t, it performs some operation and returns back to the code that called it. If you’re not familiar with code that calls itself, I recommend sitting down with a paper and a pencil and tracing the program flow until you are convinced it actually works. Let’s take a look at some recursive code now. To make our code recursive, we first encapsulate the operation of scanning a directory and acting upon its contents in a subroutine called ScanDirectory(). ScanDirectory() takes a single argument, the di- rectory it is supposed to scan. It figures out the current directory, enters the requested directory, and scans it. When it has completed this scan, it returns to the directory from which it was called. Here’s the new code: #!/usr/bin/perl -s # Note the use of -s for switch processing. Under Windows, you will need to # call this script explicitly with -s (i.e., perl -s script) if you do not # have perl file associations in place. # -s is also considered 'retro' - many programmers prefer to load # a separate module (from the Getopt:: family) for switch parsing. use Cwd; # module for finding the current working directory # This subroutine takes the name of a directory and recursively scans # down the filesystem from that point looking for files named "core" sub ScanDirectory { my $workdir = shift; my $startdir = cwd; # keep track of where we began chdir $workdir or die "Unable to enter dir $workdir: $!\n"; opendir my $DIR, '.' or die "Unable to open $workdir: $!\n"; my @names = readdir $DIR or die "Unable to read $workdir: $!\n"; closedir $DIR; foreach my $name (@names) { next if ( $name eq '.' ); next if ( $name eq '..' ); if ( -d $name ) { # is this a directory? ScanDirectory($name); next; } Walking or Traversing the Filesystem by Hand | 23 if ( $name eq 'core' ) { # is this a file named "core"? # if -r specified on command line, actually delete the file if ( defined $r ) { unlink $name or die "Unable to delete $name: $!\n"; } else { print "found one in $workdir!\n"; } } } chdir $startdir or die "Unable to change to dir $startdir: $!\n"; } ScanDirectory('.'); The most important change from the previous example is our code’s behavior when it finds a subdirectory in the directory it has been requested to scan. If it finds a directory, instead of printing “found a directory!” as our previous sample did, it recursively calls itself to examine that directory first. Once that entire subdirectory has been scanned (i.e., when the call to ScanDirectory() returns), it returns to looking at the rest of the contents of the current directory. To make our code fully functional as a core file-destroyer, we’ve also added file deletion functionality to it. Pay attention to how that code is written: it will only delete files if the script is started with the command-line switch -r (for remove). We’re using Perl’s built-in -s switch for automatic option parsing as part of the invo- cation line (#!/usr/bin/perl -s). This is the simplest way to parse command-line options;* for more sophistication, we’d probably use something from the Getopt:: module family. If a command-line switch is present (e.g., -r), a global scalar variable with the same name (e.g., $r) is set when the script is run. In our code, if Perl is not invoked with -r, we revert to the past behavior of just announcing that a core file has been found. When you write automatic tools, you should make destructive actions harder to perform. Take heed: Perl, like most powerful languages, allows you to nuke your filesystem without breaking a sweat. Now, lest any Windows-focused readers among you think the previous example didn’t apply to you, let me point out that this code could be made useful for you as well. A single line change from: if ($name eq 'core') { to: if ($name eq 'MSCREATE.DIR') { * -s doesn’t play nicely with use strict by default, so don’t use it for anything but the most trivial scripts. 24 | Chapter 2: Filesystems will create a program that deletes the annoying, hidden zero-length files certain Mi- crosoft program installers used to leave behind. Infestation with these files isn’t as much of a problem today as it used to be, but I’m sure some other file will take their place in the list of annoyances. With this code under our belt, let’s return to the quandary that started this chapter. After my laptop kissed the floor, I found myself in desperate need of a way to determine which files could be read off the disk and which were damaged. Here’s the actual code (or a reasonable facsimile) that I used: use Cwd; # module for finding the current working directory $|=1; # turn off I/O buffering sub ScanDirectory { my $workdir = shift; my $startdir = cwd; # keep track of where we began chdir $workdir or die "Unable to enter dir $workdir: $!\n"; opendir my $DIR, '.' or die "Unable to open $workdir: $!\n"; my @names = readdir $DIR; closedir $DIR; foreach my $name (@names) { next if ( $name eq '.' ); next if ( $name eq '..' ); if ( -d $name ) { # is this a directory? ScanDirectory($name); next; } CheckFile($name) or print cwd. '/' . $name . "\n"; # print the bad filename } chdir $startdir or die "Unable to change to dir $startdir:$!\n"; } sub CheckFile { my $name = shift; print STDERR 'Scanning ' . cwd . '/' . $name . "\n"; # attempt to read the directory entry for this file my @stat = stat($name); if ( !$stat[4] && !$stat[5] && !$stat[6] && !$stat[7] && !$stat[8] ) { return 0; } # attempt to open this file open my $T, '<', "$name" or return 0; Walking or Traversing the Filesystem by Hand | 25 # read the file one byte at a time, throw away actual data in $discard for ( my $i = 0; $i < $stat[7]; $i++ ) { my $r = sysread( $T, $discard, 1 ); if ( $r != 1 ) { close $T; return 0; } } close $T; return 1; } ScanDirectory('.'); The difference between this code and our last example is the addition of a subroutine to check each file encountered. For every file, we use the stat() function to see if we can read that file’s directory information (e.g., its size). If we can’t, we know the file is damaged. If we can read the directory information, we attempt to open the file. And for a final test, we attempt to read every single byte of the file. This doesn’t guarantee that the file hasn’t been damaged (the contents could have been modified), but it does at least show that the file is readable. You may wonder why this code uses an esoteric function like sysread() to read the files instead of using < > or read(), Perl’s usual file-reading operator and function. sysread() gives us the ability to read the file byte-by-byte without any of the usual buffering. If a file is damaged at location X, we don’t want to waste time waiting for the standard library routines to attempt to read the bytes at locations X+1, X+2, X+3, and so on as part of their usual pre-fetch; we want the code to quit trying to read the file immediately. In general you will want file reads to fetch whole chunks at a time for performance’s sake, but here that’s undesirable because it would mean the laptop would spend prolonged periods of time making awful noises every time it found a damaged file. Now that you’ve seen the code I used, let me offer some closure to this story. After the script you just saw ran all night long (literally), it found 95 bad files out of 16,000 total. Fortunately, none of those files were files from the book you are now reading. I backed up the good files to another machine and got back to work; Perl saved the day. Walking the Filesystem Using the File::Find Module Now that we’ve explored the basics of filesystem walking, here’s a faster and spiffier way to do it. Perl comes with a module called File::Find that allows it to emulate the Unix find command. The easiest way to begin using this module is to use the find2perl command to generate prototypical Perl code for you. 26 | Chapter 2: Filesystems For instance, let’s say you need some code to search the /home directory for files named beesknees. The command line that uses the Unix find command is: % find /home -name beesknees -print Feed the same options to find2perl: % find2perl /home -name beesknees -print and it produces: #! /usr/bin/perl -w eval 'exec /usr/bin/perl -S $0 ${1+"$@"}' if 0; #$running_under_some_shell use strict; use File::Find (); # Set the variable $File::Find::dont_use_nlink if you're using AFS, # since AFS cheats. # for the convenience of &wanted calls, including -eval statements: use vars qw/*name *dir *prune/; *name = *File::Find::name; *dir = *File::Find::dir; *prune = *File::Find::prune; sub wanted; # traverse desired filesystems File::Find::find({wanted => \&wanted}, '/home'); exit; sub wanted { /^beesknees\z/s && print("$name\n"); } The find2perl-generated code is fairly straightforward. It loads in the necessary Find::File module, sets up some variables for convenient use (we’ll take a closer look at these a little later), and calls File::Find::find with the name of a “wanted” subroutine and the starting directory. We’ll examine this subroutine and its purpose in just a second, since it’s where all of the interesting modifications we’re about to explore will live. Before we begin modifying this code, it’s important to note a few things that may not be obvious just by looking at the sample output: • The folks who have worked on the File::Find module have gone to considerable trouble to make this module portable across platforms. File::Find’s internal rou- tines work behind the scenes so the same Perl code for filesystem walking works for Unix, Mac OS X, Windows, VMS, and so on. Walking the Filesystem Using the File::Find Module | 27 • The code generated by find2perl includes the obsolete use vars pragma, which was replaced by the our() function in Perl 5.6. I suspect it was left this way for backward compatibility. I just wanted to point this out just so you don’t pick up this con- vention by mistake. Now let’s talk about the wanted() subroutine that we will modify for our own purposes. File::Find::find() calls the wanted() subroutine with the current file or directory name once for every file or directory encountered during its filesystem walk. It’s up to the code in wanted() to select the “interesting” files or directories and operate on them accordingly. In the sample output shown earlier, it first checks to see if the file or directory name matches the string beesknees. If the name matches, the && operator causes Perl to execute the print statement to print the name of the file or directory that was found. We’ll have to address two practical concerns when we create our own wanted() sub- routines. Since wanted() is called once per file or directory name, it is important to make the code in this subroutine short and sweet. The sooner we can exit the wanted() sub- routine, the faster the File::Find::find() routine can proceed with the next file or directory, and the speedier the overall program will run. It is also important to keep in mind the behind-the-scenes portability concerns we mentioned a moment ago. It would be a shame to have a portable File::Find::find() call an OS-specific wanted() sub- routine, unless this was unavoidable. Looking at the source code for the File::Find module and the perlport documentation may offer some hints on how to avoid this situation. For our first use of File::Find, let’s rewrite our previous core-destroyer example and then extend it a bit. First we type: % find2perl -name core -print which gives us (in excerpt): use strict; use File::Find (); use vars qw/*name *dir *prune/; *name = *File::Find::name; *dir = *File::Find::dir; *prune = *File::Find::prune; File::Find::find({wanted => \&wanted}, '.'); sub wanted { /^core\z/s && print("$name\n"); } Then we add -s to the Perl invocation line and modify the wanted() subroutine: 28 | Chapter 2: Filesystems my $r; sub wanted { /^core$/ && print("$name\n") && defined $r && unlink($name); } This gives us the desired deletion functionality when the user invokes the program with -r. Here’s a tweak that adds another measure of protection to our potentially destruc- tive code: my $r; sub wanted { /^core$/ && -s $name && print("$name\n") && defined $r && unlink($name); } This checks any file called core to see if it is a non-zero-length file before printing the name or contemplating deletion. Sophisticated users sometimes create a link to /dev/null named core in their home directories to prevent inadvertent core dumps from being stored in those directories. The -s test makes sure we don’t delete links or zero-length files by mistake. If we wanted to be even more diligent, we could make two additional checks: 1. Open and examine the file to confirm that it is an actual core file, either from within Perl or by calling the Unix file command. Determining whether a file is an au- thentic core dump file can be tricky when you have filesystems remotely mounted over a network by machines of different architectures, all with different core file formats. 2. Look at the modification date of the file. If someone is actively debugging a pro- gram, she may not be happy if you delete the core file out from under her. Before we look at any more code, it would probably be helpful to explain those mysterious variable aliasing lines: *name = *File::Find::name; *dir = *File::Find::dir; *prune = *File::Find::prune; Find::File makes a number of variables available to the wanted() subroutine as it runs. The important ones are listed in Table 2-2. Table 2-2. File::Find variables Variable name Meaning $_ Current filename $File::Find::dir Current directory name $File::Find::name Full path of current filename (e.g., "$File::Find::dir/$_") Walking the Filesystem Using the File::Find Module | 29 We’ll see how these are used in our next code example. Let’s take a break from the Unix world for a bit and look at a couple of Windows- specific examples. We could make a small modification to our previous code to have it search the entire filesystem of the current drive for hidden files (i.e., those with the HIDDEN attribute set). This example works on both NTFS and FAT filesystems: use File::Find (); use Win32::File; File::Find::find( { wanted => \&wanted }, '\\' ); my $attr; # defined globably instead of in wanted() to avoid repeatedly # defining a local copy of $attr every time it is called sub wanted { -f $_ && ( Win32::File::GetAttributes( $_, $attr ) ) && ( $attr & HIDDEN ) && print "$File::Find::name\n"; } Here’s an NTFS-specific example that will look for all files that have Full Access explicitly enabled for the special group Everyone and print their names: use File::Find; use Win32::FileSecurity; # determine the DACL mask for Full Access my $fullmask = Win32::FileSecurity::MakeMask(qw(FULL)); File::Find::find( { wanted => \&wanted }, '\\' ); sub wanted { # this time we're happy to make sure we get a fresh %users each time my %users; ( -f $_ ) && eval {Win32::FileSecurity::Get( $_, \%users )} && ( defined $users{'Everyone'} ) && ( $users{'Everyone'} == $fullmask ) && print "$File::Find::name\n"; } In this code, we query the access control list for all files, checking whether that list includes an entry for the group Everyone. If it does, we compare the Everyone entry to the value for Full Access (computed by MakeMask()), printing the absolute path of the file when we find a match. 30 | Chapter 2: Filesystems You may be curious about the eval() call that popped up in the previous code sample. Despite what the documentation says about Win32::File Security nicely returning errors in $!, when it encounters certain situa- tions it instead throws a snit fit and exits abruptly. This is listed as a bug in the docs, but that’s easy to miss. Unfortunately, two common things give this module dyspepsia: the presence of a paging file it can’t read, and the presence of a null DACL (a discretionary ACL set to null). We use eval() to trap and ignore this antisocial behavior. As a related aside, some parts of the OS (e.g., Explorer) also treat a null DACL as giving the same access to Everyone our code tries to find. If we wanted to display the files with this condition, we could check $@. Here is another real-life example of how useful even simple code can be. Many moons ago, I was attempting to defragment the NTFS partition on a laptop when the software reported a “Metadata Corruption Error.” Perusing the website of the vendor who made the defragmentation software, I encountered a tech support note that suggested, “This situation can be caused by a long filename which contains more characters than is legal under Windows NT.” It then suggested locating this file by copying each folder to a new location, comparing the number of files in the copy to the original, and then, if the copied folder has fewer files, identifying which file(s) in the original folder did not get copied to the new location. This seemed like a ridiculous suggestion given the number of folders on my NT partition and the amount of time it would take. Instead, I whipped up the following code (edited to use current-day syntax) in about a minute using the methods we’ve been discussing: use File::Find; my $max; my $maxlength; File::Find::find( { wanted => \&wanted }, '.' ); print "max:$max\n"; sub wanted { return unless -f $_; if ( length($_) > $maxlength ) { $max = $File::Find::name; $maxlength = length($_); } if ( length($File::Find::name) > 200 ) { print $File::Find::name, "\n"; } } This printed out the names of all the files with names longer than 200 characters, fol- lowed by the name of the largest file found. Job done, thanks to Perl. Walking the Filesystem Using the File::Find Module | 31 When Not to Use the File::Find Module When is the File::Find method we’ve been discussing not appropriate? Three situa- tions come to mind: 1. If the filesystem you are traversing does not follow the normal semantics, you can’t use it. For instance, in the bouncing laptop scenario described at the beginning of the chapter, the Linux NTFS filesystem driver I was using had the strange property of not listing “.” or “..” in empty directories. This broke File::Find badly. 2. If you need to change the names of the directories in the filesystem you are tra- versing while you are traversing it, File::Find gets very unhappy and behaves in an unpredictable way. 3. If you need to walk a nonnative filesystem mounted on your machine (for example, an NFS mount of a Unix filesystem on a Windows machine), File::Find will attempt to use the native operating system’s filesystem semantics. It is unlikely that you’ll encounter these situations, but if you do, refer to the first filesystem-walking section of this chapter for information on how to traverse filesystems by hand. Let’s return to Unix to close this section with a more complex example. One idea that seems to get short shrift in many system administration contexts (but can yield tre- mendous benefit in the end) is the notion of empowering the user. If your users can fix their own problems with tools you provide, everybody wins. Much of this chapter is devoted to dealing with problems that arise from filesystems being filled. Often this occurs because users do not know enough about their environ- ment, or because it is too cumbersome to perform any basic disk-space management. Many a support request starts with “I’m out of disk space in my home directory and I don’t know why.” Here’s a bare-bones version of a script called needspace that can help users with this problem. All the user has to do is type needspace, and the script attempts to locate items in that user’s home directory that could be deleted. It looks for two kinds of files: known core/backup files and those that can be recreated automatically. Let’s dive into the code: use File::Find; use File::Basename; use strict; # hash of fname extensions and the extensions they can be derived from my %derivations = ( '.dvi' => '.tex', '.aux' => '.tex', '.toc' => '.tex', '.o' => '.c', ); my %types = ( 'emacs' => 'emacs backup files', 32 | Chapter 2: Filesystems 'tex' => 'files that can be recreated by running La/TeX', 'doto' => 'files that can be recreated by recompiling source', ); my $targets; # we'll collect the files we find into this hash of hashes my %baseseen; # for caching base files We start by loading the libraries we need: our friend File::Find and another useful library called File::Basename. File::Basename will come in handy for parsing path- names. We then initialize a hash table with known derivations; for instance, we know that running the command TeX or LaTeX on the file happy.tex can generate the file happy.dvi, and that happy.o could possibly be created by running a C compiler on happy.c. I say “possibly” because sometimes multiple source files are needed to generate a single derived file. We can only make simple guesses based on file extensions; gen- eralized dependency analysis is a complex problem we won’t attempt to touch here. Next, we locate the user’s home directory by finding the user ID of the person running the script ($<) and feeding it to getpwuid(). getpwuid() returns password information in list form (more on this in Chapter 3), from which an array index ([7]) selects the home directory element. There are shell-specific ways to retrieve this information (e.g., querying the $HOME environment variable), but the code as written is more portable. Once we have the home directory, we enter it and begin scanning using a find() call just like the ones we’ve seen before: my $homedir = ( getpwuid($<) )[7]; # find the user's home directory chdir($homedir) or die "Unable to change to your homedir $homedir:$!\n"; $| = 1; # print to STDOUT in an unbuffered way print 'Scanning'; find( \&wanted, '.' ); print "done.\n"; Here’s the wanted() subroutine we call. It starts by looking for core files and emacs backup and autosave files. We assume these files can be deleted without checking for their source file (perhaps not a safe assumption). If one of these files is found, its size and location is stored in a hash of hashes whose inner key is the path to the file with a value that is the size of that file. The remaining checks for derivable files are very similar. They call the routine BaseFileExists() to check whether a particular file can be derived from another file in that directory. If this routine returns true, we store the filename and size info for later retrieval: sub wanted { # print a dot for every dir so the user knows we're doing something print '.' if ( -d $_ ); Walking the Filesystem Using the File::Find Module | 33 # we're only checking files return unless -f $_; # check for core files $_ eq 'core' && ( $targets->{core}{$File::Find::name} = ( stat(_) )[7] ) && return; # check for emacs backup and autosave files ( /^#.*#$/ || /~$/ ) && ( $targets->{emacs}{$File::Find::name} = ( stat(_) )[7] ) && return; # check for derivable tex files ( /\.dvi$/ || /\.aux$/ || /\.toc$/ ) && BaseFileExists($File::Find::name) && ( $targets->{tex}{$File::Find::name} = ( stat(_) )[7] ) && return; # check for derivable .o files /\.o$/ && BaseFileExists($File::Find::name) && ( $targets->{doto}{$File::Find::name} = ( stat(_) )[7] ) && return; } Here’s the routine that checks whether a particular file can be derived from another “base” file in the same directory (i.e., whether happy.c exists if we find happy.o): sub BaseFileExists { my ( $name, $path, $suffix ) = File::Basename::fileparse( $_[0], '\..*' ); # if we don't know how to derive this type of file return 0 unless ( defined $derivations{$suffix} ); # easy, we've seen the base file before return 1 if ( defined $baseseen{ $path . $name . $derivations{$suffix} } ); # if file (or file to which link points) exists and has non-zero size # return success once we have cached the information return 1 if ( -s $name . $derivations{$suffix} && ++$baseseen{ $path . $name . $derivations{$suffix} } ); } Here’s how this code works: 1. File::Basename::fileparse() is used to separate the path into a filename, its lead- ing path, and its suffix (e.g., resume.dvi, /home/cindy/docs/, .dvi). 2. This file’s suffix is checked to determine if it is one we recognize as being derivable. If not, we return 0 (“false” in a scalar context). 3. We check whether we’ve already seen a “base file” for this particular file, and if so return true. In some situations (TeX/LaTeX in particular), a single base file can yield 34 | Chapter 2: Filesystems many derived files. This check speeds things up considerably because it saves us a trip to the filesystem. 4. If we haven’t seen a base file for this file before, we check to see if one exists and, if so, that it’s length is non-zero. If so, we cache the base file information and return 1 (“true” in a scalar context). All that’s left for us to do now is to print out the information we gathered as we walked the filesystem: foreach my $path ( keys %{ $targets->{core} } ) { print 'Found a core file taking up ' . BytesToMeg( $targets->{core}{$path} ) . 'MB in ' . File::Basename::dirname($path) . ".\n"; } foreach my $kind ( sort keys %types ) { ReportDerivFiles( $kind, $types{$kind} ); } sub ReportDerivFiles { my $kind = shift; # kind of file we're reporting on my $message = shift; # a message so we can describe it my $tempsize = 0; return unless exists $targets->{$kind}; print "\nThe following are most likely $message:\n"; foreach my $path ( keys %{ $targets->{$kind} } ) { $tempsize += $targets->{$kind}{$path}; $path =~ s|^\./|~/|; # change the path for prettier output print "$path ($targets->{$kind}{$path} bytes)\n"; } print 'These files take up ' . BytesToMeg($tempsize) . "MB total.\n\n"; } sub BytesToMeg { # convert bytes to X.XXMB return sprintf( "%.2f", ( $_[0] / 1024000 ) ); } Before I close this section, I should note that we could extend the previous example in many ways. The sky’s really the limit with this sort of program. Here are a few ideas: • Search for web browser cache directories (a common source of missing disk space). • Offer to delete files that are found. The operator unlink() and the subroutine rmpath from the File::Path module could be used to perform the deletion step. • Perform more analysis on files instead of making guesses based on filenames. Walking the Filesystem Using the File::Find Module | 35 Walking the Filesystem Using the File::Find::Rule Module File::Find provides an easy and easy-to-understand way to walk filesystems. It has the added benefit of shipping with Perl. But after you’ve written a number of File::Find- based scripts, you may notice that you tend to write the same kind of code over and over again. At that point you might start to wonder if there are any ways to avoid repeating yourself in this fashion besides just working from a standard boilerplate you create. If you are not constrained to modules that ship with Perl, I’m happy to say there is: File::Find::Rule. File::Find::Rule is a fabulous module by Richard Clamp (actually, potentially a family of modules, as you’ll see in a second) that offers two very slick interfaces to File::Find. Once you have the hang of File::Find, I definitely recommend that you check out File::Find::Rule. Let’s take a look at what makes it so cool. First off, Clamp’s module makes writing scripts that collect lists of certain files or di- rectories from the filesystem much easier. With File::Find you have to handle both the selection and the accumulation tasks by hand, but File::Find::Rule does the work for you. You tell it where to begin its file walk and then provide either a series of chained methods or a list of arguments that describe the filesystem objects that interest you. It then produces either a list of what you were looking for or a way of iterating over each item it finds, one at a time. Let’s start with the simplest expression and build from there: use File::Find::Rule; my @files_or_dirs = File::Find::Rule->in('.'); @files_or_dirs now contains all of the files and directories found in the current direc- tory or its subdirectories, as specified by the in() method. If we only wanted the files and not the directories, we could add file(): my @files = File::Find::Rule->file()->in('.'); Or if we only wanted files that ended with .pl (i.e., probably the Perl files): my @perl_files = File::Find::Rule->file()->name('*.pl')->in('.'); and so on. As you can see, we are just adding more methods into the chain that essen- tially act as filters. File::Find::Rule also offers a procedural interface, so if you’d prefer something that was less object-oriented, you would rewrite the previous line of code to say: my @perl_files = find( file => name => '*.pl', in => '.' ); I don’t find that format as easy to read, but some people may prefer it. Before I show you the second impressive thing about this module, I should mention that File::Find::Rule provides an iterator-based interface as well. This is handy for those cases where your selection can return a large number of items. For instance, if you asked for all of the readable files on just my laptop, the resulting array would have more than a million elements in it. It addition to that being a pretty sizable chunk of data to keep in memory, it would also take a decent amount of time to collect. You 36 | Chapter 2: Filesystems may prefer to get operating on the files as they are found, rather than twiddling your thumbs until they are all returned as a list. This is where an iterator comes in handy. To use this feature, we would call start() instead of in() at the beginning (or end, depending on your point of view) of the method chain: my $ffr = File::Find::Rule->file()->name('*.pl')->start('.'); This code returns an object that has a match() method. match() will hand you back the very next match found (or false if there are none) every time you call it: while ( my $perl_file = $ffr->match ){ # do something interesting with $perl_file } This allows you to walk the filesystem one matching item at a time (kind of like the wanted() subroutine we saw before, but better because you’re only handed the things you want). Now on to the second benefit of using File::Find::Rule. You’ve probably already guessed that you can construct some fairly complex chains of filter methods to get back exactly what you want. If you want all of the executable Perl files over a certain size owned by a particular set of UIDs, for example, that’s no problem at all. That code just looks like this: use File::Find::Rule; @interesting = File::Find::Rule ->file() ->executable() ->size('<1M') ->uid( 6588, 6070 ) ->name('*.pl') ->in('.'); If you’ve already peeked at the File::Find::Rule documentation, you may have noticed that you can construct chains held together by not just a Boolean “and” relationship (“true if it is this and that and that...”). File::Find::Rule lets you use or() and any() to find things that have “this or that or that...” or have “at least one of any of these things true.” You may also have noticed that there is a grep() method that can look at the contents of files found as yet another filter. But that’s still not the coolest part. Richard Clamp has designed his module so that other people can add filter methods in a seamless fashion. On first blush that may not seem all that impressive, but wait until you see some of the filter modules that are available. Here’s a small taste: • File::Find::Rule::VCS by Adam Kennedy adds methods that make it easy to ignore the administrative files kept around by various source control systems, such as CVS, Subversion, and Bazaar. • File::Find::Rule::PPI by the same author lets you search for Perl files that contain specific Perl elements (e.g., all of the files that have POD documentation, or even Walking the Filesystem Using the File::Find::Rule Module | 37 those that use subroutines). This isn’t just a simple grep(); it actually parses each of the files. • File::Find::Rule::ImageSize, an add-on by creator Richard Clamp, lets you select or reject images based on their size. • File::Find::Rule::Permissions by David Cantrell lets you select files and direc- tories based on a given user’s permissions (e.g., “can nobody change any files in this directory?”) • File::Find::Rule::MP3Info by Kake Pugh lets you find MP3 files based on arbitrary MP3 tags (e.g., find all songs by a certain artist, or over six minutes in length). Really. We’ll see this module again in the last chapter. There are quite a few more modules in this family. Most of them are a little more generic than the ones just listed (e.g., to allow you to search for files with various file permis- sions or ages), but I wanted to give you a sampling so you could see how powerful this idiom can be. Manipulating Disk Quotas Perl scripts like our core-killers from the last section can offer a way to deal with junk files that cause unnecessary disk-full situations. But even when run on a regular basis, they are still a reactive approach; the administrator deals with these files only after they’ve come into existence and cluttered the filesystem. There’s another, more proactive approach: filesystem quotas. Filesystem quotas, op- erating system permitting, allow you to constrain the amount of disk space a particular user can consume on a filesystem. All of the operating systems in play in this book support them in one form or another. Though proactive, this approach is considerably more heavy-handed than cleanup scripts because it applies to all files, not just spurious ones like core dumps. Most system administrators find using a combination of automated cleanup scripts and quotas to be the best strategy: the former help prevent the latter from being necessary. In this section, we’ll mostly deal with manipulating Unix quotas from Perl (we’ll take a quick peek at NTFS quotas at the end of the chapter). Before we get into Unix quota scripting, however, we should take a moment to understand how quotas are set and queried “by hand.” To enable quotas on a filesystem, a Unix system administrator usually adds an entry to the filesystem mount table (e.g., /etc/fstab or /etc/vfstab) and then reboots the system or manually invokes the quota enable command (usually quotaon). Here’s an example /etc/vfstab from a Solaris box: #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options /dev/dsk/c0t0d0s7 /dev/rdsk/c0d0t0d0s7 /home ufs 2 yes rq 38 | Chapter 2: Filesystems The rq option in the last column enables quotas on this filesystem. They are stored on a per-user basis. To view the quota entries for a user on all of the mounted filesystems that have quotas enabled, one can invoke the quota command like so: $ quota -v sabrams to produce output similar to this: Disk quotas for sabrams (uid 670): Filesystem usage quota limit timeleft files quota limit timeleft /home/users 228731 250000 253000 0 0 0 For our next few examples, we’re only interested in the first three numeric columns of this output. The first number is the current amount of disk space (in 1,024-byte blocks) being used by the user sabrams on the filesystem mounted at /home/users. The second is that user’s “soft quota.” The soft quota is the amount after which the OS begins complaining for a set period of time, but does not restrict space allocation. The final number is the “hard quota,” the absolute upper bound for this user’s space usage. If a program attempts to request more storage space on behalf of the user after this limit has been reached, the OS will deny this request and return an error message like “disk quota exceeded.” If we wanted to change these quota limits by hand, we’d traditionally use the edquota command. edquota pops you into your editor of choice (specified by setting the EDITOR environment variable in your shell), preloaded with a small temporary text file containing the pertinent quota information. Here’s an example buffer that shows a user’s limits on each of the four quota-enabled filesystems. This user most likely has her home directory on /exprt/server2, since that’s the only filesystem where she has quotas in place: fs /exprt/server1 blocks (soft = 0, hard = 0) inodes (soft = 0, hard = 0) fs /exprt/server2 blocks (soft = 250000, hard = 253000) inodes (soft = 0, hard = 0) fs /exprt/server3 blocks (soft = 0, hard = 0) inodes (soft = 0, hard = 0) fs /exprt/server4 blocks (soft = 0, hard = 0) inodes (soft = 0, hard = 0) Using edquota to make changes by hand may be a comfy way to edit a single user’s quota limits, but it is not a viable way to deal with tens, hundreds, or thousands of user accounts. Still, as you’ll see, it can be useful. One of Unix’s flaws is its lack of command-line tools for editing quota entries. Most Unix variants have C library routines for this task, but no Unix variant vendors ship common command-line tools that allow for higher-level scripting. True to the Perl motto “There’s more than one way to do it” (TMTOWTDI, pronounced “tim-toady”), we are going to look at two very different ways of setting quotas from Perl: performing some tricks with edquota and using the Quota module. Manipulating Disk Quotas | 39 The New Tradition? There’s actually a third way we could try, but I’m not going to demonstrate it because it is far less portable. As time has gone on, some Unix variant vendors have provided command-line tools for quota editing that can be called from Perl. For example, much of the Linux world has a setquota command. However, these are not universally avail- able, do not take the same command-line arguments, and so on. There is a package called quotatool, written by Mike Glover and maintained by Johan Ekenberg, that attempts to provide a more cross-platform utility for quota editing. It can be found at quotatool is pretty spiffy, but I’m still going to show you how to manipulate edquota instead, for two reasons: quotatool may not be available on the system you are using, and, more importantly, the technique used can be a real lifesaver for things other than just quota editing. If you learn how to use this technique in this context it will likely serve you well in others. Editing Quotas with edquota Trickery The first method involves a little trickery on our part. I just described the process for manually setting a single user’s quota: the edquota command invokes an editor to allow you to edit a small text file and then uses any changes to update the quota entries. There’s nothing in this scenario mandating that an actual human has to type at a key- board to make changes in the editor, though. In fact, there’s not even a constraint on which editor has to be used. All edquota needs is a program it can launch that will properly change a small text file. Any valid path (as specified in the EDITOR environment variable) to such a program will do. Why not point edquota at a Perl script? In this next example, we’ll look at just such a script. Our example script will need to do double duty. First, it has to get some command- line arguments from the user, set EDITOR appropriately, and call edquota. edquota will then run another copy of our program to do the real work of editing this temporary file. Figure 2-1 shows a diagram of the action. The initial program invocation must tell the second copy what to change. How it passes this information is less straightforward than one might hope. The manual page for edquota says: “The editor invoked is vi(1) unless the EDITOR environment variable speci- fies otherwise.” The idea of passing command-line arguments via EDITOR or another environment variable is a dicey prospect at best, because we don’t know how edquota will react. Instead, we’ll have to rely on one of the other interprocess communication methods available in Perl. See the sidebar “Can We Talk?” on page 42 for some of the possibilities. 40 | Chapter 2: Filesystems 1 2 3 4 autoedquota script runs edquota progam with EDITOR environment variable set. edquota writes a temporary file and then spawns a second copy of autoedquota script. Second copy of autoedquota modifies the temporary file. edquota reads temporary file back in and makes changes to quota.autoedquota autoedquota 1 EDITOR = autoedquota edquota /tmp/file autoedquota 1 edquota autoedquota 2 edquota autoedquota 2 /tmp/file autoedquota 1 edquota autoedquota 2 /tmp/file 1 filesystem quota Figure 2-1. Changing quotas using a “sleight-of-hand” approach Manipulating Disk Quotas | 41 In our case, we’re going to choose a simple but powerful method to exchange infor- mation. Since the first process only has to provide the second one with a single set of change instructions (what quotas need to be changed and their new values), we’re going to set up a standard Unix pipe between the two of them.† The first process will print a change request to its output, and the copy spawned by edquota will read this info as its standard input. Can We Talk? When you need two processes to exchange information with each other via Perl, there are a number of things you can do, including: • Pass a temporary file between them. • Create a named pipe and talk over that. • Pass AppleEvents (under Mac OS X). • Use mutexes or mutually agreed upon registry keys (under Windows). • Have them meet at a network socket. • Use a shared memory section. It’s up to you as the programmer to choose the appropriate communication method, though often the data will dictate this for you. When looking at this data, you’ll want to consider: • Direction of communication (one- or two-way?) • Frequency of communication (is this a single message or are there multiple chunks of information that need to be passed?) • Size of data (is it a 10 MB file or 20 characters?) • Format of data (is it a binary file or just text characters? fixed width, or character separated?) Finally, be conscious of how complicated you want to make your script. Let’s write the program. The first thing the program has to do when it starts up is decide what role it’s been asked to play. We can assume that the first invocation receives several command-line arguments (i.e., what to change) while the second, called by edquota, receives only one (the name of the temporary file). The program forces a set of command flags to be present if it is called with more than one argument, so we’re pretty safe in using this assumption as the basis of our role selection. Here’s the code that decides which role the script is being called for (e.g., if it’s being the $EDITOR) and handles calling edquota if necessary: † Actually, the pipe will be to the edquota program, which is kind enough to hook up its input and output streams to the Perl script being spawned. 42 | Chapter 2: Filesystems #!/usr/bin/perl use Getopt::Std; use File::Temp qw(tempfile); my $edquota = '/usr/sbin/edquota'; # edquota path my $autoedq = '/bin/'; # full path for this script my %opts; # are we the first or second invocation? # if there is more than one argument, we're the first invocation # so parse the arguments and call the edquota binary if ( @ARGV != 1 ) { # colon (:) means this flag takes an argument # $opts{u} = user ID, $opts{f} = filesystem name, # $opts{s} = soft quota amount, $opts{h} = hard quota amount getopt( 'u:f:s:h:', \%opts ); die "USAGE: $0 -u -f -s -h \n" unless ( exists $opts{u} and exists $opts{f} and exists $opts{s} and exists $opts{h} ); CallEdquota(); } # else - we're the second invocation and will have to perform the edits else { EdQuota(); } The code to actually call edquota over a pipe is pretty simple: sub CallEdquota { $ENV{'EDITOR'} = $autoedq; # set the EDITOR variable to point to us open my $EPROCESS, '|-', "$edquota $opts{u}" or die "Unable to start $edquota: $!\n"; # send the changes line to the second script invocation print $EPROCESS "$opts{f}|$opts{s}|$opts{h}\n"; close $EPROCESS; } Here’s the second part of the action (the part of the script that edits the file edquota hands it): sub EdQuota { my $tfile = $ARGV[0]; # get the name of edquota's temp file Manipulating Disk Quotas | 43 open my $TEMPFILE, '<', $tfile or die "Unable to open temp file $tfile:$!\n"; my ( $SCRATCH_FH, $scratch_filename ) = tempfile() or die "Unable to open scratch file: $!\n"; # receive line of input from first invocation and lop off the newline chomp( my $change = ); my ( $fs, $soft, $hard ) = split( /\|/, $change ); # parse the communique # Read in a line from the temp file. If it contains the # filesystem we wish to modify, change its values. Write the input # line (possibly changed) to the scratch file. while ( my $quotaline = <$TEMPFILE> ) { if ( $quotaline =~ /^fs \Q$fs\E\s+/ ) { $quotaline =~ s/(soft\s*=\s*)\d+(, hard\s*=\s*)\d+/$1$soft$2$hard/; } print $SCRATCH_FH $quotaline; } close $TEMPFILE; close $SCRATCH_FH; # overwrite the temp file with our modified scratch file so # edquota will get the changes rename( $scratch_filename, $tfile ) or die "Unable to rename $scratch_filename to $tfile: $!\n"; } This code will only work if: 1. The script starts with the “shebang” line at the beginning that indicates that it should call the Perl interpreter (rather than being treated like a shell script). 2. The file itself is marked as being executable: chmod o+x /bin/ The preceding code is bare bones, but it still offers a way to make automated quota changes. If you’ve ever had to change many quotas by hand, this should be good news. Before putting something like this into production, considerable error checking and a mechanism that prevents multiple concurrent changes should be added. In any case, you may find this sort of sleight-of-hand technique useful in other situations besides quota manipulation. Editing Quotas Using the Quota Module Once upon a time, the previous method (to be honest, the previous hack) was the only way to automate quota changes. Perl’s XS extension mechanism provides a way to glue the C quota library into Perl, however, so it was only a matter of time before someone produced a Quota module for Perl. Thanks to Tom Zoerner and some other porting 44 | Chapter 2: Filesystems help, setting quotas from Perl is now much more straightforward—if this module sup- ports your variant of Unix. If it doesn’t, the previous method should work fine. Here’s some sample code that takes the same arguments as our last quota-editing example: use Getopt::Std; use Quota; my %opts; getopt( 'u:f:s:h:', \%opts ); die "USAGE: $0 -u -f -s -h \n" unless ( exists $opts{u} and exists $opts{f} and exists $opts{s} and exists $opts{h} ); my $dev = Quota::getqcarg( $opts{f} ) or die "Unable to translate path $opts{f}: $!\n"; my ( $curblock, $soft, $hard, $btimeo, $curinode, $isoft, $ihard, $itimeo ) = Quota::query( $dev, $opts{u} ) or die "Unable to query quota for $opts{u}: $!\n"; Quota::setqlim( $dev, $opts{u}, $opts{s}, $opts{h}, $isoft, $ihard ) == undef or die 'Unable to set quota: ' . Quota::strerr() . "\n"; After we parse the arguments, there are three simple steps. First, we use Quota::getqcarg() to get the correct device identifier to feed to the other quota routines. Next, we feed this identifier and the user ID to Quota::query() to get the current quota settings, which we need in order to avoid perturbing the quota limits we are not inter- ested in changing (e.g., the number of files). Finally, we set the quota. That’s all it takes: three lines of Perl code. Remember, the Perl slogan states that “there’s more than one way to do it,” not nec- essarily “several equally good ways.” Editing NTFS Quotas Under Windows There are basically two strata to be considered when you start talking about quotas in the current Windows-based operating systems. On the basic level, each NTFS filesys- tem can enforce quotas on a per-user, per-volume basis (i.e., a particular user can use only X amount of space on volume Y). The users can either be local to the machine or be found in Active Directory. Windows Server 2003R2 enhances this model by offering per-volume and per-folder quotas that are not tied to individual users. The second layer to the quota story comes into play when you’re administering quotas for people on a whole set of machines. It would be impractical to set up the quotas on each individual machine, so instead group policy objects (GPOs) are used to specify quota policies that can be applied to multiple machines in an organizational unit (OU). Editing NTFS Quotas Under Windows | 45 In this section we’re only going to look at the first layer, because the creation and maintenance of GPOs is a little too far off our current path to really explore here. For more info on how to mess with GPOs from Perl, I recommend you check out Robbie Allen and Laura Hunter’s excellent Active Directory Cookbook (O’Reilly). And at the risk of increasing the hand waving to the level of a comfortable breeze, the code shown here will come with only a modicum of explanation. It uses the Windows Management Instrumentation (WMI), a technology we’ll explore in depth in Chapter 4. If you’re not already familiar with WMI, I’d recommend bookmarking this page and coming back to it after you’ve had a chance to peruse the WMI discussion in Chapter 4. Here’s some code to create a quota entry for a user named dnb who lives in the domain called WINDOWS. The entry gets created for the C: volume of the local machine (or, if it already exists, its values are set): use Win32::OLE; my $wobj = Win32::OLE->GetObject('winmgmts:\\\\.\\root\\cimv2'); # next line requires elevated privileges to work under Vista my $quota = $wobj->Get( 'Win32_DiskQuota.QuotaVolume=\'Win32_LogicalDisk.DeviceID="c:"\',' . 'User=\'Win32_Account.Domain="WINDOWS",Name="dnb"\'' ); $quota->{Limit} = 1024 * 1024 * 100; # 100MB $quota->{WarningLimit} = 1024 * 1024 * 80; # 80MB $quota->Put_; In short, this script first gets an object that references the WMI namespace. It then uses that object to retrieve an object representing the quota entry for the user (identified by the domain and username) and volume (the volume c:) we care about. With that object in hand, we can set the two properties of interest (Limit and WarningLimit) and push the changes back via Put_ to make them active.‡ If we wanted to just query the existing data, we could read those properties instead of setting them and leave off the call to Put_. Note that in order to perform these operations under Vista, you will need to run the script with elevated privileges (not just from an administrator account); see Chap- ter 1 for more information. Querying Filesystem Usage We’ve just explored a variety of methods of controlling filesystem usage, and it’s only natural to want to keep track of how well they work. Let’s look at a method for querying the filesystem usage on each of the operating systems discussed in this book. ‡ In case you’re curious, to set the disk quota to “no limit,” the Scripting Guys at Microsoft say you need to set the value to 18446744073709551615 (seriously). See this column for details: technet/scriptcenter/resources/qanda/jan08/hey0128.mspx. 46 | Chapter 2: Filesystems If we wanted to query filesystem usage on a Windows machine, we could use Mike Blazer’s Win32::DriveInfo module: use Win32::DriveInfo; my ($sectors, $bytessect, $freeclust, $clustnum, $userfree, $total, $totalfree ) = Win32::DriveInfo::DriveSpace('c'); # if quotas are in effect we can show the amount free from # our user's perspective by printing $userfree instead print "$totalfree bytes of $total bytes free\n"; Win32::DriveInfo can also provide other information, such as which directory letters are active and whether a drive (e.g., a CD-ROM) is in use, so it’s handy to have around. Several Unix modules are also available, including Filesys::DiskSpace by Fabien Tassin, Filesys::Df by Ian Guthrie, and Filesys::DiskFree by Alan R. Barclay. The first two of these make use of the Unix system call statvfs(), while the last one actually parses the output of the Unix command df on all of the systems it supports. Choosing between these modules is mostly a matter of personal preference and operating system support. I prefer Filesys::Df because it offers a rich feature set and does not spawn another process (a potential security risk, as discussed in Chapter 1) as part of a query. Here’s one way to write code equivalent to the previous example: use Filesys::Df; my $fobj = df('/'); print $fobj->{su_bavail}* 1024 . ' bytes of ' . $fobj->{su_blocks}* 1024 . " bytes free\n"; We have to do a little bit of arithmetic (i.e., * 1024) because Filesys::Df returns values in terms of blocks, and each block is 1024 bytes on our system. (The df() function for this module can be passed a second optional argument for block size if necessary.) Also worth noting about this code are the two hash values we’ve requested. su_bavail and su_blocks are the values returned by this module for the “real” size and disk usage information. On most Unix filesystems, the df command will show a value that hides the standard 10% of a disk set aside for superuser overflow. If we wanted to see the total amount of space available and the current amount free from a normal user’s per- spective, we would have used user_blocks and user_bavail instead. Guthrie has also written a related module called Filesys::DfPortable, which has very similar syntax to Filesys::Df. It adds Windows support for essentially the same type of disk usage queries. If you don’t need the additional information about your drives that Win32::DriveInfo provides, it may suit your purposes on those platforms as well. With the key pieces of Perl code we’ve just seen, it is possible to build more sophisti- cated disk monitoring and management systems. These filesystem watchdogs will help you deal with space problems before they occur. Querying Filesystem Usage | 47 Module Information for This Chapter Name CPAN ID Version MacOSX::File DANKOGA 0.71 File::Find (ships with Perl) 1.12 File::Spec (ships with Perl as part of the PathTools module) KWILLIAMS 3.2701 Path::Class KWILLIAMS 0.16 Cwd (ships with Perl as part of the PathTools module) KWILLIAMS 3.2701 Win32::File (ships with ActiveState Perl) JDB 0.06 Win32::FileSecurity (ships with ActiveState Perl) JDB 1.06 File::Basename (ships with Perl) 2.76 File::Find::Rule RCLAMP 0.30 Getopt::Std (ships with Perl) File::Temp (ships with Perl) TJENNESS 0.20 Quota TOMZO 1.6.2 Win32::OLE (ships with ActiveState Perl) JDB 0.1709 Win32::DriveInfo MBLAZ 0.06 Filesys::Df IGUTHRIE 0.92 Filesys::DfPortable IGUTHRIE 0.85 References for More Information For good information on platform differences for Perl programmers, the perlport man- ual page is invaluable. Active Directory Cookbook, Second Edition, by Robbie Allen and Laura Hunter, and Windows Server Cookbook, by Robbie Allen (both from O’Reilly) are excellent collec- tions of examples on how to script many of the important Windows-based operating system areas, including filesystem-related items. Allen has a website (http://techtasks .com) that serves as a repository for the code samples in all of the books he has authored or coauthored (and one or two others); on this site you can view all of the examples, which are in various languages (including Perl translations of all of the VBScript code), and you can buy the books and the individual code repositories. It truly is the mother lode of examples—one of the single most helpful websites you’ll ever find for this sort of programming. I highly recommend supporting this worthy effort by purchasing the code (and the books!). 48 | Chapter 2: Filesystems CHAPTER 3 User Accounts Here’s a short pop quiz. If it weren’t for users, system administration would be: 1. More pleasant 2. Nonexistent Despite the comments you may hear from system administrators on their most beleag- uered days, 2 is the best answer to this question. As I mentioned in the first chapter, ultimately system administration is about making it possible for people to use the available technology. Why all the grumbling, then? Users introduce two things that make the systems and networks we administer significantly more complex: nondeterminism and individual- ity. We’ll address the nondeterminism issues when we discuss user activity in the next chapter; for now, let’s focus on individuality. In most cases, users want to retain their own separate identities. Not only do they want unique names, but they want unique “stuff” too. They want to be able to say, “These are my files. I keep them in my directories. I print them with my print quota. I make them available from my web page.” Modern operating systems keep an account of all of these details for each user. But who keeps track of all of the accounts on a system or network of systems? Who is ultimately responsible for creating, protecting, and disposing with these little shells for individuals? I’d hazard a guess and say “You, dear reader”—or if not you personally, the tools you’ll build to act as your proxy. This chapter is designed to help you with that responsibility. Let’s begin our discussion of users by addressing some of the pieces of information that form a user’s identity and how that information is stored on a system. We’ll start by looking at Unix and Unix-variant users, and then we’ll address the same issues for Windows-based operating system users. Once we’ve covered identity information for both types of operating system, we’ll construct a basic account system. 49 Unix User Identities When exploring this topic, we’ll have to putter around in a few key files that store the persistent definition of a user’s identity. By “persistent definition,” I mean those attributes of a user that exist during the user’s entire lifespan, persisting even when he is not actively using a computer. Another word that we’ll use for this persistent identity is account. If you have an account on a system, you can log in and become one of that system’s users. Users come into being on a system when their information is first added to the password file (or the directory service that holds the same information). They leave the scene when this entry is removed. Let’s dive right in and look at how such a user identity is stored. The Classic Unix Password File Let’s start off with the classic password file format and then get more sophisticated from there. I call this format “classic” because it is the parent of all of the other Unix password file formats currently in use. It is still in use today in many Unix variants, including Solaris, AIX, and Linux. Usually found on the system as /etc/passwd, this file consists of lines of ASCII text, each line representing a different account on the system or a link to another directory service. A line in this file is composed of several colon- separated fields. We’ll take a close look at all of these fields as soon as we see how to retrieve them. Here’s an example line from /etc/passwd: dnb:fMP.olmno4jGA6:6700:520:David N. Blank-Edelman:/home/dnb:/bin/zsh There are at least two ways to go about accessing this information from Perl: • We can access it “by hand,” treating this file like any random text file and parsing it accordingly: my $passwd = '/etc/passwd'; open my $PW, '<', $passwd or die "Can't open $passwd:$!\n"; my ( $name, $passwd, $uid, $gid, $gcos, $dir, $shell ); while ( chomp( $_ = <$PW> ) ) { ( $name, $passwd, $uid, $gid, $gcos, $dir, $shell ) = split(/:/); ; } close $PW; • We can “let the system do it,” in which case Perl makes available some of the Unix system library calls that parse this file for us. For instance, another way to write that last code snippet is: 50 | Chapter 3: User Accounts my ( $name, $passwd, $uid, $gid, $quota, $comment, $gcos, $dir, $shell, $expire ); while ( ( $name, $passwd, $uid, $gid, $quota, $comment, $gcos, $dir, $shell, $expire ) = getpwent() ) { ; } endpwent(); Using these calls offers the tremendous advantage of automatically tying in to any OS- level name service being used, such as Network Information Service (NIS), the Light- weight Directory Access Protocol (LDAP), Kerberos, or NIS+. We’ll see more of these library call functions in a moment (including an easier way to use getpwent()), but for now let’s look at the fields our code returns: Name The login name field holds the short (usually eight characters or less), unique nom de la machine for each account on the system. The Perl function getpwent(), which we saw earlier being used in a list context, will return the name field if we call it in a scalar context: $name = getpwent( ); User ID On Unix systems, the user ID (UID) is actually more important than the login name for most things. Each file on a system is owned by a UID, not a login name. If we change the login name associated with UID 2397 in /etc/passwd from danielr to drinehart, danielr’s files instantly show up as being owned by drinehart instead. The UID is the persistent part of a user’s identity internal to the operating system. The Unix kernel and filesystems keep track of UIDs, not login names, for ownership and resource allocation, meaning that as far as the OS is concerned, multiple ac- counts with different login names but the same UID are actually the same account. A login name can be considered to be the part of a user’s identity that is external to the core OS; it exists to make things easier for humans. Here’s some simple code to find the next available unique UID in a password file. This code looks for the highest UID in use and produces the next number: my $passwd = '/etc/passwd'; open my $PW, '<', $passwd or die "Can't open $passwd:$!\n"; my @fields; my $highestuid; while ( chomp( $_ = <$PW> ) ) { @fields = split(/:/); $highestuid = ( $highestuid < $fields[2] ) ? $fields[2] : $highestuid; } Unix User Identities | 51 close $PW; print 'The next available UID is ' . ++$highestuid . "\n"; This example is a little too simple for real-world use, because op- erating systems often come with preassigned high-numbered ac- counts (e.g., nobody, nfsnobody), and the UID has an upper limit. Also, many institutions also have policies about how their UIDs are assigned (certain classes of users are assigned UIDs from a pre- determined range, and so on). All of these things have to be taken into account when writing code like this. Table 3-1 lists other useful name- and UID-related Perl functions and variables. Table 3-1. Login name- and UID-related variables and functions Function/variable Use getpwnam($name) In a scalar context, returns the UID for the specified login name; in a list context, returns all of the fields of a password entry getpwuid($uid) In a scalar context, returns the login name for the specified UID; in a list context, returns all of the fields of a password entry $> Holds the effective UID of the currently running Perl program $< Holds the real UID of the currently running Perl program Primary group ID On multiuser systems, users often want to share files and other resources with a select set of other users. Unix provides a user grouping mechanism to assist in this process. An account on a Unix system can be part of several groups, but it must be assigned to one primary group. The primary group ID (GID) field in the pass- word file lists the primary group for that account. Group names, GIDs, and group members are usually stored in the /etc/group file. This file holds a listing of secondary groups. To make an account part of several secondary groups, you just list that account in several places in the file (bearing in mind that some OSs have a hard limit on the number of groups an account can join—eight used to be a common restriction). Here are a couple of lines from an /etc/group file: bin::2:root,bin,daemon sys::3:root,bin,sys,adm The first field is the group name, the second is the password (some systems allow people to join a group by entering a password), the third is the GID of the group, and the last field is a list of the users in this group. 52 | Chapter 3: User Accounts Schemes for group ID assignment are site-specific, because each site has its own particular administrative and project boundaries. Groups can be created to model certain populations (students, salespeople, etc.), roles (backup operators, network administrators, etc.), or account purposes (backup accounts, batch processing accounts, etc.). Dealing with group files via Perl is a very similar process to the passwd parsing we did earlier. We can either treat a group file as a standard text file or use special Perl functions to perform the tasks. Table 3-2 lists the group-related Perl functions and variables. Table 3-2. Group name- and GID-related variables and functions Function/variable Use getgrent() In a scalar context, returns the group name; in a list context, returns the fields $name, $passwd, $gid, and $members getgrnam($name) In a scalar context, returns the group ID; in a list context, returns the same fields mentioned for getgrent()a getgrgid($gid) In a scalar context, returns the group name; in a list context, returns the same fields mentioned for getgrent() $) Holds the effective GID of the currently running Perl program $( Holds the real GID of the currently running Perl program a If you list the members of a group using multiple lines in /etc/group (e.g., if there are too many members to fit on one line), getgrgid() and getgrnam() may return only first line’s information when called in a list context. In that case, you will need to manually construct the list of members using repeated getgrent() calls. The “encrypted” password So far we’ve seen three key parts of how a user’s identity is stored on a Unix ma- chine. The next field is not part of this identity, but it is used to verify that someone should be allowed to assume all of the rights, responsibilities, and privileges be- stowed upon a particular user ID. This is how the computer knows that someone presenting himself as mguerre, for example, should be allowed to assume the UID associated with that username. Other, better forms of authentication now exist (e.g., public key cryptography), but this one has been inherited from the early days of Unix. It is common to see a line in a password file with just an asterisk (*) for the pass- word. Since the standard encryption algorithms won’t generate an asterisk as part of the encrypted password, placing that character in the password field (by editing the password file) will effectively lock the account. This convention is usually used when an administrator wants to disable the user from logging into an account without removing the account altogether. Simply adding an asterisk to the en- crypted password string will also lock the account, while making it easy to restore access without needing to know the password. Unix User Identities | 53 You may also see *LK* used to lock an account, and *NP* or NP used to indicate that there is no password in that file (although there might be one elsewhere, such as in /etc/shadow; we’ll deal with that in a moment). Dealing with user passwords is a topic unto itself. Chapter 11 of this book addresses this topic. GCOS/GECOS field The GCOS/GECOS* field is the least important field (from the computer’s point of view). This field usually contains the full name of the user (e.g., “Roy G. Biv”). Often, people put their titles and/or phone extensions in this field as well. System administrators who are concerned about privacy issues on behalf of their users (as all should be) need to watch the contents of this field. It is a standard source for account-name-to-real-name mappings. On most Unix systems, this field is available as part of a world-readable /etc/passwd file or directory service, and hence the information is available to everyone on the system. Many Unix programs, such as mail clients, also consult this field when they attach a user’s login name to some piece of information. If you have any need to withhold a user’s real name from other people (e.g., if that user is a political dissident, a federal witness, or a famous person), this is one of the places you must monitor. As a side note, if you maintain a site with a less mature or professional user base, it is often a good idea to disable mechanisms that allow users to change their GCOS field values to any random string (for the same reasons that user-selected login names can be problematic). You may not want your password file to contain expletives or other unprofessional information. Home directory The next field contains the name of the user’s home directory. This is the directory where the user begins his time on the system. This is also usually where the files that configure that user’s environment live. It is important for security purposes that an account’s home directory be owned and writable by that account only. World-writable home directories allow trivial account hacking. There are cases, however, where even a user-writable home di- rectory is problematic. For example, in restricted shell scenarios (where accounts can only log in to perform specific tasks and do not have permission to change anything on the system), a user-writable home directory is a big no-no because it could let an outsider modify the restrictions. Here’s some Perl code to make sure that every user’s home directory is owned by that user and is not world-writable: * For some amusing details on the origin of the name of this field, see the GCOS entry in the Jargon Dictionary ( 54 | Chapter 3: User Accounts use User::pwent; use File::stat; # note: this code will beat heavily upon any machine using # automounted homedirs while ( my $pwent = getpwent() ) { # make sure we stat the actual dir, even through layers of symlink # indirection my $dirinfo = stat( $pwent->dir . '/.' ); unless ( defined $dirinfo ) { warn 'Unable to stat ' . $pwent->dir . ": $!\n"; next; } warn $pwent->name . ''s homedir is not owned by the correct uid (' . $dirinfo->uid . ' instead ' . $pwent->uid . ")!\n" if ( $dirinfo->uid != $pwent->uid ); # world writable is fine if dir is set "sticky" (i.e., 01000); # see the manual page for chmod for more information warn $pwent->name . "'s homedir is world-writable!\n" if ( $dirinfo->mode & 022 and ( !$dirinfo->mode & 01000 ) ); } endpwent(); This code looks a bit different from our previous parsing code because it uses two magic modules by Tom Christiansen: User::pwent and File::stat. These modules override the normal getpwent() and stat() functions, causing them to return something different from the values mentioned earlier: when User::pwent and File::stat are loaded, these functions return objects instead of lists or scalars. Each object has a method named after a field that normally would be returned in a list context. So, code like this that queries the metadata for a file to retrieve its group ID: $gid = (stat('filename'))[5]; can be written more legibly as: use File::stat; my $stat = stat('filename'); my $gid = $stat->gid; or even: use File::stat; my $gid = stat('filename')->gid; User shell The final field in the classic password file format is the user shell field. This field usually contains one of a set of standard interactive programs (e.g., sh, csh, tcsh, ksh, zsh), but it can actually be set to the full path of any executable program Unix User Identities | 55 or script. This field is often set to a noninteractive program (e.g., /bin/false or /sbin/nologin) in order to prevent logins to daemon or locked accounts. From time to time, people have joked (half-seriously) about setting their shells to be the Perl interpreter. Some have also contemplated embedding a Perl interpreter in the zsh shell (and possibly others), though this has yet to happen. However, some serious work has been done to create a Perl shell (see http://www.focusre and and to embed Perl into Emacs, an editor that could easily pass for an operating system ( Perl has also been embedded in most of the recent vi editor implementations (nvi, vile, and Vim). On occasion, you might have reason to list nonstandard interactive programs in this field. For instance, if you wanted to create a menu-driven account, you could place the menu program’s name here. In these cases, you have to take care to prevent someone using the account from reaching a real shell and wreaking havoc. A common mistake is including a mail program that allows the user to launch an editor or pager for composing and reading mail, as that editor or pager could have a shell-escape function built in. Caution when using nonstandard interactive programs is warran- ted in all circumstances. For example, if you allow people to ssh in and you try to lock their accounts using such a program, be sure your SSH server isn’t configured to pay attention to their .ssh/ environment files (off by default in OpenSSH). If that file is enabled, the user can play some really fun tricks by setting LD_PRELOAD. A list of standard, acceptable shells on a system is often kept in /etc/shells. Most FTP daemons will not allow a normal user to connect to a machine if her shell in /etc/passwd (or the networked password file) is not on that list. On some systems, the chsh program also checks that file to validate any shell-changing requests from users. Here’s some Perl code to report accounts that do not have approved shells: use User::pwent; my $shells = '/etc/shells'; open my $SHELLS, '<', $shells or die "Unable to open $shells:$!\n"; my %okshell; while (<$SHELLS>) { chomp; $okshell{$_}++; } close $SHELLS; while ( my $pwent = getpwent() ) { warn $pwent->name . ' has a bad shell (' . $pwent->shell . ")!\n" 56 | Chapter 3: User Accounts unless ( exists $okshell{ $pwent->shell } ); } endpwent(); Changes to the Password File in BSD 4.4 Systems At the Berkeley Software Distribution (BSD) 4.3 to 4.4 upgrade point, the BSD variants added two twists to the classic password file format: additional fields were inserted between the GID and GCOS fields, and a binary database format was introduced to store account information. Extra fields in passwd files The first field BSD 4.4 systems added was the class field, which allows a system ad- ministrator to partition the accounts on a system into separate classes (e.g., different login classes might be given different resource limits, such as CPU time restrictions). BSD variants also add change and expire fields to hold an indication of when a password must be changed and when the account will expire. We’ll see fields like these when we get to the next Unix password file format as well. Perl also supports a few other fields (specific to certain operating systems) that can be found in password files. Some operating systems provide the ability to include addi- tional information about a user, including that user’s disk quota and a free-form com- ment. When compiled under an operating system that supports these extra fields, Perl includes the contents of the extra fields in the return values of functions like getpwent(). This is one good reason to use getpwent() in your programs instead of split()ing the password file entries by hand. The binary database format The second twist BSD 4.4 added to the password mechanisms was a database format, rather than plain text, for primary storage of password file information. BSD machines keep their password file information in DB format, a greatly updated version of the older Unix Database Management (DBM) libraries. This change allows the systems to quickly look up password information. The program pwd_mkdb takes the name of a password text file as its argument, creates and moves into place two database files, and then moves the text file into /etc/master.passwd. The two databases provide a shadow password scheme, dif- fering in their read permissions and encrypted password field contents. We’ll talk more about this in the next section. Perl has the ability to work with DB files directly (we’ll work with this format later, in Chapter 7), but in general I would not recommend editing the databases while the system is in use. The issue here is one of locking: it’s very important not to change a crucial database like the one storing your passwords without making sure that other Unix User Identities | 57 programs are not similarly trying to write to or read from it. Standard operating system programs like chpasswd and vipw handle this locking for you.† The sleight-of- hand approach we saw for quotas in Chapter 2, which used the EDITOR variable, can often be used with these utilities as well. Shadow Passwords Earlier I emphasized the importance of protecting the contents of the GCOS field, since this information is publicly available through a number of different mechanisms. An- other fairly public, yet rather sensitive piece of information is the list of encrypted passwords for all of the users on the system. Even though the password information is cryptologically hidden, exposing it in a world-readable file creates a significant vulner- ability, thanks to the powerful password crackers available today.‡ Parts of the pass- word file need to be world-readable (e.g., the UID and login name mappings), but not all of it. There’s no need to provide a list of encrypted passwords to users who may be tempted to run password-cracking programs. One very common alternative to leaving encrypted passwords exposed is to banish the encrypted password string for each user to a special file that is only readable by root. This second file is known as a “shadow password” file, since it contains lines that shadow the entries in the real password file. This mechanism is standard on most modern OS distributions. With this approach, the original password file is left intact, with one small change: the encrypted password field contains a special character or characters to indicate that password shadowing is in effect. Placing an x in this field is common, though the insecure copy of the BSD database uses an asterisk (*). I’ve heard of some shadow password suites that insert a special, normal- looking string of characters in this field. If your password file goes awanderin’, this provides a lovely time for the recipient, who will attempt to crack a password file of random strings that bear no relation to the real passwords. Most operating systems take advantage of this second shadow password file to store more information about the accounts. This additional information resembles that in the surplus fields we saw in the BSD files (e.g., account expiration data and information on password changing and aging). † pwd_mkdb may or may not perform this locking for you (depending on the BSD flavor and version), so caveat implementor. ‡ Not to mention the highly effective technique of using rainbow tables ( _table). 58 | Chapter 3: User Accounts In most cases Perl’s normal password functions, such as getpwent(), can handle shadow password files. As long as the C libraries shipped with the OS do the right thing, so will Perl. Here’s what “do the right thing” means: when your Perl script is run with the appropriate privileges (i.e., as root), these routines will return the encrypted password. Under all other conditions, that password will not be accessible to those routines. Unfortunately, Perl may not retrieve the additional fields found in the shadow file. Eric Estabrooks has written Passwd::Solaris and Passwd::Linux modules that can help, but only if you are running one of those operating systems. If these fields are important to you, or you want to play it safe, the sad truth (in conflict with my earlier recommen- dation to use getpwent()) is that it is often simpler to open the shadow file by hand and parse it manually. Windows-Based Operating System User Identities Now that we’ve explored the pieces of information that Unix systems cobble together to form a user’s identity, let’s take a look at the same topic for Windows users. Much of this info is conceptually similar, so we’ll dwell mostly on the differences between the two operating systems. One important note before we continue: Windows systems by default store user iden- tity information in one of two places: locally (on the machine itself, in a way not shared with other machines) or domain-wide (where it most likely lives in Active Directory on a domain controller). In the latter case, this information is shared with the user’s local machine and stored on that machine for at least the duration of the user’s session. As in our discussion of Unix user identities, we’ll focus here on local accounts. For more on how to work with Active Directory or other directory services, see Chapter 9. Windows User Identity Storage and Access Windows stores the persistent identity information for a user in a database called the SAM (Security Accounts Manager), or directory, database. The SAM database is part of the Windows registry, located at %SYSTEMROOT%\system32\config. The files that make up the registry are all stored in a binary format, meaning normal Perl text- manipulation idioms cannot be used to read from or write changes to this database. It is theoretically possible to use Perl’s binary data operators (i.e., pack() and unpack()) with the SAM database, provided you do so when Windows is not running, but this way lie madness and misery. Luckily, there are better ways to access and manipulate this information via Perl. One approach is to call an external binary to interact with the OS for you. Every Win- dows machine has a feature-bloated command called net that you can use to add, delete, and view users. The net command is quirky and limited, though, and is generally the method of last resort. Windows-Based Operating System User Identities | 59 For example, here’s the net command in action on a machine with two accounts: C:\> net users User accounts for \\HOTDIGGITYDOG ---------------------------------- Administrator Guest The command completed successfully. The output of this program could easily be parsed from Perl. There are also commercial packages that offer command-line executables to perform similar tasks. Darn That Bitrot Here’s a sad tale of bitrot that has taken place since the first edition of this book was published. In the first edition, I recommended using several third-party modules for performing user administration tasks on Windows systems: Win32::UserAdmin (as de- scribed in the O’Reilly book Windows NT User Administration, with code distributed from the O’Reilly site), David Roth’s Win32::AdminMisc and Win32::Perms (distributed from, or Jens Helberg’s Win32::Lanman (hidden away in his CPAN directory at BERG/). As far as I can tell, no one has touched Win32::UserAdmin in quite some time. David Roth left Perl behind when he went off to work for Microsoft back in 1999. When I spoke to David in 2005 he indicated that he was happy to continue to make the work he had done available, but that he did not have any further time to maintain and update his modules. He had hoped someone else would take on their maintenance, but that hasn’t happened as of this writing. Similarly, Jens Helberg hasn’t really been active in the Perl world since at least 2003. It’s a pity these modules have fallen into disrepair, because they were some of the handiest Windows modules available. I can’t recommend using Win32::Lanman or Win32::AdminMisc/Win32::Perms at this point because their maintenance is so dicey, but if you still want to get a copy that can be loaded using ppm in the ActiveState distribution, there was a version of Win32::AdminMisc available for 5.10 as of this writing at http:// (Roth’s site has a 5.8 version available of both Win32::AdminMisc and Win32::Perms) and a version of Win32::Lanman for Perl 5.8 in the repository described at Instead you’ll find the text in this edition almost exclusively sticks to modules like Win32API::Net that are part of the official libwin32 set of modules shepherded by Jan Dubois and plus a few other Windows modules with their own active maintainers. Another approach is to use the Perl module Win32API::Net, which is bundled with the ActiveState Perl distribution. Here’s some example code that shows the users on the local machine and some details about them. It prints out lines that look similar to the contents of /etc/passwd under Unix: 60 | Chapter 3: User Accounts use Win32API::Net qw(:User); UserEnum( '', \my @users ); foreach my $user (@users) { # '3' in the following call refers to the "User info level", # basically a switch for how much info we want back. Here we # ask for one of the more verbose user info levels (3). UserGetInfo( '', $user, 3, \my %userinfo ); print join( ':', $user, '*', $userinfo{userId}, $userinfo{primaryGroupId}, '', $userinfo{comment}, $userinfo{fullName}, $userinfo{homeDir}, '' ),"\n"; } Finally, you can use the Win32::OLE module to access the Active Directory Service Interfaces (ADSI) functionality built into Windows. We’ll go into this topic in great detail in Chapter 9, so I won’t present an example here. We’ll look at more Perl code to access and manipulate Windows users later, but for the time being let’s return to our exploration of the differences between Unix and Windows users. Windows User ID Numbers User IDs in Windows are not created by mortals, and they cannot be reused. Unlike in Unix, where we can simply pick a UID number out of the air, the OS uniquely generates the identifier in Windows when a new user is created: a unique user identifier (which Windows calls a relative ID, or RID) is combined with machine and domain IDs to create a large ID number called a security identifier, or SID, which acts as the user’s UID. An example RID is 500. The RID is part of a longer SID that looks like this: S-1-5-21-2046255566-1111630368-2110791508-500 The RID is the number we get back as part of the UserGetInfo() call shown in the last code snippet. Here’s the code necessary to print the RID for a particular user: use Win32API::Net qw(:User); UserGetInfo( '', $user, 3, \my %userinfo ); print $userinfo{userId},"\n"; You can’t (by normal means) recreate a user after that user has been deleted. Even if you create a new user with the same name as the deleted user, the SID will not be the same, and the new user will not have access to the predecessor’s files and resources. Windows-Based Operating System User Identities | 61 This is why some Windows books recommend renaming accounts that are due to be inherited by another person. That is, if a new employee is supposed to receive all of the files and privileges of a departing employee, they suggest renaming the existing account to preserve the SID rather than creating a new account, transferring the files, and then deleting the old account. I personally find this method for account handoffs to be a little uncouth, because it means the new employee will inherit all of the corrupted and useless registry settings of his predecessor. However, it’s the most expedient method, and sometimes that is important. Part of the rationale for this recommendation comes from the pain associated with transferring ownership of files. In Unix, a privileged user can say, “Change the owner- ship of all of these files so that the new user owns them.” In Windows, however, there’s no giving of ownership; there’s only taking (i.e., an admin can say, “I own these files now”). Luckily, there are two ways to get around this restriction and pretend we’re using Unix’s semantics. From Perl, we can: • Call a separate binary, such as: — The chown binary from the Cygwin distribution found at http://www.cygnus .com (free). If you have a Unix background and work on Windows machines, you definitely should check out Cygwin. For a commercial version of Unix-like tools such as chown, check out the MKS Toolkit ( — The SubInACL binary, available for download from the Microsoft Download Center ( It has the plus of being provided by Microsoft, but it requires a small learning curve. — SetACL from is similar to SubInACL but has its own twists. If you are considering using SubInACL, be sure to check out this program as well because it may be more to your liking. • Use a Perl module such as: — Win32::Security by Toby Ovod-Everett. Here’s an example of using this module to change the owner of a single file: use Win32::Security::NamedObject; my $nobj = Win32::Security::NamedObject->new('FILE',$filename); $nobj->ownerTrustee($NewAccountName); Two asides about Win32::Security. First, it ships with a lovely set of utility scripts, including to show inherited and noninherited permis- sions and to fix permission issues such as broken inherited permis- sions from files that have been moved. Second, according to the author, as of this writing Win32::Security can have issues with objects that have both permit and deny ACLs (if they share the same trustees), so be sure to test carefully if you use deny ACLs. — Win32::OLE by Jan Dubois to call WMI functions (see Chapter 4 for an in-depth look at WMI). This is a little tricky because it is much easier to take ownership of a file (i.e., change a file to be owned by the user running the script) than it is 62 | Chapter 3: User Accounts to change the ownership of the file to an arbitrary user.* Taking ownership is performed via the TakeOwnership() method of the CIM_DataFile object in the cimv2 namespace. — Win32::Perms by Dave Roth, located at and documented at (Be sure to read the sidebar “Darn That Bitrot” on page 60 before depending on this module.) Here’s some sample code using this module that will change the owner of a directory and its contents, including subdirectories: use Win32::Perms; $my acl = new Win32::Perms( ); $acl->Owner($NewAccountName); my $result = $acl->SetRecurse($dir); $acl->Close( ); Windows Passwords Don’t Play Nice with Unix Passwords The algorithms used to obscure the passwords that protect access to a user’s identity in Windows and Unix are cryptologically incompatible. Once it’s been encrypted, you cannot transfer password information between these two operating system families and expect to use it for password changes or account creations, as you can when transferring encrypted passwords between different operating systems (Linux, Solaris, Irix, etc.) in the Unix family. As a result, two separate sets of passwords have to be used and/or kept in sync. This difference is the bane of every system administrator who has to administer a mixed Unix/Windows environment. Some administrators get around this by using custom authentication modules, commercial or otherwise. As a Perl programmer, the only thing you can do if you are not using custom authen- tication mechanisms is to create a system whereby users provide their passwords in plain text. The plain-text passwords are then used to perform two separate password- related operations (changes, etc.), one for each OS. Windows Groups So far, I’ve been able to gloss over any distinction between storage of a user’s identity on a local machine and storage in some network service, like NIS. For the information we’ve encountered, it hasn’t really mattered if that information was used on a single system or all of the systems in a network or workgroup. But in order to talk cogently about Windows user groups and their intersection with Perl, we unfortunately have to move beyond this simplified view. * I hunted and hunted for an example of changing ownership (versus taking it) via WMI and could not find one in any language. I don’t want to claim that it is impossible, but color me dubious. Windows-Based Operating System User Identities | 63 On Windows systems, a user’s identity can be stored in one of two places: in the SAM database on a specific machine or in the Active Directory (AD) store on a domain controller. This is the distinction between a local user, who can only log into a single machine, and a domain user, who can log in to any of the permitted machines that participate in a domain as part of an AD instance. Often, users have information stored in both places. For example, this would allow a user to log in from any Windows machine in the domain and access his desktop environment as stored on a fileserver, or log into his own PC without referring to network resources for authentication or file sharing. There are different kinds of groups in the Windows model. To understand the differ- ence between them, we have to consider two things: where a group can be used (its scope) and what a group can contain (its members). The following list starts with the smallest “jurisdiction” and works outward: Local groups Can be used only on the local machine to control access to resources on that machine. Can contain local accounts, domain accounts, and global groups. Domain local groups Can be used to control access to resources in a domain (e.g., a shared printer). Can contain accounts, global groups, universal groups from any domain, and other domain local groups (from the same domain). Global groups Often used as container groups included in other groups in any domain. Can con- tain accounts and other global groups from the same domain where the global group is defined. Universal groups Can be used across domains and forests (i.e., sets of directory trees) in the same AD instance to hold other groups. Can contain accounts, global groups, and uni- versal groups from the same forest where the universal group is defined Local groups are machine-specific. People seldom add or remove local groups; they mostly just change the membership of the default groups. Given this, let’s look instead at how the other kinds of groups get used. The key to this story is the use of groups nested in other groups. Suppose you want to control access to some resource (the classic example is a shared printer) that a number of people will share. Instead of listing each person in some access list associated with the printer, it is far more convenient to say “anyone in a particular domain local group” can print to the printer. The domain local group is assigned the access to the printer. You could just start adding users to that domain local group, and everyone you added would happily be able to print, but that approach would start to get old once you began to accumulate all the domain local groups that parts of your organization get access to as part of their job functions. If every time someone is hired into the facilities planning 64 | Chapter 3: User Accounts department you have to add them into three printer access groups, the plotter access group, plus a few others, it becomes unpleasant for you as the administrator. In addition to all of the manual labor necessary for granting access, the chance for error is pretty good. One bad way to solve this problem would be to grant rights for each resource to the group that contains the accounts for the facilities planning department itself. That idea breaks down as soon as multiple groups need overlapping access. Let’s say the facilities planning department runs out of room on its floor and needs to use some desks on another floor. The people who are moved to the new floor will need to share a printer on that floor. If the right to print to that printer is granted to each department on that floor separately, it becomes a pain to determine which departments have access (since you have to look at each department’s group in turn). The right way to fix this is to nest global groups (like the group that holds the accounts in the facilities planning department) in the domain local groups that control access to each printer. When this is done, the users in each global group are automatically given the printing rights they deserve. Dealing with situations where two departments have to share a resource is easy; you just put both global groups into the appropriate domain local group. If you need to know which groups have access to a printer, you can look at the membership of the domain local group that controls access to the printer. Fig- ure 3-1 shows this nested group idea in a pictorial form. Domain local group: Printer Access (members of group can access printer) Global group: Facilities Planning Global group: Customer Service Figure 3-1. How Windows groups nest The term “global” is a bit of a misnomer because it sounds like you should be able to insert accounts from any domain in your AD tree, but in fact global groups can only hold accounts and other groups from within the domain in which they were created. This is where universal groups fit in. Universal groups let you aggregate global groups from different domains. If you want to have a single group for all of your accounts in Windows-Based Operating System User Identities | 65 different domains, you can construct a universal group that nests the right global groups from each domain. You can subsequently nest this universal group in some other permission-granting group, and all of your users will inherit that permission. This scheme would be even handier if it didn’t complicate our Perl programming. We potentially have to, or at least may want to, use different modules or different functions based on group type. Here are the choices you have: 1. If you are working with universal groups, you have no choice but to use Win32::OLE to perform ADSI calls. 2. If you are working with local, domain local, or global groups, you can use ADSI (via either the WinNT or LDAP providers, depending on the group), or you may find it easier to use a module like the one we saw before: Win32API::Net. The ad- vantage of using ADSI via Win32::OLE is consistency (you are using it for all group operations); the advantage of using Win32API::Net is that it is considerably easier (it has built-in functions for the tasks). Let’s take a quick look at each approach. If we stick to using Win32API::Net, we are immediately faced with a choice of group type: local or global? Win32API::Net has different functions for each kind of group, as listed in Table 3-3. Table 3-3. Win32API::Net functions for local and global groups Local functions Global functions LocalGroupAdd() GroupAdd() LocalGroupDel() GroupDel() LocalGroupAddMembers() GroupAddUser() LocalGroupDelMembers() GroupDelUser() LocalGroupGetMembers() GroupGetUsers() LocalGroupGetInfo() GroupGetInfo() LocalGroupSetInfo() GroupSetInfo() LocalGroupEnum() GroupEnum() The functions in the first column let you set local groups (both local to the machine and to the domain), while those in the second work strictly with global groups. The first argument to all of these functions determines where the change is made. For ex- ample, to create a group local to the machine, the first argument can be empty (''). To create a domain local group or a global group, the first argument should be the name of an appropriate domain controller. To find the appropriate domain controller, you can call GetDCName(): # $server is the server whose DC you need to find, # $domainname is the domain you need the DC for, # the answer gets placed into $dcname GetDCName($server, $domainname, $dcname); 66 | Chapter 3: User Accounts This duality means that your code may have to call two functions for the same opera- tion. For example, if you need to obtain all of the groups a user may be in, you may have to call two functions, one for local groups and the other for global groups. The group functions in Table 3-3 are pretty self-explanatory. Here’s a quick example of adding a user to a global group: use Win32API::Net qw(:Get :Group); my $domain = 'my-domain'; # Win32::FormatMessage converts the numeric error code to something # we can read GetDCName('' , $domain , my $dc) or die Win32::FormatMessage( Win32::GetLastError() ); GroupAddUser($dc,'Domain Admins','dnbe') or die Win32::FormatMessage( Win32::GetLastError() ); Here’s a quick tip found in Roth’s books (listed in the references section at the end of the chapter): your program must run with administrative privileges to access the list of local groups, but global group names are available to all users. If we wanted to create a universal group using ADSI, we could use code like this (see Chapter 9 for a description of just what is going on here): use Win32::OLE; $Win32::OLE::Warn = 3; # throw verbose errors # from ADS_GROUP_TYPE_ENUM in the Microsoft ADSI Doc my %ADSCONSTANTS = ( ADS_GROUP_TYPE_GLOBAL_GROUP => 0x00000002, ADS_GROUP_TYPE_DOMAIN_LOCAL_GROUP => 0x00000004, ADS_GROUP_TYPE_LOCAL_GROUP => 0x00000004, ADS_GROUP_TYPE_UNIVERSAL_GROUP => 0x00000008, ADS_GROUP_TYPE_SECURITY_ENABLED => 0x80000000 ); my $groupname = 'testgroup'; my $descript = 'Test Group'; my $group_OU = 'ou=groups,dc=windows,dc=example,dc=edu'; my $objOU = Win32::OLE->GetObject( 'LDAP://' . $group_OU ); my $objGroup = $objOU->Create( 'group', "cn=$groupname" ); $objGroup->Put( 'samAccountName', $groupname ); $objGroup->Put( 'groupType', $ADSCONSTANTS{ADS_GROUP_TYPE_UNIVERSAL_GROUP} | $ADSCONSTANTS{ADS_GROUP_TYPE_SECURITY_ENABLED} ); $objGroup->Put( 'description', $descript ); $objGroup->SetInfo; Windows-Based Operating System User Identities | 67 Windows User Rights The last difference between Unix and Windows user identities that we’ll address is the concept of a “user right.” In the traditional Unix rights schema, the actions a user can take are constrained either by file permissions or by the superuser/nonsuperuser dis- tinction. Under Windows, the permission scheme is better explained with a superhero analogy: users (and groups) can be imbued with special powers that become part of their identities.† For instance, one can attach the user right Change the System Time to an ordinary user, allowing that user to set the system clock on the local machine. Some people find the user rights concept confusing because they have attempted to use the Local Security Policy Editor or Group Policy/Group Policy Object Editor. The list of policies shown when you navigate to User Rights Assignment (Figure 3-2) presents the information in exactly the opposite manner that most people expect to see it: it shows a list of the possible user rights and expects you to add groups or users to a list of entities that already have this right. Figure 3-2. Assigning user rights via the Local Security Policy Editor A more user-centric UI would offer a way to add user rights to or remove them from users, instead of the other way around.‡ This is how we will operate on rights using Perl. † Most modern Unix systems can use access control lists and role-based access control (RBAC) to manage user rights in similar detail, but this is not commonly done, as it is under Windows. ‡ To their credit, the new interface for this sort of thing, the Group Policy Management Console, does improve on the situation by making the object that is receiving a policy setting paramount. It also offers the ability to script many meta-GPO operations. Unfortunately, the one thing you can’t do out of the box (as of this writing) is directly twiddle the settings in a GPO from a script. Sigh. 68 | Chapter 3: User Accounts One approach is to call the program ntrights.exe from the Microsoft 2000/2003 Resource Kit. The resource kit tools for 2003, including ntrights.exe, are (as of this writing) available for download for free from Microsoft. If you haven’t heard of the resource kits, be sure to read the sidebar about them. The Microsoft Windows Resource Kits “You must have the Windows {Anything} Resource Kit” is the general consensus among serious Windows administrators and the media that covers this field. Microsoft Press usually publishes at least one large tome for each OS version, full of nitty-gritty operational information. It is not this information that makes these books so desirable, though; rather, it is the CD-ROMs or sometimes the direct downloads associated with the books that make them worth their weight in zlotniks. These add-ons contain a grab bag of crucial utilities for Windows administration. Many of the utilities were contributed by the OS development groups, who wrote their own code because they couldn’t find the tools they needed anywhere else. For example, there are utilities that add users, change filesystem security information, show installed printer drivers, work with roaming profiles, help with debugging domain and network browser services, and so on. The tools in the resource kits are provided “as is,” meaning there is virtually no support available for them. This no-support policy may sound harsh, but it serves the important purpose of allowing Microsoft to put a variety of useful code in the hands of adminis- trators without having to pay prohibitive support costs. The utilities in the resource kits have a few small bugs, but on the whole they work great. Updates that fix bugs in some of these utilities have been posted to Microsoft’s website. Using ntrights.exe is straightforward; just call the program from Perl like you would any other (i.e., using backticks or the system() function). In this case, we’ll call ntrights.exe with a command line of the form: C:\> ntrights.exe +r +u [-m \\machinename] to give a right to a user or group (on an optionally specified machine named machinename). To take that right away, we use a command line of the form: C:\> ntrights.exe -r +u [-m \\machinename] Unix users will be familiar with the use of the + and - characters (as in chmod), in this case used with the r switch, to give and take away privileges. The list of names of rights that can be assigned (for example, SetSystemtimePrivilege to set the system time) can be found in the Microsoft resource kit documentation for the ntrights command. If for some reason you don’t want to use the resource kit tools, the Cygwin distribution that was touted earlier also provides an editrights utility package that can do similar things. A second, Perl module–based approach entails using the Win32::Lanman module by Jens Helberg, which can be found in PPM form at Windows-Based Operating System User Identities | 69 or in source form in Helberg’s CPAN directory at thors/id/J/JH/JHELBERG/ (this won’t be found in a standard CPAN search, so you have to go to that directory directly). So, after all of the angst in the bitrot sidebar about modules like Win32::Lanman, why am I still describing how to use it here? I’ve searched high and low, and there does not appear to be (as of this writing) a reasonable substitute for this module in this context. To the best of my knowledge, you can’t do these sorts of things via WMI or ADSI. I’d love to be proven wrong! I believe it would be possible to recreate all of the Win32::Lanman calls via Win32::API, since they can both access the same underlying Win32 API, but this is beyond the depth of knowledge in Windows program- ming that I’ve ever aspired to acquire. If you do rewrite the Win32::Lanman functions listed in this section to use Win32::API (and plan to maintain/support your code), I’ll be delighted to switch to your module and write about it in future editions of this book. Let’s start by looking at the process of retrieving an account’s user rights. This is a multiple-step process. First, we need to load the module: use Win32::Lanman; my $server = 'servername'; Then, we need to retrieve the actual SID for the account we wish to query or modify. In the following sample, we’ll get the Guest account’s SID: Win32::Lanman::LsaLookupNames( $server, ['Guest'], \my @info ) or die "Unable to lookup SID: " . Win32::Lanman::GetLastError() . "\n"; @info now contains an array of references to anonymous hashes: one element for each account we query (in this case, it is just a single element for Guest). Each hash contains the following keys: domain, domainsid, relativeid, sid, and use. We only care about sid for our next step. Now we can query the rights: Win32::Lanman::LsaEnumerateAccountRights( $server, ${ $info[0] }{sid}, \my @rights ) or die "Unable to query rights: " . Win32::Lanman::GetLastError() . "\n"; @rights now contains a set of names describing the rights apportioned to Guest. Figuring out the API name of a user right and what it represents is tricky. The easiest way to learn which names correspond to which rights and what each right offers is to look at the software development kit (SDK) documentation at .com. This documentation is easy to find because Helberg has kept the standard SDK function names for his Perl function names. To find the names of the available rights, search the Microsoft’s Developer Network site for “LsaEnumerateAccountRights”; you’ll find pointers to them quickly. 70 | Chapter 3: User Accounts This information also comes in handy for the modification of user rights. For instance, if we wanted to add a user right to allow our Guest account to shut down the system, we could use: use Win32::Lanman; my $server = 'servername'; Win32::Lanman::LsaLookupNames( $server, ['Guest'], \my @info ) or die "Unable to lookup SID: " . Win32::Lanman::GetLastError() . "\n"; Win32::Lanman::LsaAddAccountRights( $server, ${ $info[0] }{sid}, [&SE_SHUTDOWN_NAME] ) or die "Unable to change rights: " . Win32::Lanman::GetLastError() . "\n"; In this case, we found the SE_SHUTDOWN_NAME right in the SDK doc and used &SE_SHUTDOWN_NAME (a subroutine defined by Win32::Lanman), which returns the value for this SDK constant. Win32::Lanman::LsaRemoveAccountRights(), a function that takes similar arguments to those we used to add rights, is used to remove user rights. Before we move on to other topics, it is worth mentioning that Win32::Lanman also provides a function that works just like the Local Security Policy Editor’s broken in- terface, described earlier: instead of matching users to rights, we can match rights to users. If we use Win32::Lanman::LsaEnumerateAccountsWithUserRight(), we can retrieve a list of SIDs that have a specific user right. Enumerating this list could be useful in certain situations. Building an Account System to Manage Users Now that we’ve had a good look at user identities, we can begin to address the admin- istration aspect of user accounts. Rather than just showing you the select Perl subrou- tines or function calls you need for adding and deleting users, I’ll take this topic to the next level by showing these operations in a larger context. In the remainder of this chapter, we’ll work toward writing a bare-bones* account system that starts to manage both Windows and Unix users. Our account system will be constructed in four parts: user interface, data storage, process scripts (Microsoft would call them the “business logic”), and low-level library routines. From a process perspective, they work together (see Figure 3-3). Requests come into the system through a user interface and get placed into an “add account queue” file for processing. We’ll just call this an “add queue” from here on. A process script reads this queue, performs the required account creations, and stores information about the created accounts in a separate database. That takes care of add- ing the users to our system. * Where “bare-bones” means “toy.” This is really meant to be very simple code just to demonstrate the underlying concepts behind the construction of a system like this. Building an Account System to Manage Users | 71 User Account Add/removeaccount queue User interface Process Script Low-levelLibrary Account database Figure 3-3. The structure of a basic account system For removing a user, the process is similar. A user interface is used to create a “remove queue.” A second process script reads this queue, deletes the indicated users from our system, and updates the central database. We isolate these operations into separate conceptual parts because it gives us the max- imum possible flexibility should we decide to change things later. For instance, if some day we decide to change our database backend, we only need to modify the database routines. Similarly, if we want our user addition process to include more steps (perhaps cross-checking against another database in human resources), we will only need to change the process script in question. Let’s start by looking at the first component: the user interface used to create the initial account queue. For the bare-bones purposes of this book, we’ll use a simple text-based user interface to query for account parameters: sub CollectInformation { use Term::Prompt; # we'll move these use statements later use Crypt::PasswdMD5; # list of fields init'd here for demo purposes - this should # really be kept in a central configuration file my @fields = qw{login fullname id type password}; my %record; foreach my $field (@fields) { 72 | Chapter 3: User Accounts # if it is a password field, encrypt it using a random salt before storing if ( $field eq 'password' ) { # believe it or not, we may regret the decision to store # the password in a hashed form like this; we'll talk about # this a little later on in this chapter $record{$field} = unix_md5_crypt( prompt( 'p', 'Please enter password:', '', '' ), undef ); } else { $record{$field} = prompt( 'x', "Please enter $field:", '', '' ); } } print "\n"; $record{status} = 'to_be_created'; $record{modified} = time(); return \%record; } This routine creates a list and populates it with the different fields of an account record. As the comment mentions, this list is inlined in the code only for brevity’s sake. Good software design suggests the field name list really should be read from an additional configuration file. Ideally, that config file would also provide better prompts and vali- dation information describing the kinds of input allowed for each field (rather than just making a distinction between password and nonpassword inputs, as we do here). Once the list has been created, the routine iterates through it and requests the value for each field. Each value is then stored back into the record hash. At the end of the question and answer session, a reference to this hash is returned for further processing. Our next step will be to write the information to the add queue. Before we get to that code, though, we should talk about data storage and data formats for our account system. The Backend Database The center of any account system is a database. Some administrators use the /etc/ passwd file or SAM database/AD store as the only record of the users on their system, but this practice is often shortsighted. In addition to the pieces of a user’s identity that we’ve already discussed, a separate database can be used to store metadata about each account, like its creation and expiration date, the account sponsor (if it’s a guest ac- count), the user’s phone numbers, etc. And once a database is in place, it can be used for more than just basic account management; it can be useful for all sorts of niceties, such as automatic mailing list creation. Building an Account System to Manage Users | 73 Why the Really Good System Administrators Create Account Systems System administrators fall into roughly two categories: mechanics and architects. Me- chanics spend most of their time in the trenches dealing with details. They have amazing amounts of arcane knowledge about the hardware and software they administer. If something breaks, they know just the command, file, or spanner wrench to wield. Talented mechanics can scare you with their ability to diagnose and fix problems even while standing halfway across the room from the problem machine. Architects spend their time surveying the computing landscape from above. They think more abstractly about how individual pieces can be put together to form larger and more complex systems. Architects are concerned about issues of scalability, extensi- bility, and reusability. Both types bring important skills to the system administration field. The system ad- ministrators I respect the most are those who can function as mechanics but whose preferred mindset is more closely aligned to that of an architect. They fix a problem and then spend time after the repair determining which systemic changes can be made to prevent it from occurring again. They think about how even small efforts on their part can be leveraged for future benefit. Well-run computing environments require both architects and mechanics working in a symbiotic relationship. A mechanic is most useful while working in a solid framework constructed by an architect. In the automobile world, we need mechanics to fix cars, but mechanics rely on the car designers to engineer slow-to-break, easy-to-repair vehicles. They need infrastructure like assembly lines, service manuals, and spare-part channels to do their job well. If an architect performs her job well, the mechanic’s job is made easier. How do these roles play out in the context of our discussion? Well, a mechanic will probably use the built-in OS tools for user management. He might even go so far as to write small scripts to help make individual management tasks, like adding users, easier. On the other hand, an architect looking at the same tasks will immediately start to construct an account system. An architect will think about issues like: • The repetitive nature of user management and how to automate as much of the process as possible. • The sorts of information an account system collects, and how a properly built account system can be leveraged as a foundation for other functionality. For in- stance, LDAP directory services and automatic website generation tools could be built on top of an account system. • Protecting the data in an account system (i.e., security). • Creating a system that will scale if the number of users increases. • Creating a system that other sites might be able to use. • How other system administration architects have dealt with similar problems. 74 | Chapter 3: User Accounts Mentioning the creation of a separate database makes some people nervous. They think, “Now I have to buy a really expensive commercial database, invest in another machine for it to run on, and hire a database administrator.” If you have thousands or tens of thousands of user accounts to manage, yes, you do need to do all of those things (though you may be able to get by with a noncommercial SQL database such as PostgreSQL or MySQL). If this is the case for you, you may want to turn to Chap- ter 7 for more information on dealing with databases like these in Perl. However, in this chapter when I say “database,” I’m using the term in the broadest sense. A flat-file, plain-text database will work fine for smaller installations. Windows users could even use an Access database file (e.g., database.mdb).† For portability and simplicity, we’ll use the very cool module DBM::Deep by Rob Kinyon. This module provides a surprising amount of power. The documentation describes it as follows: A unique flat-file database module, written in pure perl. True multilevel hash/array sup- port (unlike MLDBM, which is faked), hybrid OO / tie() interface, cross-platform FTPa- ble files, ACID transactions, and is quite fast. Can handle millions of keys and unlimited levels without significant slow-down. Written from the ground-up in pure perl—this is NOT a wrapper around a C-based DBM. Out-of-the-box compatibility with Unix, Mac OS X, and Windows. Choosing a Backend Database Format The first edition of this book used plain-text files in XML format as its backend data- base, so I think I need to clear the air about choosing a database format. XML was used as the basis of the backend database in that edition because it offered a relatively simple data format with a number of pluses. One of those pluses was pedagogical—it was clear to me even back before the turn of the millennium that being able to sling XML was going to be an important skill for a system administrator. Over time, that prediction proved to be true well beyond my guess at the time, and XML settled into several niches in the sysadmin world. One of those niches was configuration files, so you’ll find expanded and improved examples of XML use in Chapter 6. Its use for this kind of database and queue modeling isn’t all that prevalent, though, so I’ve swapped in a different database format. So what’s the best format to use for something like this? There’s a continuum of options to choose from. In rough order of increasing complexity, it goes something like this: • Flat-file text databases of various flavors (CSV, key/value pairs, YAML, XML, etc.) • DBM databases (ndbm, gdbm, BerkeleyDB, DBM::Deep, etc.) • File-based SQL (e.g., SQLite via DBI using DBD::SQLite) † But don’t. It will lead to heartbreak and misery. I’ve seen it happen too many times not to say, “Friends don’t let friends use Access as a multiuser database.” Building an Account System to Manage Users | 75 • Server(s)-based SQL (e.g., DBI used with MySQL, PostgreSQL, MS SQL, or Oracle) • Some sort of object-relational mapper, or ORM (DBIx::Class, Rose::DB::Object, Jifty::DBI, etc.) That last item isn’t a format per se, it’s more of a way of interacting with your data in a way that can potentially make your programming easier (read: you don’t have to sling SQL). Still, it is another layer on top of any of the previous items, so I list it later even if it has the potential to simplify things. Which format you choose will ultimately be dictated by your (present and future!) needs. I can’t give an easy recipe for choosing the right one because there are so many variables (amount of data, amount of concurrent read or read/write access, portability, and so on). In this chapter we’re going to use the principle of picking the simplest thing that will work (but no simpler). If I were to build a system like this for production use, I’d no doubt use some sort of SQL-based server (likely wrapped by an ORM). For the examples in this chapter we’ll use DBM::Deep because it is highly useful in many contexts and allows us to keep on topic without digressions about SQL or DBI (for more on those topics, see Chapter 7). Using DBM::Deep is a walk in the park if you’ve ever worked with hash references in Perl. Here’s a little sample: use DBM::Deep; my $db = DBM::Deep->new('accounts.db'); # imagine we received a hash of hashes constructed from repeated # invocations of CollectInformation() foreach my $user ( keys %users ) { # could also be written as $db->put($login => $users{$login}); $db->{$login} = $users{$login}; } # then, later on in the program or in another program... foreach my $login ( keys %{$db} ) { print $db->{$login}->{fullname}, "\n"; } The two emphasized lines show that the syntax is just your standard Perl hash reference syntax. They also show that it is possible to look up the hash key and have the entire hash of a hash stored in the database come back into memory. DBM::Deep can also use traditional OOP calls, as demonstrated by the comment in the code. 76 | Chapter 3: User Accounts Adding to the account queue Let’s start by returning to the cliffhanger we left off with in “Building an Account System to Manage Users” on page 71. I mentioned that we needed to write the account infor- mation we collected with CollectInformation() to our add queue file, but we didn’t actually look at the code to perform this task. Let’s see how the record data is written with DBM::Deep: sub AppendAccount { use DBM::Deep; # will move this to another place in the script # receive the full path to the file my $filename = shift; # receive a reference to an anonymous record hash my $record = shift; my $db = DBM::Deep->new($filename); $db->{ $record->{login} } = $record; } It really is that simple. This subroutine is just a wrapper around the addition of another key in the DBM::Deep magic hash we’re keeping. I should fess up about two things: • This really isn’t a queue in the classic sense of the word, because placing the items in the hash isn’t preserving any sort of order. If that really bugs you, you could pull the items out of the hash and sort them by record modification time (one of the fields we added in CollectInformation()) before processing. • If you passed in two records with the same login field, the second would overwrite the first. That may actually be a desirable quality in this context. Changing this behavior would be pretty simple; all you’d need to do would be to first test for the presence of that key in the DBM::Deep data structure using exists(). The example in this chapter is intentionally meant to be toy-sized. When you write your production system, you’ll be adding in all sorts of error checking and business logic appropriate to your environment. Now we can use just one line to collect data and write it to our add queue file: AppendAccount( $addqueue, &CollectInformation ); Reading this information back out of the queue files will be as easy as a hash lookup, so I’ll pass on showing you the code to do that until we look at the final program. Building an Account System to Manage Users | 77 The Low-Level Component Library Now that we have all the data under control, including how it is acquired, written, read, and stored, let’s look at how it is actually used deep in the bowels of our account system. We’re going to explore the code that actually creates and deletes users. The key to this section is the notion that we are building a library of reusable components. The better you are able to compartmentalize your account system routines, the easier it will be to change only small pieces when it comes time to migrate your system to some other operating system or make changes. This may seem like unnecessary caution on our part, but the one constant in system administration work is change. Unix account creation and deletion routines Let’s begin with the code that handles Unix account creation. Most of this code will be pretty trivial, because we’re going to take the easy way out: our account creation and deletion routines will call vendor-supplied “add user,” “delete user,” and “change password” executables with the right arguments. Why the apparent cop-out? We are using this method because we know the OS-specific executable will play nice with the other system components. Specifically, this method: • Handles the locking issues for us (i.e., avoids the data corruption problems that two programs simultaneously trying to write to the password file can cause). • Handles the variations in password file formats (including password encoding) that we discussed earlier. • Is likely to be able to handle any OS-specific authentication schemes or password distribution mechanisms. For instance, under at least one Unix variant I have seen, the external “add user” executable can directly add a user to the NIS maps on a master server. Drawbacks of using an external binary to create and remove accounts include: OS variations Each OS has a different set of binaries, located at a different place on the system, and those binaries take slightly different arguments. In a rare show of compatibility, almost all of the major Unix variants (Linux included, BSD variants excluded) have mostly compatible add and remove account binaries called useradd and userdel. The BSD variants use adduser and rmuser, two programs with similar purposes but very different argument names. Such variations tend to increase the complexity of our code.‡ There are some efforts (e.g., the POSIX standards) to standardize com- mands like these, but in practice I haven’t found things to be homogenous enough to depend on any one convention. ‡ If you want to get agitated about variations, take a look at OS X. It doesn’t (at this time) even have a user- account-specific set of commands. Instead, you get to learn dscl, a throwback to NetInfo. Nostalgic for NeXT cubes, anyone? 78 | Chapter 3: User Accounts Single machine scope Most user command-line tools operate only on the local machine. If most of your users are (as is the best practice these days) in a centralized authentication store like LDAP, these commands seldom know how to create users in that central sys- tem. Windows’s net command is one notable exception to this. It’s pretty common for people to write their own user* commands (in Perl, even) to perform these functions. Security concerns The program we call and the arguments passed to it will be exposed to users wield- ing the ps command. If accounts are created only on a secure machine (say, a master server), this reduces the data leakage risk considerably. Added dependency If the executable changes for some reason or is moved, our account system is kaput. Loss of control We have to treat a portion of the account creation process as being atomic; in other words, once we run the executable we can’t intervene or interleave any of our own operations. Error detection and recovery become more difficult. These programs rarely do it all It’s likely that these programs will not perform all of the steps necessary to instan- tiate an account at your site. If you need to add specific user types to specific aux- iliary groups, place users on a site-wide mailing list, or add users to a license file for a commercial package, you’ll have to add some more code to handle these specificities. This isn’t a problem with the approach itself; it’s more of a heads up that any account system you build will probably require more work on your part than just calling another executable. This will not surprise most system adminis- trators, since system administration is very rarely a walk in the park. For the purposes of our demonstration account system, the positives of this approach outweigh the negatives, so let’s take a look at some code that uses external executables. To keep things simple, we’ll use show code that works under Linux on a local machine only, ignoring complexities like NIS and BSD variations. If you’d like to see a more complex example of this method in action, you may find the CfgTie family of modules by Randy Maas instructive. After the example Linux code, we’ll take a quick look at some of the lessons that can be learned from other Unix variants that are less friendly to command-line administration. Here’s our basic account creation routine: # these variables should really be set in a central configuration file Readonly my $useraddex => '/usr/sbin/useradd'; # location of useradd cmd Readonly my $homeUnixdirs => '/home'; # home directory root dir Readonly my $skeldir => '/home/skel'; # prototypical home directory Readonly my $defshell => '/bin/zsh'; # default shell Building an Account System to Manage Users | 79 sub CreateUnixAccount { my ( $account, $record ) = @_; ### construct the command line, using: # -c = comment field # -d = home dir # -g = group (assume same as user type) # -m = create home dir # -k = copy in files from this skeleton dir # -p = set the password # (could also use -G group, group, group to add to auxiliary groups) my @cmd = ( $useraddex, '-c', $record->{fullname}, '-d', "$homeUnixdirs/$account", '-g', $record->{type}, '-m', '-k', $skeldir, '-s', $defshell, '-p', $record->{password}, $account ); # this gets the return code of the @cmd called, not of system() itself my $result = 0xff & system @cmd; # the return code is 0 for success, non-0 for failure, so we invert return ( ($result) ? 0 : 1 ); } This adds the appropriate entry to our password file, creates a home directory for the account, and copies over some default environment files (.profile, .tcshrc, .zshrc, etc.) from a skeleton directory. For symmetry’s sake, here’s the simpler account deletion code: # this variable should really be set in a central configuration file Readonly my $userdelex => '/usr/sbin/userdel'; # location of userdel cmd sub DeleteUnixAccount { my ( $account, $record ) = @_; ### construct the command line, using: # -r = remove the account's home directory for us my @cmd = ( $userdelex, '-r', $account ); my $result = 0xff & system @cmd; # the return code is 0 for success, non-0 for failure, so we invert return ( ($result) ? 0 : 1 ); } 80 | Chapter 3: User Accounts Unix account creation and deletion routines—a variation Before we get to the Windows examples, I want to show you one variation on the code we just looked at because it is instructive on a number of levels. The variation I have in mind not only demonstrates a cool technical trick but also brings to sharp relief how one little difference between operating systems can cause ripples throughout your code. Here’s the innocent little detail that is about to bite us: Solaris’s useradd command does not have a –p switch to set the (hashed) password on a new account. It does have a –p switch, but it doesn’t do the same thing as its counterpart in Linux. “Ho hum,” you say, “I’ll just change the part of the CreateUnixAccount() code that sets @cmd to reflect the command-line argument that Solaris does use for this purpose.” A quick read of the Solaris manpage for useradd, however, will send your naiveté packing, as you’ll soon see that Solaris doesn’t have a supported way to provide a hashed password for a new account. Instead, every account is locked until the password is changed for that account. This impacts the code in a number of ways. First, we have to add something to CreateUnixAccount() so it will perform a password change after creating an account. That’s easy enough. We can just add something like this: $result = InitUnixPasswd( $account, $record->{'password'} ) ); return 0 if (!$result); and then write an InitUnixPasswd() routine. But that’s not the most important change to the code. The biggest change is that now we have to store the plain-text password for the account in our queue, since there’s no way to use a one-way-hashed password as input into a password changing routine. Remember the ominous comment in the code presented at the very beginning of this section, for CollectInformation(): # if it is a password field, encrypt it using a random salt before storing if ( $field eq 'password' ) { # believe it or not, we may regret the decision to store # the password in a hashed form like this; we'll talk about # this a little later on in this chapter $record{$field} = unix_md5_crypt( prompt( 'p', 'Please enter password:', '', '' ), undef ); } Here’s where we regret that decision. We’ll have a similar regret in a moment when we get to create accounts in Windows, because we’ll need the plain-text password there too. I’m not going to show an example here, but perhaps the best middle ground would be to use a cipher module from the Crypt:: namespace to store the password in a fashion that can be decrypted later.* I point all of this out because it is ripple situations along these lines that can make attempts to decouple the parts of your program hard at times. * This means you’ll have to protect the secret used to encrypt/decrypt the account password by either protecting the script or the script’s config files at the OS level. This is the digression I’m not going to entertain at this point. Building an Account System to Manage Users | 81 Once you’ve made all of the necessary changes to the password prompting and storing code, you then have to sit down and write the password changing code. The bucket of cold water gets dumped on your head at the point where you realize Solaris doesn’t ship with a noninteractive password-changing program.† Setting the password requires a little sleight of hand, so we’ll encapsulate that step in a separate subroutine to keep the details out of our way. The Solaris manual pages say, “The new login remains locked until the passwd(1) com- mand is executed.” passwd will change that account’s password, which may sound simple enough. However, there’s a problem lurking here. The passwd com- mand expects to prompt the user for the password, and it takes great pains to make sure it is talking to a real user by interacting directly with the user’s terminal device. As a result, the following will not work: # this code DOES NOT WORK open my $PW, "|passwd $account"; print $PW $newpasswd,"\n"; print $PW $newpasswd,"\n"; close $PW; We have to be craftier than usual; somehow faking passwd into thinking it is dealing with a human rather than our Perl code. We can achieve this level of duplicity by using Expect, a Perl module by Austin Schutz (now maintained by Roland Giersig) that sets up a pseudoterminal (pty) within which another program will run. Expect is heavily based on the famous Tcl program Expect by Don Libes. This module is part of the family of bidirectional program interaction modules. We’ll see its close relative, Jay Rogers’s Net::Telnet, in Chapter 9. These modules function using the same basic conversational model: wait for output from a program, send it some input, wait for a response, send some data, and so on. The following code starts up passwd in a pty and waits for it to prompt for the password. The discussion we have with passwd should be easy to follow: Readonly my $passwdex => '/usr/bin/passwd'; # location of passwd executable sub InitUnixPasswd { use Expect; # we'll move this later my ( $account, $passwd ) = @_; # return a process object my $pobj = Expect->spawn( $passwdex, $account ); die "Unable to spawn $passwdex:$!\n" unless ( defined $pobj ); # do not log to stdout (i.e., be silent) $pobj->log_stdout(0); † If you are willing to use software that doesn’t ship with Solaris for this purpose, you could look at changepass, part of the cgipaf package at 82 | Chapter 3: User Accounts # wait for password & password re-enter prompts, # answering appropriately $pobj->expect( 10, 'New password: ' ); # Linux sometimes prompts before it is ready for input, so we pause sleep 1; print $pobj "$passwd\r"; $pobj->expect( 10, 'Re-enter new password: ' ); print $pobj "$passwd\r"; # did it work? $result = ( defined( $pobj->expect( 10, 'successfully changed' ) ) ? 1:0 ); # close the process object, waiting up to 15 secs for # the process to exit $pobj->soft_close(); return $result; } The Expect module meets our meager needs well in this routine, but it is worth noting that the module is capable of much more complex operations. See the documentation and tutorial included with the Expect module for more information. Before we move on, I do want to mention one other alternative to using Expect. I don’t like this alternative because it bypasses the usual password changing code path, but it may serve a purpose for you. If you don’t want to script the running of passwd, Eric Estabrook’s Passwd::Solaris module, mentioned earlier in this chapter, can be used to operate directly on the Solaris /etc/passwd and /etc/shadow files to change a user’s pass- word. It does accept a hashed password for this purpose. If you are going to hash your own passwords and then insert them into your passwd and shadow files, be sure that you have Solaris (9, 12/02, or later) configured for the compatible hashing algorithm in /etc/secur- ity/policy.conf. Windows account creation and deletion routines The process of creating and deleting an account under Windows is slightly easier than the process under Unix, because standard API calls for these operations exist in Win- dows. As in Unix, we could call an external executable to handle the job (e.g., the ubiquitous net command with its USERS /ADD switch), but it is easy to use the native API calls from a handful of different modules, some of which we’ve mentioned earlier. Account creation functions exist in Win32::NetAdmin, Win32::UserAdmin, Win32API::Net, and Win32::Lanman, to name a few. Active Directory users will find the ADSI material in Chapter 9 to be their best route. Building an Account System to Manage Users | 83 Picking among these Windows-centric modules is mostly a matter of personal prefer- ence. To illustrate the differences between them, we’ll take a quick look behind the scenes at the native user creation API calls. These calls are described in the Network Management SDK documentation on (search for “NetUserAdd” if you have a hard time finding it). NetUserAdd() and the other calls each take a parameter that specifies the information level of the data being submitted. For instance, with information level 1, the C structure that is passed in to the user creation call looks like this: typedef struct _USER_INFO_1 { LPWSTR usri1_name; LPWSTR usri1_password; DWORD usri1_password_age; DWORD usri1_priv; LPWSTR usri1_home_dir; LPWSTR usri1_comment; DWORD usri1_flags; LPWSTR usri1_script_path; } If information level 2 is used, the structure expected is expanded considerably: typedef struct _USER_INFO_2 { LPWSTR usri2_name; LPWSTR usri2_password; DWORD usri2_password_age; DWORD usri2_priv; LPWSTR usri2_home_dir; LPWSTR usri2_comment; DWORD usri2_flags; LPWSTR usri2_script_path; DWORD usri2_auth_flags; LPWSTR usri2_full_name; LPWSTR usri2_usr_comment; LPWSTR usri2_parms; LPWSTR usri2_workstations; DWORD usri2_last_logon; DWORD usri2_last_logoff; DWORD usri2_acct_expires; DWORD usri2_max_storage; DWORD usri2_units_per_week; PBYTE usri2_logon_hours; DWORD usri2_bad_pw_count; DWORD usri2_num_logons; LPWSTR usri2_logon_server; DWORD usri2_country_code; DWORD usri2_code_page; } Levels 3 and 4 (4 being the one Microsoft recommends you use‡) look like this: ‡ Showing you the user info level 4 structure is a bit of a tease, because as of this writing none of the Perl modules support it. It won’t be too big of a loss should this still be true when you read this (level 3 and level 4 aren’t that different), but I thought you should know. 84 | Chapter 3: User Accounts typedef struct _USER_INFO_3 { LPWSTR usri3_name; LPWSTR usri3_password; DWORD usri3_password_age; DWORD usri3_priv; LPWSTR usri3_home_dir; LPWSTR usri3_comment; DWORD usri3_flags; LPWSTR usri3_script_path; DWORD usri3_auth_flags; LPWSTR usri3_full_name; LPWSTR usri3_usr_comment; LPWSTR usri3_parms; LPWSTR usri3_workstations; DWORD usri3_last_logon; DWORD usri3_last_logoff; DWORD usri3_acct_expires; DWORD usri3_max_storage; DWORD usri3_units_per_week; PBYTE usri3_logon_hours; DWORD usri3_bad_pw_count; DWORD usri3_num_logons; LPWSTR usri3_logon_server; DWORD usri3_country_code; DWORD usri3_code_page; DWORD usri3_user_id; DWORD usri3_primary_group_id; LPWSTR usri3_profile; LPWSTR usri3_home_dir_drive; DWORD usri3_password_expired; } and: typedef struct _USER_INFO_4 { LPWSTR usri4_name; LPWSTR usri4_password; DWORD usri4_password_age; DWORD usri4_priv; LPWSTR usri4_home_dir; LPWSTR usri4_comment; DWORD usri4_flags; LPWSTR usri4_script_path; DWORD usri4_auth_flags; LPWSTR usri4_full_name; LPWSTR usri4_usr_comment; LPWSTR usri4_parms; LPWSTR usri4_workstations; DWORD usri4_last_logon; DWORD usri4_last_logoff; DWORD usri4_acct_expires; DWORD usri4_max_storage; DWORD usri4_units_per_week; PBYTE usri4_logon_hours; DWORD usri4_bad_pw_count; DWORD usri4_num_logons; Building an Account System to Manage Users | 85 LPWSTR usri4_logon_server; DWORD usri4_country_code; DWORD usri4_code_page; PSID usri4_user_sid; DWORD usri4_primary_group_id; LPWSTR usri4_profile; LPWSTR usri4_home_dir_drive; DWORD usri4_password_expired; } Without knowing anything about these parameters, or even much about C at all, you can still tell that a change in level increases the amount of information that can and must be specified as part of the user creation process. Also, it should be obvious that each subsequent information level is a superset of the previous one. What does this have to do with Perl? Each of the modules I’ve mentioned makes two decisions: • Should the notion of “information level” be exposed to the Perl programmer? • Which information level (i.e., how many parameters) can the programmer use? Win32API::Net and Win32::UserAdmin both allow the programmer to choose an infor- mation level. Win32::NetAdmin and Win32::Lanman do not. Of these modules, Win32::NetAdmin exposes the least number of parameters; for example, you cannot set the full_name field as part of the user creation call. If you choose to use Win32::NetAdmin, you will probably have to supplement it with calls from another module to set the additional parameters it does not expose. Now you have some idea why the module choice really boils down to personal prefer- ence. A good strategy might be to first decide which parameters are important to you, store the values for each of these parameters in the database, and then find a comfort- able module that supports them. For our demonstration subroutines we’ll use Win32API::Net, to stay consistent with our previous examples. Here’s the user creation and deletion code for our account system: use Win32API::Net qw(:User :LocalGroup); # for account creation use Win32::Security::NamedObject; # for home directory perms use Readonly; # each user will get a "data dir" in addition to her home directory # (the OS will create the home dir for us with the right permissions the first # time the user logs in) Readonly my $homeWindirs => '\\\\homeserver\\home'; # home directory root dir Readonly my $dataWindirs => '\\\\homeserver\\data'; # data directory root dir sub CreateWinAccount { my ( $account, $record ) = @_; my $error; # used to send back error messages in next call # ideally the default values for this sort of add would come out of a database 86 | Chapter 3: User Accounts my $result = UserAdd( '', # create this account on the local machine 3, # will specify USER_INFO_3 level of detail { acctExpires => −1, # no expiration authFlags => 0, # read only, no value necessary badPwCount => 0, # read only, no value necessary codePage => 0, # use default comment => '', # didn't ask for this from user countryCode => 0, # use default # must use these flags for normal acct flags => ( Win32API::Net::UF_SCRIPT() & Win32API::Net::UF_NORMAL_ACCOUNT() ), fullName => $record->{fullname}, homeDir => "$homeWindirs\\$account", homeDirDrive => 'H', # we map H: to home dir lastLogon => 0, # read only, no value necessary lastLogoff => 0, # read only, no value necessary logonHours => [], # no login restrictions logonServer => '', # read only, no value necessary maxStorage => −1, # no quota set name => $account, numLogons => 0, # read only, no value necessary parms => '', # unused password => $record->{password}, # plain-text passwd passwordAge => 0, # read only passwordExpired => 0, # don't force user to immediately change passwd primaryGroupId => 0x201, # magic value as instructed by doc priv => USER_PRIV_USER(), # normal (not admin) user profile => '', # don't set one at this time scriptPath => '', # no logon script unitsPerWeek => 0, # for logonHours, not used here usrComment => '', # didn't ask for this from user workstations => '', # don't specify specific wkstns userId => 0, # read only }, $error ); return 0 unless ($result); # could return Win32::GetLastError() # add to appropriate LOCAL group # we assume the group name is the same as the account type $result = LocalGroupAddMembers( '', $record->{type}, [$account] ); return 0 if (!$result); # create data directory mkdir "$dataWindirs\\$account", 0777 or (warn "Unable to make datadir:$!" && return 0); # change the owner of the directory my $datadir = Win32::Security::NamedObject->new( 'FILE', "$dataWindirs\\$account" ); eval { $datadir->ownerTrustee($account) }; Building an Account System to Manage Users | 87 if ($@) { warn "can't set owner: $@"; return 0; } # we give the user full control of the directory and all of the # files that will be created within it my $dacl = Win32::Security::ACL->new( 'FILE', [ 'ALLOW', 'FULL_INHERIT', 'FULL', $account ], ); eval { $datadir->dacl($dacl) }; if ($@) { warn "can't set permissions: $@"; return 0; } } The user deletion code looks like this: use Win32API::Net qw(:User :LocalGroup); # for account deletion use File::Path 'remove_tree'; # for recursive directory deletion use Readonly; sub DeleteWinAccount { my ( $account, $record ) = @_; # Remove user from LOCAL groups only. If we wanted to also # remove from global groups we could remove the word "Local" from # the two Win32API::Net calls (e.g., UserGetGroups/GroupDelUser) # also: UserGetGroups can take a flag to search for indirect group # membership (for example, if user is in group because that group # contains another group that has that user as a member) UserGetLocalGroups( '', $account, \my @groups ); foreach my $group (@groups) { return 0 if (! LocalGroupDelMembers( '', $group, [$account] ); } # delete this account on the local machine # (i.e., empty first parameter) unless ( UserDel( '', $account ) ) { warn 'Can't delete user: ' . Win32::GetLastError(); return 0; } # delete the home and data directory and its contents # remove_tree puts its errors into $err (ref to array of hash references) # note: remove_tree() found in File::Path 2.06+; before it was rmtree remove_tree( "$homeWindirs\\$account", { error => \my $err } ); if (@$err) { warn "can't delete $homeWindirs\\$account\n" ; return 0; } 88 | Chapter 3: User Accounts remove_tree( "$dataWindirs\\$account", { error => \my $err } ); if (@$err) { warn "can't delete $dataWindirs\\$account\n" ; return 0; } else { return 1; } } As a quick aside, the preceding code uses the portable File::Path module to remove an account’s home directory. If we wanted to do something Windows-specific, like move the home directory to the Recycle Bin instead, we could use a module called Win32::FileOp by Jenda Krynicky, available at In this case, we’d use Win32::FileOp and change the rmtree() line to: # will move directories to the Recycle Bin, potentially confirming # the action with the user if our account is set to confirm # Recycle Bin actions my $result = Recycle("$homeWindirs\\$account"); my $result = Recycle("$dataWindirs\\$account"); This same module also has a Delete() function that will perform the same operation as the remove_tree() call, in a less portable (although quicker) fashion. The Process Scripts Once we have a backend database, we’ll want to write scripts that encapsulate the day- to-day and periodic processes that take place for user administration. These scripts are based on a low-level component library ( we’ll create by concatenating all of the subroutines we just wrote together into one file. To make it load properly as a module, we’ll need to add a 1; at the end. The other change we’ll make in this conver- sion is to move all of the module and variable initialization code to an initialization subroutine and remove those parts (leaving behind our statements as necessary) from the other subroutines. Here’s the initialization subroutine we’ll use: sub InitAccount { # we use these modules in both the Linux and Win32 routines use DBM::Deep; use Readonly; use Term::Prompt; # we use these global variables for both the Linux and Win32 routines Readonly our $record => { fields => [ 'login', 'fullname', 'id', 'type', 'password' ] }; Readonly our $addqueue => 'add.db'; # name of add account queue file Readonly our $delqueue => 'del.db'; # name of del account queue file Readonly our $maindata => 'acct.db'; # name of main account database file # load the Win32-only modules and set the Win32-only global variables if ( $^O eq 'MSWin32' ) { Building an Account System to Manage Users | 89 require Win32API::Net; import Win32API::Net qw(:User :LocalGroup); require Win32::Security::NamedObject; require File::Path; import File::Path 'remove_tree'; # location of account files Readonly our $accountdir => "\\\\server\\accountsystem\\"; # mail lists, example follows Readonly our $maillists => $accountdir . "maillists\\"; # home directory root Readonly our $homeWindirs => "\\\\homeserver\\home"; # data directory root Readonly our $dataWindirs => "\\\\homeserver\\home"; # name of account add subroutine Readonly our $accountadd => \&CreateWinAccount; # name of account del subroutine Readonly our $accountdel => \&DeleteWinAccount; } # load the Unix-only modules and set the Unix-only global variables else { require Expect; # for Solaris password changes require Crypt::PasswdMD5; # location of account files Readonly our $accountdir => '/usr/accountsystem/'; # mail lists, example follows Readonly our $maillists => '$accountdir/maillists/'; # location of useradd executable Readonly our $useraddex => '/usr/sbin/useradd'; # location of userdel executable Readonly our $userdelex => '/usr/sbin/userdel'; # location of passwd executable Readonly our $passwdex => '/usr/bin/passwd'; # home directory root dir Readonly our $homeUnixdirs => '/home'; # prototypical home directory Readonly our $skeldir => '/home/skel'; # default shell Readonly our $defshell => '/bin/zsh'; # name of account add subroutine 90 | Chapter 3: User Accounts Readonly our $accountadd => \&CreateUnixAccount; # name of account del subroutine Readonly our $accountdel => \&DeleteUnixAccount; } } Let’s look at some sample scripts. Here’s the script that processes the add queue: use Account; # read in our low-level routines &InitAccount; # read the contents of the add account "queue" my $queue = ReadAddQueue(); # attempt to create all accounts in the queue ProcessAddQueue($queue); # write account record to main database, or back to queue # if there is a problem DisposeAddQueue($queue); # read in the add account queue to the $queue data structure sub ReadAddQueue { our ( $accountdir, $addqueue ); my $db = DBM::Deep->new( $accountdir . $addqueue ); my $queue = $db->export(); return $queue; } # iterate through the queue structure, attempting to create an account # for each request (i.e., each key) in the structure sub ProcessAddQueue { my $queue = shift; our $accountadd; foreach my $login ( keys %{$queue} ) { my $result = $accountadd->( $login, $queue->{$login} ); if ( $result ) { $queue->{$login}{status} = 'created'; } else { $queue->{$login}{status} = 'error'; } } } # Now iterate through the queue structure again. For each account with # a status of "created," append to main database. All others get written # back to the add queue database, overwriting the record's information. sub DisposeAddQueue { Building an Account System to Manage Users | 91 my $queue = shift; our ( $accountdir, $addqueue, $maindata ); my $db = DBM::Deep->new( $accountdir . $addqueue ); foreach my $login ( keys %{$queue} ) { if ( $queue->{$login}{status} eq 'created' ) { $queue->{$login}{login} = $login; $queue->{$login}{creation_date} = time; AppendAccount( $accountdir . $maindata, $queue->{$login} ); delete $queue->{$login}; # delete from in-memory representation delete $db->{$login}; # delete from disk database file } } # all we have left in $queue at this point are the accounts that # could not be created # merge in the queue info my $db = DBM::Deep->new( $accountdir . $addqueue ); my $queue = $db->import($queue); } Our “process the delete user queue file” script is almost identical: use Account; # read in our low-level routines &InitAccount; # read the contents of the del account "queue" my $queue = ReadDelQueue(); # attempt to delete all accounts in the queue ProcessDelQueue($queue); # write account record to main database, or back to queue # if there is a problem DisposeDelQueue($queue); # read in the add account queue to the $queue data structure sub ReadDelQueue { our ( $accountdir, $delqueue ); my $db = DBM::Deep->new( $accountdir . $delqueue ); my $queue = $db->export(); return $queue; } # iterate through the queue structure, attempting to create an account # for each request (i.e., each key) in the structure sub ProcessDelQueue { 92 | Chapter 3: User Accounts my $queue = shift; our $accountdel; foreach my $login ( keys %{$queue} ) { my $result = $accountdel->( $login, $queue->{$login} ); if ( !defined $result ) { $queue->{$login}{status} = 'deleted'; } else { $queue->{$login}{status} = 'error'; } } } # Now iterate through the queue structure again. For each account with # a status of "deleted," change the main database information. All that # could not get be deleted gets merged back into the del queue file, # updating it. sub DisposeDelQueue { my $queue = shift; our ( $accountdir, $delqueue, $maindata ); my $maindata = DBM::Deep->new( $accountdir . $maindata ); my $delqueue = DBM::Deep->new( $accountdir . $delqueue ); foreach my $login ( keys %{$queue} ) { if ( $queue->{$login}{status} eq 'deleted' ) { $maintada->{$login}{deletion_date} = time; delete $queue->{$login}; # delete from in-memory representation delete $delqueue->{$login}; # delete from on disk del queue file } } # All we have left in $queue at this point are the accounts that # could not be deleted. We merge these status changes back # into the delete queue for future action of some sort. $delqueue->import($queue); } You can probably imagine writing many other process scripts. For example, we could certainly use scripts that perform data export and consistency checking tasks (e.g., does the user’s home directory match up with the main database’s account type? is that user in the appropriate group?). We don’t have space to cover the whole array of possible programs, so let’s end this section with a single example of the data export variety. Earlier I mentioned that a site might want a separate mailing list for each type of user on the system. The following code reads our main database and creates a set of files that contain usernames, with one file per user type: use Account; # just to get the file locations &InitAccount; Building an Account System to Manage Users | 93 # clearly this doesn't work so well on a large data set my $database = ReadMainDatabase(); WriteFiles($database); # read the main database into a hash of hashes sub ReadMainDatabase { our ( $accountdir, $maindata ); my $db = DBM::Deep->new( $accountdir . $maindata ); my $database = $db->export(); return $database; } # Iterate through the keys, compile the list of accounts of a certain # type, and store them in a hash of lists. Then write out the contents of # each key to a different file. sub WriteFiles { my $database = shift; our ( $accountdir, $maillists ); my %types; foreach my $account ( keys %{$database} ) { next if $database->{$account}{status} eq 'deleted'; push( @{ $types{ $database->{$account}{type} } }, $account ); } foreach my $type ( keys %types ) { open my $OUT, '>', $maillists . $type or die 'Unable to write to ' . $maillists . $type . ": $!\n"; print $OUT join( "\n", sort @{ $types{$type} } ) . "\n"; close $OUT; } } If we look at the mailing list directory, we see: > dir faculty staff Each of those files contains the appropriate list of user accounts. Account System Wrap-Up Now that we’ve explored four components of our account system, let’s wrap up this section by talking about what’s missing (besides oodles of functionality): 94 | Chapter 3: User Accounts Error checking Our demonstration code has only a modicum of error checking. Any self-respecting account system would grow another 40%–50% in code size because it would check for data and system interaction problems every step of the way. Error reporting The code is abysmal (for simplicity’s sake) at reporting back errors in a way that could help with debugging processes gone wrong. The routines pass back a 0 to indicate failure, but what they really should be doing is handing back exceptions or exception objects that contain more detail. Often we can get that detail from the system. For example in the case of the Win32API::Net calls in the Windows code, we could return the information from Win32::GetLastError() (or Win32::FormatMessage(Win32::GetLastError()) if we wanted to be super cool). Object orientation Even though I readily admit to having come to the land of object-oriented pro- gramming (OOP) late in life, I recognize that all of the global variables floating around this code are icky. The code could be much cleaner if it was rewritten to use objects instead, but I did not want to assume OOP knowledge just for the sake of this example. Scalability Our code could probably work in a small or mid-sized environment, but any time you see “read the entire file into memory,” it should set off warning bells. To scale, we would need to change our data storage and retrieval techniques, at the very least. Security This is related to the first item on error checking. In addition to a few truck-sized security holes (e.g., storing plain-text passwords), we do not perform any security checks in our code. We do not confirm that the data sources we use, such as the queue files, are trustworthy. Another 20%–30% should be added to the code size to take care of this issue. Multiuser Perhaps the largest flaw in our code as written is that we make no provision for multiple users or even multiple scripts running at once. In theory DBM::Deep is handling locking for us, but the code isn’t explicit enough in this regard. This is such an important issue that I’ll take a few moments to discuss it before concluding this section. Maintenance Addressing these weaknesses, even without adding features, would dramatically increase the size and complexity of the code. The result would be a large, complex, multi-OS program with functions that are critical to the business. Does the enter- prise have the staff and expertise to support ongoing software maintenance, and should that responsibility lie with the sysadmin who creates the code? These are questions that must be asked (and answered) in each environment. Building an Account System to Manage Users | 95 One way to help with the multiuser deficiency is to carefully introduce explicit file locking. File locking allows the different scripts to cooperate. If a script plans to read or write to a file, it can attempt to lock the file first. If it can obtain a lock, it knows it is safe to manipulate the file. If it cannot lock the file (because another script is using it), it knows not to proceed with an operation that could corrupt data. Of course, there’s considerably more complexity involved with locking and multiuser access in general than just this simple description reveals, as you’ll see if you consult any fundamental operating or distributed systems text. It gets especially tricky when dealing with files residing on network filesystems, where there may not be a good locking mechanism. DBM::Deep’s documentation makes explicit mention of not handling locking on NFS filesystems. If you don’t want to trust the built-in locking, here are a few hints that may help you when you approach this topic using Perl: • There are smart ways to cheat. My favorite method is to use the lockfile program distributed with the popular mail filtering program procmail (http://www.procmail .org). The procmail installation procedure takes great pains to determine safe lock- ing strategies for the filesystems you are using. lockfile does just what its name suggests, hiding most of the complexity in the process. • If you don’t want to use an external executable, there are a plethora of locking modules available: for example, File::Flock by David Muir Sharnoff, File::LockDir from the Perl Cookbook by Tom Christiansen and Nathan Torking- ton (O’Reilly), File::Lock by Kenneth Albanowski, File::Lockf by Paul Henson, and Lockfile::Simple by Raphael Manfredi. They differ mostly in terms of their interfaces, though Lockfile::Simple attempts to perform locking without using Perl’s flock() function. Shop around and pick the best one for your needs. • Locking is easier to get right if you remember to lock before attempting to change data (or read data that could change) and unlock only after making sure that data has been written (e.g., after the file has been closed). For more information on this, see the previously mentioned Perl Cookbook, the Perl Frequently Asked Questions list (, and the documentation that comes with Perl on the flock() function and the DB_File module. This ends our look at user account administration and how it can be taken to the next level with a bit of an architectural mindset. These concepts—particularly the “self- review” of deficiencies in the account administration program—can be applied to many projects and can be very helpful in architecting system administration tools, rather than just writing scripts. In this chapter we’ve concentrated on the beginning and the end of an account’s life- cycle. In the next chapter, we’ll examine what users do in between these two points. 96 | Chapter 3: User Accounts Module Information for This Chapter Name CPAN ID Version User::pwent (ships with Perl) 1.00 File::stat (ships with Perl) 1.01 Passwd::Solaris EESTABROO 1.2 Passwd::Linux EESTABROO 1.2 Win32API::Net JDB 0.12 Win32::Security(::NamedObject, ::ACL) TEVERETT 0.50 Win32::OLE JDB 0.1709 Term::Prompt PERSICOM 1.04 Crypt::PasswdMD5 LUISMUNOZ 1.3 DBM::Deep RKINYON 1.0014 Readonly ROODE 1.03 Expect RGIERSIG 1.21 File::Path (ships with Perl) DLAND 2.07 Win32::FileOp JENDA 0.14.1 References for More Information Using a set of central databases from which configuration files are automatically gen- erated is a best practice that shows up in a number of places in this book; credit for my exposure to this methodology goes to Rémy Evard. Though it is now in use at many sites, I first encountered it when I inherited the Tenwen computing environment he built (as described in the Tenwen paper at proceedings/lisa94/evard.html). See the section “Implemented the Hosts Database” for one example of this methodology in action. contains a number of Jon Finke’s published papers on the use of relational databases for system administration. Many of his papers were published at the LISA conference; see for the archives of past proceedings. Unix Password Files is where the FreeBSD Project provides access to the online manual pages for *BSD and other Unix variants. This is a handy way to compare the file formats and user administration commands (useradd, etc.) for several operating systems. References for More Information | 97 Practical Unix & Internet Security, Third Edition, by Simson Garfinkel et al. (O’Reilly), is an excellent place to start for information about password files. Windows User Administration is another site with useful Win32 modules applicable to user administration. hosts the Perl-Win32-Admin and Perl-Win32- Users mailing lists. Both lists and their archives are invaluable resources for Windows Perl programmers. Win32 Perl Programming: The Standard Extensions, Second Edition, and Win32 Perl Scripting: The Administrator’s Handbook, both by Dave Roth (Sams, 2001 and 2002), are a little dated but are still some of the best references for Win32 Perl module pro- gramming available. There are a whole slew of superb books that have Robbie Allen as author or coauthor, including Active Directory, Third Edition (O’Reilly), Active Directory Cookbook, Second Edition (O’Reilly), Managing Enterprise AD Services (Addison-Wesley), Windows Server Cookbook (O’Reilly), Windows Server 2003 Networking Recipes (Apress), Win- dows Server 2003 Security Cookbook (orm:hideurl:ital) (O’Reilly), and Windows XP Cookbook (orm:hideurl:ital) (O’Reilly). All of these are well worth reading, but it’s not the books I want to gush about. Allen has a website at that makes all of the code samples in all of the languages (including Perl translations of all the VBScript code) from all of these books available for viewing and for purchase. It truly is the mother lode of examples—one of the single most helpful websites for this sort of programming that you’ll ever find. Definitely buy the books and the code repository; this sort of effort deserves your support. has a wiki devoted to all things Win32-Perl related. The PPM repositories link at that site is especially helpful when you are trying to track down more modules for the ActiveState Perl distribution. is (as of this writing) the home for the Microsoft resource kits. is (again, as of this writing, they love to shuffle URLs in Redmond) a good place to search for the freely downloadable utilities from the resource kits (search for “resource kit”). 98 | Chapter 3: User Accounts CHAPTER 4 User Activity In the previous chapter, we explored the parts of a user’s identity and how to manage and store it. Now let’s talk about how to manage users while they are active on our systems and networks. Typical user activities fall into four domains: Processes Users run processes that can be spawned, killed, paused, and resumed on the ma- chines we manage. These processes compete for a computer’s finite processing power, adding resource issues to the list of problems a system administrator needs to mediate. File operations Most of the time, operations like writing, reading, creating, deleting, and so on take place when a specific user process interacts with files and directories in a filesystem. But under Unix, there’s more to this picture. Unix uses the filesystem as a gateway to more than just file storage. Device control, input/output, and even some process control and network access operations are file operations. We dealt with filesystem administration in Chapter 2, but in this chapter we’ll approach this topic from a user administration perspective. Network usage Users can send and receive data over network interfaces on our machines. There is material elsewhere in this book on networking, but we’ll address this issue here from a different perspective. OS-specific activities This last domain is a catchall for the OS-specific features that users can access via different APIs. Included in this list are things like GUI element controls, shared memory usage, file-sharing APIs, sound, and so on. This category is so diverse that it would be impossible to do it justice in this book. I recommend that you track down the OS-specific web sites for information on these topics. 99 Process Management We’ll begin by looking at ways to deal with the first three of these domains using Perl. Because we’re interested in user administration, the focus here will be on dealing with processes that other users have started. Windows-Based Operating System Process Control We’re going to briefly look at four different ways to deal with process control on Win- dows, because each of these approaches opens up a door to interesting functionality outside the scope of our discussion that is likely to be helpful to you at some point. We’re primarily going to concentrate on two tasks: finding all of the running processes and killing select processes. Using external binaries There are a number of programs available to us that display and manipulate processes. The first edition of this book used the programs pulist.exe and kill.exe from the Win- dows 2000 Resource Kit. Both are still available for download from Microsoft as of this writing and seem to work fine on later versions of the operating system. Another ex- cellent set of process manipulation tools comes from the Sysinternals utility collection, which Mark Russinovich and Bryce Cogswell formerly provided on their Sysinternals web site and which is now available through Microsoft (see the references section at the end of this chapter). This collection includes a suite of utilities called PsTools that can do things the standard Microsoft-supplied tools can’t handle. For our first example, we’re going to use two programs Microsoft ships with the operating system. The programs tasklist.exe and taskkill.exe work fine for many tasks and are a good choice for scripting in cases where you won’t want to or can’t download other programs to a machine. By default tasklist produces output in a very wide table that can sometimes be difficult to read. Adding /FO list provides output like this: Image Name: System Idle Process PID: 0 Session Name: Console Session#: 0 Mem Usage: 16 K Status: Running User Name: NT AUTHORITY\SYSTEM CPU Time: 1:09:06 Window Title: N/A Image Name: System PID: 4 Session Name: Console Session#: 0 Mem Usage: 212 K 100 | Chapter 4: User Activity Status: Running User Name: NT AUTHORITY\SYSTEM CPU Time: 0:00:44 Window Title: N/A Image Name: smss.exe PID: 432 Session Name: Console Session#: 0 Mem Usage: 372 K Status: Running User Name: NT AUTHORITY\SYSTEM CPU Time: 0:00:00 Window Title: N/A Image Name: csrss.exe PID: 488 Session Name: Console Session#: 0 Mem Usage: 3,984 K Status: Running User Name: NT AUTHORITY\SYSTEM CPU Time: 0:00:08 Window Title: N/A Image Name: winlogon.exe PID: 512 Session Name: Console Session#: 0 Mem Usage: 2,120 K Status: Running User Name: NT AUTHORITY\SYSTEM CPU Time: 0:00:08 Window Title: N/A Another format option for tasklist makes using it from Perl pretty trivial: CSV (Comma/ Character Separated Values). We’ll talk more about dealing with CSV files in Chap- ter 5, but here’s a small example that demonstrates how to parse that data: use Text::CSV_XS; my $tasklist = "$ENV{'SystemRoot'}\\SYSTEM32\\TASKLIST.EXE"; my $csv = Text::CSV_XS->new(); # /v = verbose (includes User Name), /FO CSV = CSV format, /NH - no header open my $TASKPIPE, '-|', "$tasklist /v /FO CSV /NH" or die "Can't run $tasklist: $!\n"; my @columns; while (<$TASKPIPE>) { next if /^$/; # skip blank lines in the input $csv->parse($_) or die "Could not parse this line: $_\n"; @columns = ( $csv->fields() )[ 0, 1, 6 ]; # grab name, PID, and User Name print join( ':', @columns ), "\n"; Process Management | 101 } close $TASKPIPE; tasklist can also provide some other interesting information, such as the dynamic link libraries (DLLs) used by a particular process. Be sure to run it with the /? switch to see its usage information. The other program I mentioned, taskkill.exe, is equally easy to use. It takes as an argument a task name (called the “image name”), a process ID, or a more complex filter to determine which processes to kill. I recommend the process ID format to stay on the safe side, since it is very easy to kill the wrong process if you use task names. taskkill offers two different ways to shoot down processes. The first is the polite death: taskkill.exe /PID will ask the specified process to shut itself down. However, if we add /F to the command line, it forces the issue: taskkill.exe /F /PID works more like the native Perl kill() function and kills the process with extreme prejudice. Using the Win32::Process::Info module The second approach* uses the Win32::Process::Info module, by Thomas R. Wyant. Win32::Process::Info is very easy to use. First, create a process info object, like so: use Win32::Process::Info; use strict; # the user running this script must be able to use DEBUG level privs my $pi = Win32::Process::Info->new( { assert_debug_priv => 1 } ); The new() method can optionally take a reference to a hash containing configuration information. In this case we set the config variable assert_debug_priv to true because we want our program to use debug-level privileges when requesting information. This is necessary if getting a list of all of the process owners is important to you. If you leave this out, you’ll find that the module (due to the Windows security system) will not be able to fetch the owner of some of the processes. There are some pretty scary warnings in the module’s documentation regarding this setting; I haven’t had any problems with it to date, but you should be sure to read the documentation before you follow my lead. Next, we retrieve the process information for the machine: my @processinfo = $pi->GetProcInfo(); @processinfo is now an array of references to anonymous hashes. Each anonymous hash has a number of keys (such as Name, ProcessId, CreationDate, and ExecutablePath), each with its expected value. To display our process info in the same fashion as the example from the last section, we could use the following code: * In the first edition of this book, this section was called “Using the Win32::IProc module.” Win32::IProc shared the fate of the module I describe in the sidebar “The Ephemeral Nature of Modules” on page 127. 102 | Chapter 4: User Activity use Win32::Process::Info; my $pi = Win32::Process::Info->new( { assert_debug_priv => 1 } ); my @processinfo = $pi->GetProcInfo(); foreach my $process (@processinfo) { print join( ':', $process->{'Name'}, $process->{'ProcessId'}, $process->{'Owner'} ), "\n"; } Once again, we get output like this: System Idle Process:0: System:4: smss.exe:432:NT AUTHORITY\SYSTEM csrss.exe:488:NT AUTHORITY\SYSTEM winlogon.exe:512:NT AUTHORITY\SYSTEM services.exe:556:NT AUTHORITY\SYSTEM lsass.exe:568:NT AUTHORITY\SYSTEM svchost.exe:736:NT AUTHORITY\SYSTEM svchost.exe:816:NT AUTHORITY\NETWORK SERVICE svchost.exe:884:NT AUTHORITY\SYSTEM svchost.exe:960:NT AUTHORITY\SYSTEM svchost.exe:1044:NT AUTHORITY\NETWORK SERVICE svchost.exe:1104:NT AUTHORITY\LOCAL SERVICE ccSetMgr.exe:1172:NT AUTHORITY\SYSTEM ccEvtMgr.exe:1200:NT AUTHORITY\SYSTEM spoolsv.exe:1324:NT AUTHORITY\SYSTEM ... Win32::Process::Info provides more info about a process than just these fields (perhaps more than you will ever need). It also has one more helpful feature: it can show you the process tree for all processes or just a particular process. This allows you to display the subprocesses for each process (i.e., the list of processes that process spawned) and the subprocesses for those subprocesses, and so on. So, for example, if we wanted to see all of the processes spawned by one of the processes just listed, we could write the following: use Win32::Process::Info; use Data::Dumper; my $pi = Win32::Process::Info->new( { assert_debug_priv => 1 } ); # PID 884 picked for this example because it has a small number of children my %sp = $pi->Subprocesses(884); print Dumper (\%sp); Process Management | 103 This yields: $VAR1 = { '3320' => [], '884' => [ 3320 ] }; which shows that this instance of svchost.exe (PID 884) has one child, the process with PID 3320. That process does not have any children. Using the GUI control modules (Win32::Setupsup and Win32::GuiTest) Of the approaches we’ll consider, this third approach is probably the most fun. In this section we’ll look at a module by Jens Helberg called Win32::Setupsup and a module by Ernesto Guisado, Jarek Jurasz, and Dennis K. Paulsen called Win32::GuiTest. They have similar functionality but achieve the same goals a little differently. We’ll look primarily at Win32::Setupsup, with a few choice examples from Win32::GuiTest. In the interest of full disclosure, it should be mentioned that (as of this writing) Win32::Setupsup had not been developed since October 2000 and is kind of hard to find (see the references at the end of this chapter). It still works well, though, and it has features that aren’t found in Win32::GuiTest; hence its inclusion here. If its orphan status bothers you, I recommend looking at Win32::GuiTest first to see if it meets your needs. Win32::Setupsup is called “Setupsup” because it is primarily designed to supplement software installation (which often uses a program called setup.exe). Some installers can be run in so-called “silent mode” for totally automated installation. In this mode they ask no questions and require no “OK” buttons to be pushed, freeing the administrator from having to babysit the install. Software installation mechanisms that do not offer this mode (and there are far too many of them) make a system ad- ministrator’s life difficult. Win32::Setupsup helps deal with these deficiencies: it can find information on running processes and manipulate them (or manipulate them dead if you so choose). For instructions on getting and installing Win32::Setupsup, refer to the section “Module Information for This Chapter” on page 97. With Win32::Setupsup, getting the list of running processes is easy. Here’s an example: 104 | Chapter 4: User Activity use Win32::Setupsup; use Perl6::Form; my $machine = ''; # query the list on the current machine # define the output format for Perl6::Form my $format = '{<<<<<<<} {<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<}'; my ( @processlist, @threadlist ); Win32::Setupsup::GetProcessList( $machine, \@processlist, \@threadlist ) or die 'process list error: ' . Win32::Setupsup::GetLastError() . "\n"; pop(@processlist); # remove the bogus entry always appended to the list print <<'EOH'; Process ID Process Name ========== =============================== EOH foreach my $processlist (@processlist) { print form $format, $processlist->{pid}, $processlist->{name}; } Killing processes is equally easy: KillProcess($pid, $exitvalue, $systemprocessflag) or die 'Unable to kill process: ' . Win32::Setupsup::GetLastError( ) . "\n"; The last two arguments are optional. The second argument kills the process and sets its exit value accordingly (by default, it is set to 0). The third argument allows you to kill system-run processes (providing you have the Debug Programs user right). That’s the boring stuff. We can take process manipulation to yet another level by in- teracting with the windows a running process may have open. To list all of the windows available on the desktop, we use: Win32::Setupsup::EnumWindows(\@windowlist) or die 'process list error: ' . Win32::Setupsup::GetLastError( ) . "\n"; @windowlist now contains a list of window handles that are converted to look like normal numbers when you print them. To learn more about each window, you can use a few different functions. For instance, to find the titles of each window, you can use GetWindowText() like so: use Win32::Setupsup; my @windowlist; Win32::Setupsup::EnumWindows( \@windowlist ) or die 'process list error: ' . Win32::Setupsup::GetLastError() . "\n"; my $text; foreach my $whandle (@windowlist) { if ( Win32::Setupsup::GetWindowText( $whandle, \$text ) ) { print "$whandle: $text", "\n"; Process Management | 105 } else { warn "Can't get text for $whandle" . Win32::Setupsup::GetLastError() . "\n"; } } Here’s a little bit of sample output: 66130: chapter04 - Microsoft Word 66184: Style 194905150: 66634: setupsup - WordPad 65716: Fuel 328754: DDE Server Window 66652: 66646: 66632: OleMainThreadWndName As you can see, some windows have titles, while others do not. Observant readers might notice something else interesting about this output. Window 66130 belongs to a Mi- crosoft Word session that is currently running (it is actually the one in which this chapter was composed). Window 66184 looks vaguely like the name of another window that might be connected to Microsoft Word. How can we tell if they are related? Win32::Setupsup has an EnumChildWindows() function that can show us the children of any given window. Let’s use it to write something that will show us a basic tree of the current window hierarchy: use Win32::Setupsup; my @windowlist; # get the list of windows Win32::Setupsup::EnumWindows( \@windowlist ) or die 'process list error: ' . Win32::Setupsup::GetLastError() . "\n"; # turn window handle list into a hash # NOTE: this conversion populates the hash with plain numbers and # not actual window handles as keys. Some functions, like # GetWindowProperties (which we'll see in a moment), can't use these # converted numbers. Caveat implementor. my %windowlist; for (@windowlist) { $windowlist{$_}++; } # check each window for children my %children; foreach my $whandle (@windowlist) { my @children; if ( Win32::Setupsup::EnumChildWindows( $whandle, \@children ) ) { # keep a sorted list of children for each window $children{$whandle} = [ sort { $a <=> $b } @children ]; 106 | Chapter 4: User Activity # remove all children from the hash; we won't directly # iterate over them foreach my $child (@children) { delete $windowlist{$child}; } } } # iterate through the list of windows and recursively print # each window handle and its children (if any) foreach my $window ( sort { $a <=> $b } keys %windowlist ) { PrintFamily( $window, 0, %children ); } # print a given window handle number and its children (recursively) sub PrintFamily { # starting window - how deep in a tree are we? my ( $startwindow, $level, %children ) = @_; # print the window handle number at the appropriate indentation print( ( ' ' x $level ) . "$startwindow\n" ); return unless ( exists $children{$startwindow} ); # no children, done # otherwise, we have to recurse for each child $level++; foreach my $childwindow ( @{ $children{$startwindow} } ) { PrintFamily( $childwindow, $level, %children ); } } There’s one last window property function we should look at before moving on: GetWindowProperties(). GetWindowProperties() is basically a catchall for the rest of the window properties we haven’t seen yet. For instance, using GetWindowProperties() we can query the process ID for the process that created a specific window. This could be combined with some of the functionality we just saw for the Win32::Process::Info module. The Win32::Setupsup documentation contains a list of the available properties that can be queried. Let’s use one of them to write a very simple program that will print the coordinates of a rectangular window on the desktop. GetWindowProperties() takes three arguments: a window handle, a reference to an array that contains the names of the properties to query, and a reference to a hash where the query results will be stored. Here’s the code we need for our task: use Win32::Setupsup; # Convert window ID into a form that GetWindowProperties can cope with. # Note: 'U' is a pack template that is only available in Perl 5.6+ releases. my $whandle = unpack 'U', pack 'U', $ARGV[0]; Process Management | 107 my %info; Win32::Setupsup::GetWindowProperties( $whandle, ['rect'], \%info ); print "\t" . $info{rect}{top} . "\n"; print $info{rect}{left} . ' -' . $whandle . '- ' . $info{rect}{right} . "\n"; print "\t" . $info{rect}{bottom} . "\n"; The output is a bit cutesy. Here’s a sample showing the top, left, right, and bottom coordinates of the window with handle 66180: 154 272 −66180- 903 595 GetWindowProperties() returns a special data structure for only one property, rect. All of the others will simply show up in the referenced hash as normal keys and values. If you are uncertain about the properties being returned by Perl for a specific window, the windowse utility ( is often helpful. Now that we’ve seen how to determine various window properties, wouldn’t it be spiffy if we could make changes to some of these properties? For instance, it might be useful to change the title of a particular window. With this capability, we could create scripts that used the window title as a status indicator: "Prestidigitation In Progress ... 32% complete" Making this change to a window is as easy as a single function call: Win32::Setupsup::SetWindowText($handle,$text); We can also set the rect property we just saw. This code makes the specified window jump to the position we’ve specified: use Win32::Setupsup; my %info; $info{rect}{left} = 0; $info{rect}{right} = 600; $info{rect}{top} = 10; $info{rect}{bottom} = 500; my $whandle = unpack 'U', pack 'U', $ARGV[0]; Win32::Setupsup::SetWindowProperties( $whandle, \%info ); I’ve saved the most impressive function for last. With SendKeys(), it is possible to send arbitrary keystrokes to any window on the desktop. For example: use Win32::Setupsup; my $texttosend = "\\DN\\Low in the gums"; my $whandle = unpack 'U', pack 'U', $ARGV[0]; Win32::Setupsup::SendKeys( $whandle, $texttosend, 0 ,0 ); This will send a “down cursor key” followed by some text to the specified window. The arguments to SendKeys() are pretty simple: window handle, text to send, a flag to determine whether a window should be activated for each keystroke, and an optional 108 | Chapter 4: User Activity time between keystrokes. Special key codes like the down cursor are surrounded by backslashes. The list of available keycodes can be found in the module’s documentation. Before we move on to another tremendously useful way to work with user processes in the Windows universe, I want to briefly look at a module that shares some functionality with Win32::Setupsup but can do even more interesting stuff. Like Win32::Setupsup, Win32::GuiTest can return information about active windows and send keystrokes to applications. However, it offers even more powerful functionality. Here’s an example slightly modified from the documentation (stripped of comments and error checking, be sure to see the original) that demonstrates some of this power: use Win32::GuiTest qw(:ALL); system("start notepad.exe"); sleep 1; MenuSelect("F&ormat|&Font"); sleep(1); my $fontdlg = GetForegroundWindow(); my ($combo) = FindWindowLike( $fontdlg, '', 'ComboBox', 0x470 ); for ( GetComboContents($combo) ) { print "'$_'" . "\n"; } SendKeys("{ESC}%{F4}"); This code starts up notepad, asks it to open its font settings by choosing the appropriate menu item, and then reads the contents of the resulting dialog box and prints what it finds. It then sends the necessary keystrokes to dismiss the dialog box and tell notepad to quit. The end result is a list of monospaced fonts available on the system that looks something like this: 'Arial' 'Arial Black' 'Comic Sans MS' 'Courier' 'Courier New' 'Estrangelo Edessa' 'Fixedsys' 'Franklin Gothic Me 'Gautami' 'Georgia' 'Impact' 'Latha' 'Lucida Console' 'Lucida Sans Unicod 'Mangal' 'Marlett' 'Microsoft Sans Ser Process Management | 109 'Modern' 'MS Sans Serif' Let’s look at one more example (again, adapted from the module’s documentation because it offers great example code): use Win32::GuiTest qw(:ALL); system 'start notepad'; sleep 1; my $menu = GetMenu( GetForegroundWindow() ); menu_parse($menu); SendKeys("{ESC}%{F4}"); sub menu_parse { my ( $menu, $depth ) = @_; $depth ||= 0; foreach my $i ( 0 .. GetMenuItemCount($menu) - 1 ) { my %h = GetMenuItemInfo( $menu, $i ); print ' ' x $depth; print "$i "; print $h{text} if $h{type} and $h{type} eq 'string'; print "------" if $h{type} and $h{type} eq 'separator'; print "UNKNOWN" if not $h{type}; print "\n"; my $submenu = GetSubMenu( $menu, $i ); if ($submenu) { menu_parse( $submenu, $depth + 1 ); } } } As in the previous example, we begin by spinning up notepad. We can then examine the menus of the application in the foreground window, determining the number of top-level menu items and then iterating over each item (printing the information and looking for submenus of each item as we go). If we find a submenu, we recursively call menu_parse() to examine it. Once we’ve completed the menu walk, we send the keys to close the notepad window and quit the application. The output looks like this: 0 &File 0 &New Ctrl+N 1 &Open... Ctrl+O 2 &Save Ctrl+S 3 Save &As... 4 ------ 5 Page Set&up... 6 &Print... Ctrl+P 7 ------ 8 E&xit 110 | Chapter 4: User Activity 1 &Edit 0 &Undo Ctrl+Z 1 ------ 2 Cu&t Ctrl+X 3 &Copy Ctrl+C 4 &Paste Ctrl+V 5 De&lete Del 6 ------ 7 &Find... Ctrl+F 8 Find &Next F3 9 &Replace... Ctrl+H 10 &Go To... Ctrl+G 11 ------ 12 Select &All Ctrl+A 13 Time/&Date F5 2 F&ormat 0 &Word Wrap 1 &Font... 3 &View 0 &Status Bar 4 &Help 0 &Help Topics 1 ------ 2 &About Notepad Triggering known menu items from a script is pretty cool, but it’s even cooler to have the power to determine which menu items are available. This lets us write much more adaptable scripts. We’ve only touched on a few of Win32::GuiTest’s advanced features here. Some of the other impressive features include the ability to read the text context of a window using WMGetText() and the ability to select individual tabs in a window with SelectTabItem(). See the documentation and the example directory (eg) for more details. With the help of these two modules, we’ve taken process control to an entirely new level. Now it is possible to remotely control applications (and parts of the OS) without the explicit cooperation of those applications. We don’t need them to offer command- line support or a special API; we have the ability to essentially script a GUI, which is useful in a myriad of system administration contexts. Using Windows Management Instrumentation (WMI) Let’s look at one final approach to Windows process control before we switch to an- other operating system. By now you’ve probably figured out that each of these ap- proaches is not only good for process control, but also can be applied in many different ways to make Windows system administration easier. If you had to pick the approach that would yield the most reward in the long term to learn, WMI-based scripting is probably it. The first edition of this book called Windows Management Instrumenta- tion “Futureland” because it was still new to the scene when the book was being written. In the intervening time, Microsoft, to its credit, has embraced the WMI framework as Process Management | 111 its primary interface for administration of not just its operating systems, but also its other products, such as MS SQL Server and Microsoft Exchange. Unfortunately, WMI is one of those not-for-the-faint-of-heart technologies that gets very complex very quickly. It is based on an object-oriented model that has the power to represent not only data, but also relationships between objects. For instance, it is possible to create an association between a web server and the storage device that holds the data for that server, so that if the storage device fails, a problem for the web server will be reported as well. We don’t have the space to deal with this complexity here, so we’re just going to skim the surface of WMI by providing a small and simple intro- duction, followed by a few code samples. If you want to get a deeper look at this technology, I recommend searching for WMI- related content at You should also have a look at the infor- mation found at the Distributed Management Task Force’s website (http://www.dmtf .org). In the meantime, here is a brief synopsis to get you started. WMI is the Microsoft implementation and extension of an unfortunately named ini- tiative called the Web-Based Enterprise Management initiative, or WBEM for short. Though the name conjures up visions of something that requires a browser, it has virtually nothing to do with the World Wide Web. The companies that were part of the Distributed Management Task Force (DMTF) wanted to create something that could make it easier to perform management tasks using browsers. Putting the name aside, it is clearer to say that WBEM defines a data model for management and instru- mentation information. It provides specifications for organizing, accessing, and moving this data around. WBEM is also meant to offer a cohesive frontend for accessing data provided by other management protocols, such as the Simple Network Management Protocol (SNMP), discussed in Chapter 12, and the Common Management Informa- tion Protocol (CMIP). Data in the WBEM world is organized using the Common Information Model (CIM). CIM is the source of the power and complexity in WBEM/WMI. It provides an exten- sible data model that contains objects and object classes for any physical or logical entity one might want to manage. For instance, there are object classes for entire net- works, and objects for single slots in specific machines. There are objects for hardware settings and objects for software application settings. On top of this, CIM allows us to define object classes that describe relationships between other objects. This data model is documented in two parts: the CIM Specification and the CIM Schema. The former describes the how of CIM (how the data will be specified, its con- nection to prior management standards, etc.), while the latter provides the what of CIM (the actual objects). This division may remind you of the SNMP SMI and MIB rela- tionship (see Appendix G and Chapter 12). In practice, you’ll be consulting the CIM Schema more than the CIM Specification once you get the hang of how the data is represented. The schema format (called MOF, for Managed Object Format) is fairly easy to read. 112 | Chapter 4: User Activity The CIM Schema has two layers: • The core model for objects and classes useful in all types of WBEM interaction. • The common model for generic objects that are vendor- and operating system- independent. Within the common model there are currently 15 specific areas, including Systems, Devices, Applications, Networks, and Physical. Built on top of these two layers can be any number of extension schemas that define objects and classes for vendor- and OS-specific information. WMI is one WBEM im- plementation that makes heavy use of this extension mechanism. A crucial part of WMI that distinguishes it from generic WBEM implementations is the Win32 Schema, an extension schema for Win32-specific information built on the core and common models. WMI also adds to the generic WBEM framework by providing Win32-specific access mechanisms to the CIM data.† Using this schema extension and set of data access methods, we can explore how to perform process control operations using WMI in Perl. WMI offers two different approaches for getting at management data: object-oriented and query-based. With the former you specify the specific object or container of objects that contains the information you seek, while with the latter you construct a SQL- like‡ query that returns a result set of objects containing your desired data. We’ll give a simple example of each approach so you can see how they work. The Perl code that follows does not appear to be particularly complex, so you may wonder about the earlier “gets very complex very quickly” description. The code looks simple because: • We’re only scratching the surface of WMI. We’re not even going to touch on sub- jects like associations (i.e., relationships between objects and object classes). • The management operations we are performing are simple. Process control in this context will consist of querying the running processes and being able to terminate them at will. These operations are easy in WMI using the Win32 Schema extension. • Our samples hide the complexity of translating WMI documentation and code samples in VBScript/JScript to Perl code. See Appendix F for some help with that task. • Our samples hide the opaqueness of the debugging process. When WMI-related Perl code fails (especially code of the object-oriented flavor), it provides very little information that would help you debug the problem. You may receive error mes- sages, but they never say ERROR: YOUR EXACT PROBLEM IS.... You’re more likely to † As much as Microsoft would like to see these data access mechanisms become ubiquitous, the likelihood of finding them in a non-Win32 environment is slight. This is why I refer to them as “Win32-specific.” ‡ Microsoft provides WQL, a scaled-down query language based on SQL syntax, for this purpose. Once upon a time it also provided ODBC-based access to the data, but that approach has been deprecated in more recent OS releases. Process Management | 113 get back a message like wbemErrFailed 0x8004100 or just an empty data structure. To be fair to Perl, most of this opaqueness comes from Perl’s role in this process: it is acting as a frontend to a set of fairly complex multilayered operations that don’t concern themselves with passing back useful feedback when something fails. I know this sounds pretty grim, so let me offer some potentially helpful advice before we actually get into the code itself: • Look at all of the Win32::OLE sample code you can lay your hands on. The Active- State Win32-Users mailing list archive found at Mail is a good source for this code. If you compare this sample code to equivalent VBScript examples, you’ll start to understand the necessary translation idioms. Appendix F and the section “Active Directory Service Interfaces” on page 354 in Chapter 9 may also help. • Make friends with the Perl debugger, and use it to try out code snippets as part of this learning process. There are also several REPL*-modules available on CPAN, such as App::REPL, Devel::REPL, and Shell::Perl, that can make interactive pro- totyping easier. Other integrated development environment (IDE) tools may also offer this functionality. • Keep a copy of the WMI SDK handy. The documentation and the VBScript code examples are very helpful. • Use the WMI object browser in the WMI SDK frequently. It helps you get the lay of the land. Now let’s get to the Perl part of this section. Our initial task will be to determine what information we can retrieve about Windows processes and how we can interact with that information. First we need to establish a connection to a WMI namespace. A namespace is defined in the WMI SDK as “a unit for grouping classes and instances to control their scope and visibility.” In this case, we’re interested in connecting to the root of the standard cimv2 namespace, which contains all of the data that is interesting to us. We will also have to set up a connection with the appropriate security privileges and impersonation level. Our program will need to be given the privilege to debug a process and to impersonate us; in other words, it has to run as the user calling the script. After we get this connection, we will retrieve a Win32_Process object (as defined in the Win32 Schema). * REPL stands for Read-Eval-Print Loop, a term from the LISP (LISt Processing) world. A REPL lets you type code into a prompt, have it be executed by the language’s interpreter, and then review the results. 114 | Chapter 4: User Activity There is a hard way and an easy way to create this connection and get the object. We’ll look at both in the first example, so you get an idea of what the methods entail. Here’s the hard way, with its explanation to follow: use Win32::OLE('in'); my $server = ''; # connect to local machine # get an SWbemLocator object my $lobj = Win32::OLE->new('WbemScripting.SWbemLocator') or die "can't create locator object: ".Win32::OLE->LastError()."\n"; # set the impersonation level to "impersonate" $lobj->{Security_}->{impersonationlevel} = 3; # use it to get an SWbemServices object my $sobj = $lobj->ConnectServer($server, 'root\cimv2') or die "can't create server object: ".Win32::OLE->LastError()."\n"; # get the schema object my $procschm = $sobj->Get('Win32_Process'); The hard way involves: • Getting a locator object, used to find a connection to a server object • Setting the impersonation level so our program will run with our privileges • Using the locator object to get a server connection to the cimv2 WMI namespace • Using this server connection to retrieve a Win32_Process object Doing it this way is useful in cases where you need to operate on the intermediate objects. However, we can do this all in one step using a COM moniker’s display name. According to the WMI SDK, “in Common Object Model (COM), a moniker is the standard mechanism for encapsulating the location and binding of another COM ob- ject. The textual representation of a moniker is called a display name.” Here’s an easy way to do the same thing as the previous code snippet: use Win32::OLE('in'); my $procschm = Win32::OLE->GetObject( 'winmgmts:{impersonationLevel=impersonate}!Win32_Process') or die "can't create server object: ".Win32::OLE->LastError()."\n"; Now that we have a Win32_Process object in hand, we can use it to show us the relevant parts of the schema that represent processes under Windows. This includes all of the available Win32_Process properties and methods we can use. The code to do this is fairly simple; the only magic is the use of the Win32::OLE in operator. To explain this, we need a quick digression. Process Management | 115 Our $procschm object has two special properties, Properties_ and Methods_. Each holds a special child object, known as a collection object in COM parlance. A collection object is just a parent container for other objects; in this case, they are holding the schema’s property method description objects. The in operator just returns an array with refer- ences to each child object of a container object.† Once we have this array, we can iterate through it, returning the Name property of each child object as we go. Here’s what the code looks like: use Win32::OLE('in'); # connect to namespace, set the impersonation level, and retrieve the # Win32_process object just by using a display name my $procschm = Win32::OLE->GetObject( 'winmgmts:{impersonationLevel=impersonate}!Win32_Process') or die "can't create server object: ".Win32::OLE->LastError()."\n"; print "--- Properties ---\n"; print join("\n",map {$_->{Name}}(in $procschm->{Properties_})); print "\n--- Methods ---\n"; print join("\n",map {$_->{Name}}(in $procschm->{Methods_})); The output (on a Windows XP SP2 machine) looks like this: --- Properties --- Caption CommandLine CreationClassName CreationDate CSCreationClassName CSName Description ExecutablePath ExecutionState Handle HandleCount InstallDate KernelModeTime MaximumWorkingSetSize MinimumWorkingSetSize Name OSCreationClassName OSName OtherOperationCount OtherTransferCount PageFaults PageFileUsage ParentProcessId PeakPageFileUsage PeakVirtualSize PeakWorkingSetSize Priority PrivatePageCount † See the section “Active Directory Service Interfaces” on page 354 for details on another prominent use of in. 116 | Chapter 4: User Activity ProcessId QuotaNonPagedPoolUsage QuotaPagedPoolUsage QuotaPeakNonPagedPoolUsage QuotaPeakPagedPoolUsage ReadOperationCount ReadTransferCount SessionId Status TerminationDate ThreadCount UserModeTime VirtualSize WindowsVersion WorkingSetSize WriteOperationCount WriteTransferCount --- Methods --- Create Terminate GetOwner GetOwnerSid SetPriority AttachDebugger Now let’s get down to the business at hand. To retrieve a list of running processes, we need to ask for all instances of Win32_Process objects: use Win32::OLE('in'); # perform all of the initial steps in one swell foop my $sobj = Win32::OLE->GetObject( 'winmgmts:{impersonationLevel=impersonate}') or die "can't create server object: ".Win32::OLE->LastError()."\n"; foreach my $process (in $sobj->InstancesOf("Win32_Process")){ print $process->{Name}." is pid #".$process->{ProcessId},"\n"; } Our initial display name did not include a path to a specific object (i.e., we left off !Win32_Process). As a result, we receive a server connection object. When we call the InstancesOf() method, it returns a collection object that holds all of the instances of that particular object. Our code visits each object in turn and prints its Name and ProcessId properties. This yields a list of all the running processes. If we wanted to be a little less beneficent when iterating over each process, we could instead use one of the methods listed earlier: foreach $process (in $sobj->InstancesOf("Win32_Process")){ $process->Terminate(1); } This will terminate every process running. I do not recommend that you run this code as is; customize it for your specific needs by making it more selective. Process Management | 117 One last note before we move on. Earlier in this section I mentioned that there are two ways to query information using WMI: the object-oriented and query-based ap- proaches. Up to now we’ve been looking at the fairly straightforward object-oriented approach. Here’s a small sample using the query-based approach, just to pique your interest. First, let’s recreate the output from the preceding sample. The highlighted line is the key change here, because it uses WQL instead of InstancesOf() to retrieve all of the process objects: use Win32::OLE('in'); my $sobj = Win32::OLE->GetObject('winmgmts:{impersonationLevel=impersonate}') or die 'can't create server object: ' . Win32::OLE->LastError() . "\n"; my $query = $sobj->ExecQuery('SELECT Name, ProcessId FROM Win32_Process'); foreach my $process ( in $query ) { print $process->{Name} . ' is pid #' . $process->{ProcessId}, "\n"; } Now we can start throwing in SQL-like syntax in the highlighted query string. For example, if we only wanted to see the process IDs of the svchost.exe processes running on the system, we could write: use Win32::OLE('in'); my $sobj = Win32::OLE->GetObject('winmgmts:{impersonationLevel=impersonate}') or die "can't create server object: " . Win32::OLE->LastError() . "\n"; my $query = $sobj->ExecQuery( 'SELECT ProcessId FROM Win32_Process WHERE Name = "svchost.exe"'); print "SvcHost processes: " . join( ' ', map { $_->{ProcessId} } ( in $query) ), "\n"; WQL can handle queries with other SQL-like stanzas. For example, the following is valid WQL to retrieve information on all running processes that have names that begin with “svc”: SELECT * from Win32_Process WHERE Name LIKE "svc%" If you are SQL-literate (even if the sum of your knowledge comes from Appendix D in this book), this may be a direction you want to explore. Now you have the knowledge necessary to begin using WMI for process control. WMI has Win32 extensions for many other parts of the operating system, including the reg- istry and the event log facility. This is as far as we’re going to delve into process control on Windows. Now let’s turn our attention to another major operating system. 118 | Chapter 4: User Activity Unix Process Control Strategies for Unix process control offer another multiple-choice situation. Luckily, these choices aren’t nearly as complex as those that Windows offers. When we speak of process control under Unix, we’re referring to three operations: 1. Enumerating the list of running processes on a machine 2. Changing their priorities or process groups 3. Terminating the processes For the final two of these operations, there are Perl functions to do the job: setpriority(), setpgrp(), and kill(). The first one offers us a few options. To list running processes, you can: • Call an external program like ps. • Take a crack at deciphering /dev/kmem. • Look through the /proc filesystem (for Unix versions that have one). • Use the Proc::ProcessTable module. Let’s discuss each of these approaches. For the impatient reader, I’ll reveal right now that Proc::ProcessTable is my preferred technique. You may want to just skip directly to the discussion of that module, but I recommend reading about the other techniques anyway, since they may come in handy in the future. Calling an external program Common to all modern Unix variants is a program called ps, used to list running pro- cesses. However, ps is found in different places in the filesystem on different Unix variants, and the command-line switches it takes are also not consistent across variants. Therein lies one problem with this option: it lacks portability. An even more annoying problem is the difficulty in parsing the output (which also varies from variant to variant). Here’s a snippet of output from ps on an ancient SunOS machine: USER PID %CPU %MEM SZ RSS TT STAT START TIME COMMAND dnb 385 0.0 0.0 268 0 p4 IW Jul 2 0:00 /bin/zsh dnb 24103 0.0 2.610504 1092 p3 S Aug 10 35:49 emacs dnb 389 0.0 2.5 3604 1044 p4 S Jul 2 60:16 emacs remy 15396 0.0 0.0 252 0 p9 IW Jul 7 0:01 -zsh (zsh) sys 393 0.0 0.0 28 0 ? IW Jul 2 0:02 in.identd dnb 29488 0.0 0.0 68 0 p5 IW 20:15 0:00 screen dnb 29544 0.0 0.4 24 148 p7 R 20:39 0:00 less dnb 5707 0.0 0.0 260 0 p6 IW Jul 24 0:00 -zsh (zsh) root 28766 0.0 0.0 244 0 ? IW 13:20 0:00 -:0 (xdm) Notice the third line. Two of the columns have run together, making parsing this output an annoying task. It’s not impossible, just vexing. Some Unix variants are kinder than Process Management | 119 others in this regard (for example, later operating systems from Sun don’t have this problem), but it is something you may have to take into account. The Perl code required for this option is straightforward: use open() to run ps, while(){...} to read the output, and split(), unpack(), or substr() to parse it. You can find a recipe for this in the Perl Cookbook, by Tom Christiansen and Nathan Torkington (O’Reilly). Examining the kernel process structures I only mention this option for completeness’s sake. It is possible to write code that opens up a device like /dev/kmem and accesses the current running kernel’s memory structures. With this access, you can track down the current process table in memory and read it. However, given the pain involved (taking apart complex binary structures by hand), and its extreme nonportability (a version difference within the same operating system is likely to break your program), I’d strongly recommend against using this option.‡ If you decide not to heed this advice, you should begin by memorizing the Perl docu- mentation for pack(), unpack(), and the header files for your kernel. Open the kernel memory file (often /dev/kmem), then read() and unpack() to your heart’s content. You may find it instructive to look at the source for programs like top ( pub/top) that perform this task using a great deal of C code. Our next option offers a slightly better version of this method. Using the /proc filesystem One of the more interesting additions to Unix found in most of the current variants is the /proc filesystem. This is a magical filesystem that has nothing to do with data stor- age. Instead, it provides a file-based interface for the running process table of a machine. A “directory” named after the process ID appears in this filesystem for each running process. In this directory are a set of “files” that provide information about that process. One of these files can be written to, thus allowing control of the process. It’s a really clever concept, and that’s the good news. The bad news is that each Unix vendor/developer team decided to take this clever concept and run with it in a different direction. As a result, the files found in a /proc directory are often variant-specific, both in name and format. For a description of which files are available and what they contain, you will need to consult the manual pages (usually found in sections 4, 5, or 8) for procfs or mount_ procfs on your system. The one fairly portable use of the /proc filesystem is the enumeration of running pro- cesses. If we want to list just the process IDs and their owners, we can use Perl’s direc- tory and lstat() operators: ‡ Later, we’ll look at a module called Proc::ProcessTable that can do this for you without you having to write the code. 120 | Chapter 4: User Activity opendir my $PROC, '/proc' or die "Unable to open /proc:$!\n"; # only stat the items in /proc that look like PIDs for my $process (grep /^\d+$/, readdir($PROC)){ print "$process\t". getpwuid((lstat "/proc/$process")[4])."\n"; } closedir $PROC; If you are interested in more information about a process, you will have to open and unpack() the appropriate binary file in the /proc directories. Common names for this file are status and psinfo. The manual pages cited a moment ago should provide details about the C structure found in this file, or at least a pointer to a C include file that documents this structure. Because these are operating system-specific (and OS version-specific) formats, you’re still going to run into the problem of program fragility mentioned in the discussion of the previous option. You may be feeling discouraged at this point because all of our options so far look like they require code with lots of special cases (one for each version of each operating system we wish to support). Luckily, we have one more option up our sleeve that may help in this regard. Using the Proc::ProcessTable module Daniel J. Urist (with the help of some volunteers) has been kind enough to write a module called Proc::ProcessTable that offers a consistent interface to the process table for the major Unix variants. It hides the vagaries of the different /proc or kmem imple- mentations for you, allowing you to write relatively portable code. Simply load the module, create a Proc::ProcessTable::Process object, and run meth- ods from that object: use Proc::ProcessTable; my $tobj = new Proc::ProcessTable; This object uses Perl’s tied variable functionality to present a real-time view of the system. You do not need to call a special function to refresh the object; each time you access it, it re-reads the process table. To get at this information, you call the object method table(): my $proctable = $tobj->table( ); table() returns a reference to an array with members that are references to individual process objects. Each of these objects has its own set of methods that returns informa- tion about that process. For instance, here’s how you would get a listing of the process IDs and owners: use Proc::ProcessTable; my $tobj = new Proc::ProcessTable; Process Management | 121 my $proctable = $tobj->table(); foreach my $process (@$proctable) { print $process->pid . "\t" . getpwuid( $process->uid ) . "\n"; } If you want to know which process methods are available on your Unix variant, the fields() method of your Proc::ProcessTable object ($tobj in the preceding code) will return a list for you. Proc::ProcessTable also adds three other methods to each process object—kill(), priority(), and pgrp()—which are just frontends to the built-in Perl function we men- tioned at the beginning of this section. To bring us back to the big picture, let’s look at some of the uses of these process control techniques. We started to examine process control in the context of user actions, so let’s look at a few teeny scripts that focus on these actions. We will use the Proc::ProcessTable module on Unix for these examples, but these ideas are not oper- ating system-specific. The first example is slightly modified from the documentation for Proc::ProcessTable: use Proc::ProcessTable; my $t = new Proc::ProcessTable; foreach my $p (@{$t->table}){ if ($p->pctmem > 95){ $p->kill(9); } } When run on the Unix variants that provide the pctmem() method (most do), this code will shoot down any process consuming 95% of the machine’s memory. As it stands, it’s probably too ruthless to be used in real life. It would be much more reasonable to add something like this before the kill() command: print 'about to nuke '.$p->pid."\t". getpwuid($p->uid)."\n"; print 'proceed? (yes/no) '; chomp($ans = <>); next unless ($ans eq 'yes'); There’s a bit of a race condition here: it is possible that the system state will change during the delay induced by prompting the user. Given that we are only prompting for huge processes, though, and huge processes are those least likely to change state in a short amount of time, we’re probably fine coding this way. If you wanted to be pedantic, you would probably collect the list of processes to be killed first, prompt for input, and then recheck the state of the process table before actually killing the desired processes. This doesn’t remove the race condition, but it does make it much less likely to occur. 122 | Chapter 4: User Activity There are times when death is too good for a process. Sometimes it is important to notice that a process is running while it is running so that some real-life action (like “user attitude correction”) can be taken. For example, at our site we have a policy against the use of Internet Relay Chat (IRC) bots. Bots are daemon processes that con- nect to an IRC network of chat servers and perform automated actions. Though bots can be used for constructive purposes, these days they play a mostly antisocial role on IRC. We’ve also had security breaches come to our attention because the first (and often only) thing the intruder has done is put up an IRC bot of some sort. As a result, noting their presence on our system without killing them is important to us. The most common bot by far is called eggdrop. If we wanted to look for this process name being run on our system, we could use code like this: use Proc::ProcessTable; my $logfile = 'eggdrops'; open my $LOG, '>>', $logfile or die "Can't open logfile for append:$!\n"; my $t = new Proc::ProcessTable; foreach my $p ( @{ $t->table } ) { if ( $p->fname() =~ /eggdrop/i ) { print $LOG time . "\t" . getpwuid( $p->uid ) . "\t" . $p->fname() . "\n"; } } close $LOG; If you’re thinking, “This code isn’t good enough! All someone has to do is rename the eggdrop executable to evade its check,” you’re absolutely right. We’ll take a stab at writing some less naïve bot-check code in the very last section of this chapter. In the meantime, let’s take a look at one more example where Perl assists us in managing user processes. So far all of our examples have been fairly negative, focusing on dealing with resource-hungry and naughty processes. Let’s look at something with a sunnier disposition. There are times when a system administrator needs to know which (legitimate) pro- grams users on a system are using. Sometimes this is necessary in the context of software metering, where there are legal concerns about the number of users running a program concurrently. In those cases there is usually a licensing mechanism in place to handle the bean counting. Another situation where this knowledge comes in handy is that of machine migration. If you are migrating a user population from one architecture to another, you’ll want to make sure all the programs used on the previous architecture are available on the new one. One approach to solving this problem involves replacing every non-OS binary available to users with a wrapper that first records that a particular binary has been run and then Process Management | 123 actually runs it. This can be difficult to implement if there are a large number of binaries. It also has the unpleasant side effect of slowing down every program invocation. If precision is not important and a rough estimate of which binaries are in use will suffice, we can use Proc::ProcessTable to solve this problem. Here’s some code that wakes up every five minutes and surveys the current process landscape. It keeps a simple count of all of the process names it finds, and it’s smart enough not to count processes it saw during its last period of wakefulness. Every hour it prints its findings and starts collecting again. We wait five minutes between each run because walking the process table is usually a resource-intensive operation, and we’d prefer this program to add as little load to the system as possible: use Proc::ProcessTable; my $interval = 300; # sleep interval of 5 minutes my $partofhour = 0; # keep track of where in the hour we are my $tobj = new Proc::ProcessTable; # create new process object my %last; # to keep track of info from the previous run my %current; # to keep track of data from the current run my %collection; # to keep track of info over the entire hour # forever loop, collecting stats every $interval secs # and dumping them once an hour while (1) { foreach my $process ( @{ $tobj->table } ) { # we should ignore ourselves next if ( $process->pid() == $$ ); # save this process info for our next run # (note: this assumes that your PIDs won't recycle between runs, # but on a very busy system that may not be the case) $current{ $process->pid() } = $process->fname(); # ignore this process if we saw it during the last iteration next if ( $last{ $process->pid() } eq $process->fname() ); # else, remember it $collection{ $process->fname() }++; } $partofhour += $interval; %last = %current; %current = (); if ( $partofhour >= 3600 ) { print scalar localtime(time) . ( '-' x 50 ) . "\n"; print "Name\t\tCount\n"; print "--------------\t\t-----\n"; foreach my $name ( sort reverse_value_sort keys %collection ) { print "$name\t\t$collection{$name}\n"; } %collection = (); 124 | Chapter 4: User Activity $partofhour = 0; } sleep($interval); } # (reverse) sort by values in %collection and by key name sub reverse_value_sort { return $collection{$b} <=> $collection{$a} || $a cmp $b; } There are many ways this program could be enhanced. It could track processes on a per-user basis (i.e., only recording one instance of a program launch per user), collect daily stats, present its information as a nice bar graph, and so on. It’s up to you where you might want to take it. File and Network Operations For the last section of this chapter, we’re going to lump two of the user action domains together. The processes we’ve just spent so much time controlling do more than just suck up CPU and memory resources; they also perform operations on filesystems and communicate on a network on behalf of users. User administration requires that we deal with these second-order effects as well. Our focus in this section will be fairly narrow. We’re only interested in looking at file and network operations that other users are performing on a system. We’re also only going to focus on those operations that we can track back to a specific user (or a specific process run by a specific user). With these blinders in mind, let’s go forth. Tracking File Operations on Windows If we want to track other users’ open files, the closest we can come involves using a former third-party command-line program called handle, written by Mark Russinovich (formerly of Sysinternals). See the references section at the end of this chapter for in- formation on where to get it. handle can show us all of the open handles on a particular system. Here’s an excerpt from some sample output: System pid: 4 NT AUTHORITY\SYSTEM 7C: File (-W-) C:\pagefile.sys 5DC: File (---) C:\Documents and Settings\LocalService\Local Settings\ Application Data\Microsoft\Windows\UsrClass.dat 5E0: File (---) C:\WINDOWS\system32\config\SAM.LOG 5E4: File (---) C:\Documents and Settings\LocalService\NTUSER.DAT 5E8: File (---) C:\WINDOWS\system32\config\system 5EC: File (---) C:\WINDOWS\system32\config\software.LOG 5F0: File (---) C:\WINDOWS\system32\config\software 5F8: File (---) C:\WINDOWS\system32\config\SECURITY 5FC: File (---) C:\WINDOWS\system32\config\default 600: File (---) C:\WINDOWS\system32\config\SECURITY.LOG 604: File (---) C:\WINDOWS\system32\config\default.LOG 60C: File (---) C:\WINDOWS\system32\config\SAM File and Network Operations | 125 610: File (---) C:\WINDOWS\system32\config\system.LOG 614: File (---) C:\Documents and Settings\NetworkService\NTUSER.DAT 8E0: File (---) C:\Documents and Settings\dNb\Local Settings\Application Data\Microsoft\Windows\UsrClass.dat.LOG 8E4: File (---) C:\Documents and Settings\dNb\Local Settings\Application Data\Microsoft\Windows\UsrClass.dat 8E8: File (---) C:\Documents and Settings\dNb\NTUSER.DAT.LOG 8EC: File (---) C:\Documents and Settings\dNb\NTUSER.DAT B08: File (RW-) C:\Program Files\Symantec AntiVirus\SAVRT B3C: File (R--) C:\System Volume Information\_restore{96B84597-8A49-41EE- 8303-02D3AD2B3BA4}\RP80\change.log B78: File (R--) C:\Program Files\Symantec AntiVirus\SAVRT\0608NAV~.TMP ------------------------------------------------------------------------------ smss.exe pid: 436 NT AUTHORITY\SYSTEM 8: File (RW-) C:\WINDOWS 1C: File (RW-) C:\WINDOWS\system32 You can also request information on specific files or directories: > handle.exe c:\WINDOWS\system32\config Handle v3.3 Copyright (C) 1997-2007 Mark Russinovich Sysinternals - System pid: 4 5E0: C:\WINDOWS\system32\config\SAM.LOG System pid: 4 5E8: C:\WINDOWS\system32\config\system System pid: 4 5EC: C:\WINDOWS\system32\config\software.LOG System pid: 4 5F0: C:\WINDOWS\system32\config\software System pid: 4 5F8: C:\WINDOWS\system32\config\SECURITY System pid: 4 5FC: C:\WINDOWS\system32\config\default System pid: 4 600: C:\WINDOWS\system32\config\SECURITY.LOG System pid: 4 604: C:\WINDOWS\system32\config\default.LOG System pid: 4 60C: C:\WINDOWS\system32\config\SAM System pid: 4 610: C:\WINDOWS\system32\config\system.LOG services.exe pid: 552 2A4: C:\WINDOWS\system32\config\AppEvent.Evt services.exe pid: 552 2B4: C:\WINDOWS\system32\config\Internet.evt services.exe pid: 552 2C4: C:\WINDOWS\system32\config\SecEvent.Evt services.exe pid: 552 2D4: C:\WINDOWS\system32\config\SysEvent.Evt svchost.exe pid: 848 17DC: C:\WINDOWS\system32\config\systemprofile\ Application Data\Microsoft\SystemCertificates\My ccSetMgr.exe pid: 1172 2EC: C:\WINDOWS\system32\config\systemprofile\ Application Data\Microsoft\SystemCertificates\My ccEvtMgr.exe pid: 1200 23C: C:\WINDOWS\system32\config\systemprofile\ Application Data\Microsoft\SystemCertificates\My Rtvscan.exe pid: 1560 454: C:\WINDOWS\system32\config\systemprofile\ Application Data\Microsoft\SystemCertificates\My handle can provide this information for a specific process name using the -p switch. Using this executable from Perl is straightforward, so we won’t provide any sample code. Instead, let’s look at a related and more interesting operation: auditing. Windows allows us to efficiently watch a file, directory, or hierarchy of directories for changes. You could imagine repeatedly performing stat()s on the desired object or 126 | Chapter 4: User Activity objects, but that would be highly CPU-intensive. Under Windows, we can ask the operating system to keep watch for us. There is a specialized Perl module that makes this job relatively painless for us: Win32::ChangeNotify by Christopher J. Madsen. There is also a related helper mod- ule: Win32::FileNotify by Renee Baecker. The Ephemeral Nature of Modules In the first edition of this book, this section described how to use the module Win32::AdvNotify by Amine Moulay Ramdane for filesystem auditing. It was a great module; one of several superb Windows modules by the same author, it did everything Win32::ChangeNotify could do and considerably more. Unfortunately, Ramdane was inexplicably strict about the distribution terms for his modules. He did not allow this module to be hosted on any website other than his own, and he did not want that site mirrored elsewhere. Source code was never released. According to the Wayback Machine (, by April 2002 the contents of that website had disappeared, and for all practical purposes, so had the author of all those great modules. I started getting email shortly after that date from readers of the first edition looking to follow the examples in my book using Ram- dane’s modules. All I could do was try to suggest some alternatives. I’ve removed all of the demonstration code for those modules in this edition, even though most of Ramdane’s modules can still be found on the Net if you’re willing to hunt hard enough. The total lack of support for the modules (and the lack of potential even for someone else to support them) means it is too risky to use them at this point. Grrr. Win32::ChangeNotify is pretty easy to use, but it does have one gotcha. The module uses the Win32 APIs to ask the OS to let you know if something changes in a directory. You can even specify what kind of change to look for (last write time, file or directory names/ sizes, etc.). The problem is that if you ask it to watch a directory for changes, it can tell you when something changes, but not what has changed. It’s up to the program author to determine that with some separate code. That’s where Win32::FileNotify comes in. If you just need to watch a single file, Win32::FileNotify will go the extra step of double- checking whether the change the OS reported is in the file being audited. Because they’re so small, we’ll look at examples of both modules. We’ll start with the specific case of watching to see if a file has changed: use Win32::FileNotify; my $file = 'c:\windows\temp\importantfile'; my $fnot = Win32::FileNotify->new($file); $fnot->wait(); # at this point, our program blocks until $file changes ... # go do something about the file change File and Network Operations | 127 And here’s some code to look for changes in a directory (specifically, files coming and going): use Win32::ChangeNotify; my $dir = 'c:\importantdir'; # watch this directory (second argument says don't watch for changes # to subdirectories) for changes in the filenames found there my $cnot = Win32::ChangeNotify->new( $dir, 0, 'FILE_NAME' ); while (1) { # blocks for 10 secs (10,000 milliseconds) or until a change takes place my $waitresult = $cnot->wait(10000); if ( $waitresult == 1 ) { ... # call or include some other code here to figure out what changed # reset the ChangeNotification object so we can continue monitoring $cnot->reset; } elsif ( $waitresult == 0 ) { print "no changes to $dir in the last 10 seconds\n"; } elsif ( $waitresult == −1 ) { print "something went blooey in the monitoring\n"; last; } } Tracking Network Operations on Windows That was filesystem monitoring. What about network access monitoring? There are two fairly easy ways to track network operations under Windows. Ideally, as an ad- ministrator you’d like to know which process (and therefore which user) has opened a network port. While I know of no Perl module that can perform this task, there are at least two command-line tools that provide the information in a way that could be consumed by a Perl program. The first, netstat, actually ships with the system, but very few people know it can do this (I certainly didn’t for a long time). Here’s some sample output: > netstat -ano Active Connections Proto Local Address Foreign Address State PID TCP LISTENING 932 TCP LISTENING 4 TCP LISTENING 1216 TCP LISTENING 4 UDP *:* 4 128 | Chapter 4: User Activity UDP *:* 680 UDP *:* 1068 UDP *:* 1068 UDP *:* 680 UDP *:* 1024 UDP *:* 1108 UDP *:* 1024 UDP *:* 4 UDP *:* 4 UDP *:* 1108 The second is another tool from Mark Russinovich, formerly of Sysinternals: TcpView (or more precisely, the tcpvcon utility that comes in that package). It has the nice prop- erty of being able to output the information in CSV form, like so: > tcpvcon -anc TCPView v2.51 - TCP/UDP endpoint viewer Copyright (C) 1998-2007 Mark Russinovich Sysinternals - TCP,alg.exe,1216,LISTENING,, TCP,System,4,LISTENING,, TCP,svchost.exe,932,LISTENING,, TCP,System,4,LISTENING,, UDP,svchost.exe,1024,*,,*:* UDP,lsass.exe,680,*,,*:* UDP,svchost.exe,1068,*,,*:* UDP,svchost.exe,1108,*,,*:* UDP,svchost.exe,1024,*,,*:* UDP,System,4,*,,*:* UDP,svchost.exe,1108,*,,*:* UDP,lsass.exe,680,*,,*:* UDP,System,4,*,,*:* UDP,svchost.exe,1068,*,,*:* UDP,System,4,*,,*:* This would be trivial to parse with something like Text::CSV::Simple or Text::CSV_XS. Let’s see how we’d perform the same tasks within the Unix world. Tracking File and Network Operations in Unix To handle the tracking of both file and network operations in Unix, we can use a single approach.* This is one of few times in this book where calling a separate executable is clearly the superior method. Vic Abell has given an amazing gift to the system admin- istration world by writing and maintaining a program called lsof (LiSt Open Files) that can be found at lsof can show in detail all of * This is the best approach for portability. Various OSs have their own mechanisms (inotify, dnotify, etc.), and frameworks like DTrace are very cool. Mac OS X 10.5+ has a similar auditing facility to the one we saw with Windows (Mac::FSEvents gives you easy access to it). However, none of these options is as portable as the approach described here. File and Network Operations | 129 the currently open files and network connections on a Unix machine. One of the things that makes it truly amazing is its portability. The latest version as of this writing runs on at least nine flavors of Unix (the previous version supported an even wider variety of Unix flavors) and supports several OS versions for each flavor. Here’s a snippet of lsof ’s output, showing an excerpt of the output for one of the processes I am running. lsof tends to output very long lines, so I’ve inserted a blank line between each line of output to make the distinctions clear: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME firefox-b 27189 dnb cwd VDIR 318,16168 36864 25760428 /home/dnb firefox-b 27189 dnb txt VREG 318,37181 177864 6320643 /net/csw (fileserver:/vol/systems/csw) firefox-b 27189 dnb txt VREG 136,0 56874 3680 /usr/openwin/lib/X11/fonts/Type1/outline/Helvetica-Bold.pfa firefox-b 27189 dnb txt VREG 318,37181 16524 563516 /net/csw (fileserver:/vol/systems/csw) firefox-b 27189 dnb 0u unix 105,43 0t0 3352 /devices/pseudo/tl@0:ticots->(socketpair: 0x1409) (0x300034a1010) firefox-b 27189 dnb 2u unix 105,45 0t0 3352 /devices/pseudo/tl@0:ticots->(socketpair: 0x140b) (0x300034a01d0) firefox-b 27189 dnb 4u IPv6 0x3000349cde0 0t2121076 TCP localhost:32887->localhost:6010 (ESTABLISHED) firefox-b 27189 dnb 6u FIFO 0x30003726ee8 0t0 2105883 (fifofs) ->0x30003726de0 firefox-b 27189 dnb 24r VREG 318,37181 332618 85700 /net/csw (fileserver:/vol/systems/csw) firefox-b 27189 dnb 29u unix 105,46 0t1742 3352 /devices/pseudo/tl@0:ticots->/var/tmp/orbit-dnb/linc -6a37-0-47776fee636a2 (0x30003cc1900->0x300045731f8) firefox-b 27189 dnb 31u unix 105,50 0t0 3352 /devices/pseudo/tl@0:ticots->/var/tmp/orbit-dnb/linc -6a35-0-47772fb086240 (0x300034a13a0) firefox-b 27189 dnb 43u IPv4 0x30742eb79b0 0t42210 TCP> (ESTABLISHED) This output demonstrates some of the power of this command. It shows the current working directory (VDIR), regular files (VREG), pipes (FIFO), and network connections (IPv4/IPv6) opened by this process. The easiest way to use lsof from Perl is to invoke its special “field” mode (-F). In this mode, its output is broken up into specially labeled and delimited fields, instead of the ps-like columns just shown. This makes parsing the output a cinch. 130 | Chapter 4: User Activity There is one quirk to the field mode output. It is organized into what the author calls “process sets” and “file sets.” A process set is a set of field entries referring to a single process, and a file set is a similar set for a file. This all makes more sense if we turn on field mode with the 0 option. Fields are then delimited with NUL (ASCII 0) characters, and sets with NL (ASCII 12) characters. Here’s a similar group of lines to those in the preceding output, this time in field mode (NUL is represented as ^@). I’ve added spaces between the lines again to make it easier to read: p27189^@g27155^@R27183^@cfirefox-bin^@u6070^@Ldnb^@ fcwd^@a ^@l ^@tVDIR^@N0x30001b7b1d8^@D0x13e00003f28^@s36864^@i25760428^@k90^@n/home/dnb^@ ftxt^@a ^@l ^@tVREG^@N0x3000224a0f0^@D0x13e0000913d^@s177864^@i6320 643^@k1^@n/net/csw (fileserver:/vol/systems/csw)^@ ftxt^@a ^@l ^@tVREG^@N0x30001714950^@D0x8800000000^@s35064^@i2800^@k1^@n/usr/lib/ ^@tVREG^@N0x300036226c0^@D0x8800000000^@s56874^@i3680^@k1^@n/usr/ openwin/lib/X11/fonts/Type1/outline/Helvetica-Bold.pfa^@ ftxt^@a ^@l ^@tunix^@F0x3000328c550^@C6^@G0x3;0x0^@N0x300034a1010^@D0x8800 000000^@o0t0^@i3352^@n/devices/pseudo/tl@0:ticots->(socketpair: 0x1409) (0x300034a1010)^@ f1^@au^@l ^@tDOOR^@F0x3000328cf98^@C1^@G0x2001;0x1^@N0x3000178b300^@D0x13 c00000000^@o0t0^@i54^@k27^@n/var/run (swap) (door to nscd[240])^@ f4^@au^@l ^@tIPv6^@F0x300037258f0^@C1^@G0x83;0x1^@N0x300034ace50^@d0x3000349 cde0^@o0t3919884^@PTCP^@nlocalhost:32887->localhost:6010^@TST= ESTABLISHED^@TQR=0^@TQS=8191^@TWR=49152^@TWW=13264^@ f5^@au^@l ^@tFIFO^@F0x30003724f50^@C1^@G0x3;0x0^@N0x30003726de0^@d0x30003726 de0^@o0t0^@i2105883^@n(fifofs) ->0x30003726ee8^@ f6^@au^@l ^@tFIFO^@F0x30003725420^@C1^@G0x3;0x0^@N0x30003726ee8^@d0x30003726 ee8^@o0t0^@i2105883^@n(fifofs) ->0x30003726de0^@ f7^@aw^@lW^@tVREG^@F0x30003724c40^@C1^@G0x302;0x0^@N0x30001eadbf8^ @D0x13e00003f28^@s0^@i1539532^@k1^@n/home/dnb (fileserver:/vol/homedirs/systems/dnb)^@ f8^@au^@l ^@tIPv4^@F0x30003724ce8^@C1^@G0x83;0x0^@N0x300034ac010^@d0x 300040604f0^@o0t4094^@PTCP^>web^@TST=CLOSE_WAIT^@TQR=0^@TQS=0^@TWR=49640^@TWW=6960^@ f44^@au^@l ^@tVREG^@F0x3000328c5c0^@C1^@G0x2103;0x0^@N0x300051cd3f8^@ File and Network Operations | 131 D0x13e00003f28^@s276^@i16547341^@k1^@n/home/dnb (fileserver:/vol/ homedirs/systems/dnb)^@ f45^@au^@l ^@tVREG^@F0x30003725f80^@C1^@G0x3;0x0^@N0x300026ad920^@D0x 13e00003f28^@s8468^@i21298675^@k1^@n/home/dnb (fileserver:/vol/homedirs/systems/dnb)^@ f46^@au^@l ^@tIPv4^@F0x30003724a10^@C1^@G0x83;0x0^@N0x309ab62b578^@d0x30742 eb76b0^@o0t20726^@PTCP^>216.66.26. 161:www^@TST=ESTABLISHED^@TQR=0^@TQS=0^@TWR=49640^@TWW=6432^@ f47^@au^@l ^@tVREG^@F0x3000328c080^@C1^@G0x2103;0x0^@N0x30002186098^@D0x 13e00003f28^@s66560^@i16547342^@k1^@n/home/dnb (fileserver:/vol/ homedirs/systems/dnb)^@ f48^@au^@l Let’s deconstruct this output. The first line is a process set (we can tell because it begins with the letter p): p27189^@g27155^@R27183^@cfirefox-bin^@u6070^@Ldnb^@ fcwd^@a ^@l Each field begins with a letter identifying the field’s contents (p for pid, c for command, u for uid, and L for login) and ends with a delimiter character. Together the fields on this line make up a process set. All of the lines that follow, up until the next process set, describe the open files/network connections of the process described by this process set. Let’s put this mode to use. If we wanted to show all of the open files on a system and the PIDs that are using them, we could use code like this:† use Text::Wrap; my $lsofexec = '/usr/local/bin/lsof'; # location of lsof executable # (F)ield mode, NUL (0) delim, show (L)ogin, file (t)ype and file (n)ame my $lsofflag = '-FL0tn'; open my $LSOFPIPE, '-|', "$lsofexec $lsofflag" or die "Unable to start $lsofexec: $!\n"; my $pid; # pid as returned by lsof my $pathname; # pathname as returned by lsof my $login; # login name as returned by lsof my $type; # type of open file as returned by lsof my %seen; # for a pathname cache my %paths; # collect the paths as we go † If you don’t want to parse lsof’s field mode by hand Marc Beyer’s Unix::Lsof will handle the work for you. 132 | Chapter 4: User Activity while ( my $lsof = <$LSOFPIPE> ) { # deal with a process set if ( substr( $lsof, 0, 1 ) eq 'p' ) { ( $pid, $login ) = split( /\0/, $lsof ); $pid = substr( $pid, 1, length($pid) ); } # deal with a file set; note: we are only interested # in "regular" files (as per Solaris and Linux, lsof on other # systems may mark files and directories differently) if ( substr( $lsof, 0, 5 ) eq 'tVREG' or # Solaris substr( $lsof, 0, 4 ) eq 'tREG') { # Linux ( $type, $pathname ) = split( /\0/, $lsof ); # a process may have the same pathname open twice; # these two lines make sure we only record it once next if ( $seen{$pathname} eq $pid ); $seen{$pathname} = $pid; $pathname = substr( $pathname, 1, length($pathname) ); push( @{ $paths{$pathname} }, $pid ); } } close $LSOFPIPE; foreach my $path ( sort keys %paths ) { print "$path:\n"; print wrap( "\t", "\t", join( " ", @{ $paths{$path} } ) ), "\n"; } This code instructs lsof to show only a few of its possible fields. We iterate through its output, collecting filenames and PIDs in a hash of lists. When we’ve received all of the output, we print the filenames in a nicely formatted PID list (thanks to David Muir Sharnoff’s Text::Wrap module): /home/dnb (fileserver:/vol/homedirs/systems/dnb): 12777 12933 27293 28223 /usr/lib/ 10613 12777 12933 27217 27219 27293 28147 28149 28223 28352 28353 28361 /usr/lib/ 27217 28147 28352 28353 28361 /usr/lib/ 10613 12777 12933 27217 27219 27293 28147 28149 28223 28352 28353 28361 /usr/lib/ 10613 27217 28147 28352 28353 28361 /usr/lib/ 10613 27217 27219 28147 28149 28352 28353 28361 /usr/lib/ 10613 27217 27219 28147 28149 28352 28353 28361 File and Network Operations | 133 /usr/lib/ 10613 27217 27219 28147 28149 28352 28353 28361 /usr/lib/sparcv9/ 28362 28365 /usr/lib/sparcv9/ 28362 28365 /usr/platform/sun4u-us3/lib/ 10613 12777 12933 27217 27219 27293 28147 28149 28223 28352 28353 28361 /usr/platform/sun4u-us3/lib/sparcv9/ 28362 28365 ... For our last example of tracking Unix file and network operations, let’s return to an earlier example, where we attempted to find IRC bots running on a system. There are more reliable ways to find network daemons like bots than looking at the process table. A user may be able to hide the name of a bot by renaming the executable, but he’ll have to work a lot harder to hide the open network connection. More often than not, this connection is to a server running on TCP ports 6660–7000. lsof makes looking for these processes easy: my $lsofexec = '/usr/local/bin/lsof'; # location of lsof executable my $lsofflag = '-FL0c -iTCP:6660-7000'; # specify ports and other lsof flags # This is a hash slice being used to preload a hash table, the # existence of whose keys we'll check later. Usually this gets written # like this: # %approvedclients = ('ircII' => undef, 'xirc' => undef, ...); # (but this is a cool idiom popularized by Mark-Jason Dominus) my %approvedclients; @approvedclients{ 'ircII', 'xirc', 'pirc' } = (); open my $LSOFPIPE, "$lsofexec $lsofflag|" or die "Unable to start $lsofexec:$!\n"; my $pid; my $command; my $login; while ( my $lsof = <$LSOFPIPE> ) { ( $pid, $command, $login ) = $lsof =~ /p(\d+)\000 c(.+)\000 L(\w+)\000/x; warn "$login using an unapproved client called $command (pid $pid)!\n" unless ( exists $approvedclients{$command} ); } close $LSOFPIPE; This is the simplest check we can make. It will catch users who rename eggdrop to something like pine or -tcsh, as well as those users who don’t even attempt to hide their bots. However, it suffers from a similar flaw to our other approach. If a user is smart 134 | Chapter 4: User Activity enough, she may rename her bot to something on our “approved clients” list. To con- tinue our hunt, we could take at least two more steps: • Use lsof to check that the file opened for that executable really is the file we expect it to be, and not some random binary in a user filesystem. • Use our process control methods to check that the user is running this program from an existing shell. If this is the only process running for a user (i.e., if the user has logged off but left it running), it is probably a daemon and hence a bot. This cat-and-mouse game brings us to a point that will help wrap up the chapter. In Chapter 3, we mentioned that users are fundamentally unpredictable. They do things system administrators don’t anticipate. There is an old saying: “Nothing is foolproof because fools are so ingenious.” It is important to come to grips with this fact as you program Perl for user administration. You’ll write more robust programs as a result, and when one of your programs goes “blooey” because a user did something unexpec- ted, you’ll be able to sit back calmly and admire the ingenuity. Module Information for This Chapter Module CPAN ID Version Text::CSV_XS HMBRAND 0.32 Win32::Process::Info WYANT 1.011 Win32::Setupsup JHELBERG Win32::GuiTest KARASIC 1.54 Win32::OLE (ships with ActiveState Perl) JDB 0.1703 Proc::ProcessTable DURIST 0.41 Data::Dumper (ships with Perl) GSAR 2.121 Win32::ChangeNotify JDB 1.05 Win32::FileNotify RENEEB 0.1 Text::Wrap (ships with Perl) MUIR 2006.1117 Installing Win32::Setupsup If you want to install Win32::Setupsup, you’ll need to get it from a different PPM repo- sitory than the default one configured when you first installed ActiveState Perl. It can be found (as of this writing) in the very handy supplementary repository maintained by Randy Kobes at the University of Winnipeg. I’d recommend adding this repository even if you don’t plan to use Win32::Setupsup. The easiest way to do this is from the command line, like so: $ ppm repo add uwinnipeg Module Information for This Chapter | 135 or, if using Perl 5.10: $ ppm repo add uwinnipeg You can also add it to the GUI version of PPM4 by choosing Preferences in the Edit menu and selecting the Repositories tab. More info about this repository can be found at References for More Information hosts the Perl-Win32-Admin and Perl-Win32- Users mailing lists. Both lists and their archives are invaluable resources for Win32 programmers. is the current home for WMI at This address has changed a few times since the first edition, so doing a web search for “WMI” may be a better way to locate the WMI URL du jour at Microsoft. is the home (as of this writing) of the handle program and many other valuable Windows utilities that Microsoft acquired when it bought Sysinternals and hired its principals. still exists as of this writing and redirects to the correct Microsoft URL. If you can’t find these utilities in any of Microsoft’s websites, perhaps going to that URL will point you at the current location. is the home of the Distributed Management Task Force and a good source for WBEM information. If you haven’t yet, you must download the Microsoft Scriptomatic tool (version 2 as of this writing) from This Windows tool from “the Microsoft Scripting Guys” lets you poke around the WMI namespaces on your machine. When you find something you might be interested in using, it can write a script to use it for you. Really. But even better than that, it can write the script for you in VBScript, JScript, Perl, or Python. I’m raving about this tool both here and in the other chapters that mention WMI because I like it so much. If you want to use it under Vista, though, be sure to read the section on Vista in Chapter 1. 136 | Chapter 4: User Activity CHAPTER 5 TCP/IP Name and Configuration Services The majority of the conversations between computers these days take place using the Transmission Control Protocol running over a lower layer called the Internet Protocol.* These two protocols are commonly lumped together into the acronym TCP/ IP. Every machine that participates in a TCP/IP network must be assigned at least one unique numeric identifier, called an IP address. IP addresses are usually written using the form N.N.N.N (e.g., While machines are content to address each other using strings of dot-separated num- bers, most people are less enamored of this idea. TCP/IP would have fallen flat on its face as a protocol if users had to remember unique 12-digit sequences for every machine they wanted to contact. Mechanisms had to be invented to manage and distribute IP addresses to human-friendly name mappings. Also needed was a way to let a machine automatically determine its own TCP/IP configuration (i.e., IP address) without requiring a human to drop by and type in the information by hand. This chapter describes the evolution of the network name services that allow us to access data at instead of at, and what takes place behind the scenes. We’ll also look at the most prevalent configuration service that allows a machine to retrieve its TCP/IP configuration information from a central server. Along the way we’ll combine a dash of history with a healthy serving of practical advice on how Perl can help us manage these crucial parts of any networking infrastructure. Host Files The first approach used to solve the problem of mapping IP addresses to names was the most obvious and simple one: creating a standard file to hold a table of IP addresses * This chapter will be discussing IPv4, the current (deployed) standard. IPv6 (the next generation of IP) will potentially replace it in due course. 137 and their corresponding computer names. This file exists as /etc/hosts on Unix and OS X systems and %SystemRoot%\System32\Drivers\Etc\hosts on machines running Windows-based operating systems. Here’s an example Unix-style host file: localhost everest rivendell The limitations of this approach become clear very quickly. If’s network man- ager has two machines on a TCP/IP network that communicate with each other, and she wants to add a third, she has to edit the correct file on all of her machines. If then buys yet another machine, there will be four separate host files to be maintained (one on each machine). As untenable as this may seem, this is what actually happened during the early days of the Internet/ARPAnet. As new sites were connected, every site on the net that wished to talk with the new site needed to update its host files. The central host repository, known as the Network Information Center (NIC)—or more precisely, the SRI-NIC, since it was housed at the Stanford Research Institute at the time—updated and pub- lished a host file for the entire network called HOSTS.TXT. To remain up-to-date, system administrators anonymously FTP’d this file from SRI-NIC’s NETINFO direc- tory on a regular basis. Host files are still in use today, despite their limitations and the availability of the re- placements we’ll be talking about later in this chapter. On a small network, having an up-to-date host file that includes all of the hosts on that network is useful. It doesn’t even have to reside on each machine in the network to be helpful (since the other mechanisms we’ll describe later do a much better job of distributing this information). Just having one around to consult is handy for quick manual lookups and address allocation purposes. Strangely enough, host files have made a bit of a comeback in recent years. They provide an easy way to override other network name services, which is useful in cases where you want to prevent connections to specific hosts. For example, if you find that you want to block connections to a certain web banner or web habit-tracking site, you can place its hostname in your host file with a bogus IP address. Unfortunately, virus writers have also used the same trick to break auto-update features of antivirus packages. Host Files? Get a Horse! Now that network name services like the Domain Name Service (DNS) and configu- ration services like the Dynamic Host Configuration Protocol (DHCP) are the norm and twiddling host files has become the exception, why bother talking about these files at all? Host files are really simple. The syntax and semantics are immediately understandable to anyone who glances at such a file. That’s not necessarily true for the other services we’ll be exploring later in this chapter. This simplicity means that we can look at ways 138 | Chapter 5: TCP/IP Name and Configuration Services of manipulating such files without getting distracted by the details of a specific service’s implementation, configuration file syntax, etc. The techniques we’re about to explore can be applied to any of the network name and configuration services that use plain-text configuration files. We’re going to initially show them in the context of manipulating host files, because that is the fastest way to demonstrate methods you’ll use time and time again. Later in the chapter you’ll see some of the same ideas demonstrated with other services without all of the explanation. So, if reading about host files makes you feel like an old-timer, read for the “how” and not the “what.”† Perl and host files are a natural match, given Perl’s predilection for text file processing. We’re going to use the simple host file as a springboard for a number of different explorations. To start, let’s look at the parsing of host files. Parsing a host file can be as simple as this: open( my $HOSTS, '<', '/etc/hosts' ) or die "Unable to open host file:$!\n"; my %addrs; my %names; while ( defined( $_ = <$HOSTS> ) ) { next if /^#/; # skip comments lines next if /^\s*$/; # skip empty lines s/\s*#.*$//; # delete in-line comments and preceding whitespace chomp; my ( $ip, @names ) = split; die "The IP address $ip already seen!\n" if ( exists $addrs{$ip} ); $addrs{$ip} = [@names]; for (@names) { die "The host name $_ already seen!\n" if ( exists $names{lc $_} ); $names{lc $_} = $ip; } } close $HOSTS; The previous code walks through an /etc/hosts file (skipping blank lines and comments), creating two data structures for later use. The first data structure is a hash of lists of hostnames keyed by the IP address. It looks something like this: $addrs{''} = ['localhost']; $addrs{''} = ['','rivendell']; $addrs{''} = ['','everest']; The second is a hash table of hostnames, keyed by name. For the same file, the %names hash would look like this: $names{'localhost'} = '' $names{'everest'} = '' † Plus, a real Ol’ Timer would probably point out to you that he still adds the critical machines to his hosts file and uses it as a backup (via nsswitch.conf) when he’s concerned about things breaking should DNS go south. Host Files | 139 $names{''} = '' $names{'rivendell'} = '' $names{''} = '' Note that in the simple process of parsing this file, we’ve also added some functionality. Our code checks for duplicate hostnames and IP addresses (which are bad news on a TCP/IP network unless you really mean them to be there, for virtual hosts, multihomed machines, high availability, etc.). When dealing with network-related data, use every opportunity possible to check for errors and bad information. It is always better to catch problems early in the game than to have them bite you once the data has been propagated to your entire network. Because it is so important, I’ll return to this topic later in the chapter. Generating Host Files Now we’ll turn to the more interesting topic of generating host files. Let’s assume we have the following host database file for the hosts on our network: name: shimmer address: aliases: shim shimmy shimmydoodles owner: David Davis department: software building: main room: 909 manufacturer: Sun model: M4000 -=- name: bendir address: aliases: ben bendoodles owner: Cindy Coltrane department: IT building: west room: 143 manufacturer: Apple model: Mac Pro -=- name: sulawesi address: aliases: sula su-lee owner: Ellen Monk department: design building: main room: 1116 manufacturer: Apple model: Mac Pro -=- name: sander address: aliases: sandy micky mickydoo owner: Alex Rollins department: IT 140 | Chapter 5: TCP/IP Name and Configuration Services building: main room: 1101 manufacturer: Dell model: Optiplex 740 -=- The format is simple: fieldname: value, with -=- used as a separator between records. You might find that you need other fields than those listed here, or that you have too many records to make it practical to keep them in a single flat file. Though we are using a single flat file here, the concepts we’ll show in this chapter are not backend-specific; for example, they could be generated from an LDAP directory (more on those in Chap- ter 9). Here’s some code that will parse a file like this to generate a host file: my $datafile = 'database'; my $recordsep = "-=-\n"; open my $DATAFILE, '<', "$datafile" or die "Unable to open datafile:$!\n"; { local $/ = $recordsep; # prepare to read in database file one record at a time print "#\n# host file - GENERATED BY $0\n# DO NOT EDIT BY HAND!\n#\n"; my %record; while (<$DATAFILE>) { chomp; # remove the record separator # split into key1,value1,, hash of record %record = split /:\s*|\n/; print "$record{address}\t$record{name} $record{aliases}\n"; } close $DATAFILE; } Here’s the output: # # host file - GENERATED BY createhosts # DO NOT EDIT BY HAND! # shimmer shim shimmy shimmydoodles bendir ben bendoodles sulawesi sula su-lee sander sandy micky mickydoo. Got “System Administration Database” Religion Yet? In Chapter 3, I made an impassioned plea for the use of a separate administrative da- tabase to track account information. The same arguments are doubly true for network host data. In this chapter we’re going to demonstrate how even a simple flat-file host database can be manipulated to produce impressive output that drives each of the Host Files | 141 services we’ll be discussing. For larger sites, a “real” database would serve well. If you’d like to see an example of this output, take a quick glance ahead at the output at the end of the section “Improving the Host File Output” on page 144. The host database approach is beautiful for a number of reasons. First, changes need to be made only to a single file or data source. Make the changes, run some scripts, and presto!, we’ve generated the configuration files needed for a number of services. These configuration files are significantly less likely to contain small syntax errors (like missing semicolons or comment characters), because they haven’t been touched by human hands. If we write our code correctly, we can catch most of the other possible errors during the parsing stage. If you haven’t seen the wisdom of this “best practice” yet, you will by the end of the chapter. Let’s look at a few of the more interesting Perl techniques demonstrated in this small code sample. The first unusual thing we do is set $/ from within a small code block (delimited by the braces). In this little code block, Perl treats each chunk of text that ends in -=-\n as a single record. This means the while statement will read in an entire record at a time and assign it to $_. We place the local statement within the block so our changes to $/ don’t affect any other code we might write in the future that uses this code sample. The second interesting tidbit is the split() assignment technique. Our goal is to get each record into a hash with a key as the field name and its value as the field value. You’ll see later why we go to this trouble, as we develop this example further. The first step is to break $_ into component parts using split(). The array we get back from split() is shown in Table 5-1. Table 5-1. The array returned by split() Element Value 0 name 1 shimmer 2 address 3 4 Aliases 5 shim shimmy shimmydoodles 6 Owner 7 David Davis 8 Department 9 Software 10 Building 11 Main 142 | Chapter 5: TCP/IP Name and Configuration Services Element Value 12 Room 13 909 14 Manufacturer 15 Sun 16 Model 17 M4000 Take a good look at the contents of this list. Starting with the first element (element 0), we have a key/value pair list (i.e., key=Name, value=shimmer, key=Address, value= that we can assign to populate a hash. Once this hash is created, we can print the parts we need. Error-Checking the Host File Generation Process Printing a bare host file is just the beginning of what we can do. One very large benefit of using a separate database that gets converted into another form is the ability to insert error-checking into the conversion process. As mentioned earlier, this can prevent sim- ple typos from becoming a problem before they get a chance to propagate or be put into production use. Here’s the previous code with some simple additions to check for typos: my $datafile = 'database'; my $recordsep = "-=-\n"; open my $DATAFILE, '<', "$datafile" or die "Unable to open datafile:$!\n"; { local $/ = $recordsep; # prepare to read in database file one record at a time print "#\n# host file - GENERATED BY $0\n# DO NOT EDIT BY HAND!\n#\n"; my %record; my %addrs; while (<$DATAFILE>) { chomp; # remove the record separator # split into key1,value1,... bingo, hash of record %record = split /:\s*|\n/; # check for bad hostnames if ( $record{name} =~ /[^-.a-zA-Z0-9]/ ) { warn "!!!! $record{name} has illegal host name characters, " . "skipping...\n"; next; } Host Files | 143 # check for bad aliases if ( $record{aliases} =~ /[^-.a-zA-Z0-9\s]/ ) { warn "!!!! $record{name} has illegal alias name characters, " . "skipping...\n"; next; } # check for missing address if ( !$record{address} ) { warn "!!!! $record{name} does not have an IP address, " . "skipping...\n"; next; } # check for duplicate address if ( defined $addrs{ $record{address} } ) { warn "!!!! Duplicate IP addr: $record{name} & $addrs{$record{address}}, skipping...\n"; next; } else { $addrs{ $record{address} } = $record{name}; } print "$record{address}\t$record{name} $record{aliases}\n"; } close $DATAFILE; } Improving the Host File Output Let’s borrow from Chapter 10 on logs and add some analysis to the conversion process. We can automatically add useful headers, comments, and separators to the data. Here’s some example output using the exact same database: # # host file - GENERATED BY createhosts3 # DO NOT EDIT BY HAND! # # Converted by David N. Blank-Edelman (dnb) on Sun Jun 8 00:43:24 2008 # # number of hosts in the design department: 1. # number of hosts in the software department: 1. # number of hosts in the IT department: 2. # total number of hosts: 4 # # Owned by Cindy Coltrane (IT): west/143 bendir ben bendoodles # Owned by Alex Rollins (IT): main/1101 sander sandy micky mickydoo # Owned by Ellen Monk (design): main/1116 144 | Chapter 5: TCP/IP Name and Configuration Services sulawesi sula su-lee # Owned by David Davis (software: main/909 shimmer shim shimmy shimmydoodles Here’s the code that produced that output, followed by some commentary: my $datafile = 'database'; my $recordsep = "-=-\n"; # get username on either Windows or Unix my $user = ( $^O eq 'MSWin32' ) ? $ENV{USERNAME} : (getpwuid($<))[6] . ' (' . (getpwuid($<))[0] . ')'; open my $DATAFILE, '<', "$datafile" or die "Unable to open datafile:$!\n"; my %addrs; my %entries; { local $/ = $recordsep; # read in database file one record at a time while (<$DATAFILE>) { chomp; # remove the record separator # split into key1,value1 my @record = split /:\s*|\n/; my $record = {}; # create a reference to empty hash %{$record} = @record; # populate that hash with @record # check for bad hostname if ( $record->{name} =~ /[^-.a-zA-Z0-9]/ ) { warn '!!!! ' . $record->{name} . " has illegal host name characters, skipping...\n"; next; } # check for bad aliases if ( $record->{aliases} =~ /[^-.a-zA-Z0-9\s]/ ) { warn '!!!! ' . $record->{name} . " has illegal alias name characters, skipping...\n"; next; } # check for missing address if ( !$record->{address} ) { warn '!!!! ' . $record->{name} . " does not have an IP address, skipping...\n"; next; } Host Files | 145 # check for duplicate address if ( defined $addrs{ $record->{address} } ) { warn '!!!! Duplicate IP addr:' . $record->{name} . ' & ' . $addrs{ $record->{address} } . ", skipping...\n"; next; } else { $addrs{ $record->{address} } = $record->{name}; } $entries{ $record->{name} } = $record; # add this to a hash of hashes } close $DATAFILE; } # print a nice header print "#\n# host file - GENERATED BY $0\n# DO NOT EDIT BY HAND!\n#\n"; print "# Converted by $user on " . scalar(localtime) . "\n#\n"; # count the number of entries in each department and then report on it my %depts; foreach my $entry ( keys %entries ) { $depts{ $entries{$entry}->{department} }++; } foreach my $dept ( keys %depts ) { print "# number of hosts in the $dept department: $depts{$dept}.\n"; } print '# total number of hosts: ' . scalar( keys %entries ) . "\n#\n\n"; # iterate through the hosts, printing a nice comment and the entry itself foreach my $entry ( keys %entries ) { print '# Owned by ', $entries{$entry}->{owner}, ' (', $entries{$entry}->{department}, "): ", $entries{$entry}->{building}, '/', $entries{$entry}->{room}, "\n"; print $entries{$entry}->{address}, "\t", $entries{$entry}->{name}, ' ', $entries{$entry}->{aliases}, "\n\n"; } The most significant difference between this code example and the previous one is the data representation. Because there was no need in the previous example to retain the information from a record after it had been printed, we could use the single hash %record. But for this code, we chose to read the file into a slightly more complex data structure (a hash of hashes) so we could do some simple analysis of the data before printing it. We could have kept a separate hash table for each field (similar to our needspace example in Chapter 2), but the beauty of this approach is its maintainability. If we decide later to add a serial_number field to the database, we do not need to change our program’s parsing code; it will just magically appear as $record->{serial_number}. 146 | Chapter 5: TCP/IP Name and Configuration Services The downside is that Perl’s syntax probably makes our code look more complex than it is. Here’s an easy way to look at it: we’re parsing the file in precisely the same way we did in the last example. The difference is this time we are storing each record in a newly created anonymous hash. Anonymous hashes are just like normal hash variables except they are accessed through a reference, instead of a name. To create our larger data structure (a hash of hashes), we link this new anonymous hash back into the main hash table, %entries. When we are done, %entries has a key for each machine name. Each key has a value that is a reference to a separate new hash table containing all of the fields associated with that machine (IP address, room, etc.). Perhaps you’d prefer to see the output sorted by IP address? No problem, just include a custom sort routine by changing this line: foreach my $entry (keys %entries) { to: foreach my $entry (sort byaddress keys %entries) { and adding: sub byaddress { my @a = split(/\./,$entries{$a}->{address}); my @b = split(/\./,$entries{$b}->{address}); ($a[0]<=>$b[0]) || ($a[1]<=>$b[1]) || ($a[2]<=>$b[2]) || ($a[3]<=>$b[3]); } This is one of the easiest to understand ways to sort IP addresses, but it is also one of the least efficient because of all of the split() operations that have to take place. A far better way to do this is to compare packed sort keys, a technique first proposed in a paper by Uri Guttman and Larry Rosler ( Guttman’s Sort::Maker module can assist you with implementing that method. The Sort::Key module by Salvador Fandiño García offers another easy way to perform highly efficient sorting in Perl. If you don’t want to install a separate module just to set up a sort, search for “sort ip address perl” on the Web and you’ll find other, more efficient suggestions. Here’s the relevant portion of the output, now nicely sorted: # Owned by Cindy Coltrane (IT): west/143 bendir ben bendoodles # Owned by David Davis (software): main/909 shimmer shim shimmy shimmydoodles Host Files | 147 # Owned by Ellen Monk (design): main/1116 sulawesi sula su-lee # Owned by Alex Rollins (IT): main/1101 sander sandy micky mickydoo Make the output look good to you. Let Perl support your professional and aesthetic endeavors. Incorporating a Source Code Control System In a moment we’re going to move on to the next approach to the IP address-to-name mapping problem. But before we do, we’ll want to add another twist to our host file creation process, because that single file is about to take on network-wide importance. A mistake in this file will affect an entire network of machines. To give us a safety net, we’ll want a way to back out of bad changes, essentially going back in time to a prior configuration state. The most elegant way to build a time machine like this is to add a source control system to the process. Source control systems are typically used by developers to: • Keep a record of all changes to important files. • Prevent multiple people from changing the same file (or parts of a file) at the same time, inadvertently undoing each other’s efforts. • Allow us to revert to a previous version of a file, thus backing out of problems. This functionality is extremely useful to a system administrator. The error-checking code we added to the conversion process in “Error-Checking the Host File Generation Process” on page 143 can help with certain kinds of typos and syntax errors, but it does not offer any protection against semantic errors (e.g., deleting an important hostname, assigning the wrong IP address to a host, or misspelling a hostname). You could add semantic error checks into the conversion process, but you probably wouldn’t catch all of the possible errors. As I’ve quoted before, nothing is foolproof, since fools are so ingenious. You might think it would be better to apply source control system functionality to the initial database editing process, but there are two good reasons why it is also important to apply it to the resultant output: Time For large data sets, the conversion process might take some time. If your network is flaking out and you need to revert to a previous revision, it’s discouraging to have to stare at a Perl process chugging away to generate the file you need (presuming you can even get to Perl at that point). Absence of database change control If you choose to use a real database engine for your data storage (and often this is the right choice), there may not be a convenient way to apply a source control 148 | Chapter 5: TCP/IP Name and Configuration Services mechanism like this. You’ll probably have to write your own change control mech- anisms for the database editing process. My source control system of choice‡ is the Revision Control System (RCS). RCS has some Perl- and system administration-friendly features: • It is multiplatform. There are ports of GNU RCS 5.7 to most Unix systems, Win- dows, Mac OS X, etc. • It has a well-defined command-line interface. All functions can be performed from the command line, even on GUI-centric operating systems. • It is easy to use. There’s a small command set for basic operations that can be learned in five minutes (see Appendix E). • It has keywords. Magic strings can be embedded in the text of files under RCS that are automatically expanded. For instance, any occurrence of $ Date:$ in a file will be replaced with the date the file was last entered into the RCS system. • It’s free. The source code for the GNU version of RCS is freely redistributable, and binaries for most systems are also available. A copy of the source can be found at gnu/rcs. If you’ve never dealt with RCS before, please take a moment to read Appen- dix E before going any further. The rest of this section assumes a cursory knowledge of the RCS command set. Craig Freter has written an object-oriented module called Rcs that makes using RCS from Perl easy. The steps are: 1. Load the module. 2. Tell the module where your RCS command-line binaries are located. 3. Create a new Rcs object, and configure it with the name of the file you are using. 4. Call the necessary object methods (named after their corresponding RCS commands). Let’s add this to our host file generation code so you can see how the module works. Besides the Rcs module code, we’ve also changed things so the output is sent to a specific file and not STDOUT, as in our previous versions. Only the code that has changed is shown. Refer to the previous example for the omitted lines represented by “...”: my $outputfile = "hosts.$$"; # temporary output file my $target = 'hosts'; # where we want the converted data stored ... open my $OUTPUT, '>', "$outputfile" or die "Unable to write to $outputfile:$!\n"; print $OUTPUT "#\n# host file - GENERATED BY $0\n# DO NOT EDIT BY HAND!\n#\n"; ‡ At least in this context. If you’d like to know why I recommend RCS over other, much spiffier source control systems (SVN, git, etc.) here, see Appendix E. Host Files | 149 print $OUTPUT "# Converted by $user on " . scalar(localtime) . "\n#\n"; ... foreach my $dept ( keys %depts ) { print $OUTPUT "# number of hosts in the $dept department: $depts{$dept}.\n"; } print $OUTPUT '# total number of hosts: ' . scalar( keys %entries ) . "\n#\n\n"; # iterate through the hosts, printing a nice comment and the entry itself foreach my $entry ( keys %entries ) { print $OUTPUT '# Owned by ', $entries{$entry}->{owner}, ' (', $entries{$entry}->{department}, '): ', $entries{$entry}->{building}, '/', $entries{$entry}->{room}, "\n"; print $OUTPUT $entries{$entry}->{address}, "\t", $entries{$entry}->{name}, ' ', $entries{$entry}->{aliases}, "\n\n"; } close $OUTPUT; use Rcs; Rcs->bindir('/arch/gnu/bin'); my $rcsobj = Rcs->new; $rcsobj->file($target); $rcsobj->co('-l'); rename( $outputfile, $target ) or die "Unable to rename $outputfile to $target:$!\n"; $rcsobj->ci( '-u', '-m' . 'Converted by ' . ( getpwuid($<) )[6] . ' (' . ( getpwuid($<) )[0] . ') on ' . scalar localtime ); This code assumes the target file has been checked in at least once already. To see the effect of this code addition, we can look at three entries excerpted from the output of rlog hosts: revision 1.5 date: 2007/05/19 23:34:16; author: dnb; state: Exp; lines: +1 −1 Converted by David N. Blank-Edelman (dnb) on Tue May 19 19:34:16 2007 ---------------------------- revision 1.4 date: 2007/05/19 23:34:05; author: eviltwin; state: Exp; lines: +1 −1 Converted by Divad Knalb-Namlede (eviltwin) on Tue May 19 19:34:05 2007 ---------------------------- revision 1.3 date: 2007/05/19 23:33:35; author: dnb; state: Exp; lines: +20 −0 Converted by David N. Blank-Edelman (dnb) on Tue May 19 19:33:16 2007 This example doesn’t show much of a difference between file versions (see the lines: part of the entries), but you can see that we are tracking the changes every time the file gets created. If we needed to, we could use the rcsdiff command to see exactly what 150 | Chapter 5: TCP/IP Name and Configuration Services has changed. Under dire circumstances, if one of these changes had wreaked unexpec- ted havoc on the network, we would be able to revert to a previous version. Before we move on, let’s do a quick review of the three techniques we have learned so far so we can be sure to bring them forward when we look at other name services: • Generating a configuration file from an external database of some sort is a big win. • Checking for simple errors in the data during the process, well before they can have a serious impact on the network, is a good thing. • Incorporating a source control system into the process gives you a good way to recover from more complex errors and a way of tracking changes. NIS, NIS+, and WINS Developers at Sun Microsystems realized that the “edit one file per machine” approach endemic to host files didn’t scale, so they invented Yellow Pages (YP), which was de- signed to distribute all the network-wide configuration file information found in files like /etc/hosts, /etc/passwd, /etc/services, and so on. In this chapter, we’ll concentrate on its use as a network name service to distribute machine name-to-IP address mapping information. YP was renamed the Network Information Service (NIS) in 1990, shortly after British Telecom asserted (with lawyers) that it held the trademark for “Yellow Pages” in the U.K. The ghost of the name “Yellow Pages” still haunts many a Unix box today in the names used for NIS commands and library calls (e.g., ypcat, ypmatch, yppush). All modern Unix variants support NIS. Mac OS X makes it easy to client off of existing NIS servers through (at least in Tiger and later releases) the Directory Access utility, found in /Applications/Utilities (check the box next to “BSD Flat File and NIS” and click Apply). OS X also ships with the right files (/usr/libexec/ypserv, /var/yp/*, etc.) to serve NIS, though I’ve never seen it done. The NIS and Windows story is a bit more complex. Once upon a time, back in the days of the first edition of this book, it was possible to replace one of the Windows authen- tication libraries with custom code that would talk to NIS servers instead of doing domain-based authentication. This was the NISGINA solution. If you need to have a Windows machine use NIS-sourced data, your best bet at this point is to use Samba ( as a bridge between the two worlds. On the NIS-server front, Microsoft has also built into its Windows 2003 R2 product an NIS server that allows it to serve Active Directory-based information to NIS clients. This works well if you decide to make Active Directory the center of your authentication universe but still need to serve NIS to other, non-Windows clients. To save you some hunting, Microsoft now calls this component “Identity Management for Unix” (IdMU). You’ll need to add it to your installation by hand, via Add or Remove Programs→Add/Remove Windows Components→Active Directory Services [Details]. NIS, NIS+, and WINS | 151 In NIS, an administrator designates one or more machines as servers from which other machines will receive client services. One server is the master server, and the others are slave servers. The master server holds the master copies of the actual text files all machines normally use (e.g., /etc/hosts or /etc/passwd). Changes to these files take place on the master and are then propagated to the slave servers. Any machine on the network that needs hostname-to-IP address mapping information can query a server instead of keeping a local copy of that information. A client can request this information from either the master or any of the slave servers. Client queries are looked up in the NIS maps (another name for the master’s data files after they’ve been converted to the Unix DBM database format and propagated to the slave servers). The details of this conversion process (which involves makedbm and some other ran- dom munging) can be found in the Makefile located in /var/yp on most machines. A collection of NIS servers and clients that share the same maps is called an NIS domain. With NIS, network administration becomes considerably easier. For instance, if purchases more machines for its network, it is no problem to integrate them into the network. The network manager simply edits the host file on the master NIS server and pushes the new version out to the slave servers. Every client in the NIS domain now “knows” about the new machine. NIS offers one-touch administration ease coupled with some redundancy (if one server goes down, a client can ask another) and load sharing (not all of the clients in a network have to rely on a single server). With this theory in mind, let’s see how Perl can help us with NIS-related tasks. We can start with the process of getting data into NIS. You may be surprised to know that we’ve already done the work for this task. We can import the host files we created in the previous section into NIS by just dropping them into place in the NIS master server’s source file directory and activating the usual push mechanisms (usually by typing make in /var/yp). By default, the Makefile in /var/yp uses the contents of the master server’s configuration files as the source for the NIS maps. It is usually a good idea to set up a separate directory for your NIS map source files, changing the Makefile accordingly. This allows you to keep separate data for your NIS master server and other members of your NIS domain. For example, you might not want to have the /etc/passwd file for your NIS master as the password map for the entire domain, and vice versa. A more interesting task is getting data out of NIS by querying an NIS server. The easiest way to do this is via Rik Harris’s Net::NIS module (now maintained by Ed Santiago). Here’s an example of how to grab and print the entire contents of the host map with a single function call using Net::NIS, similar to the NIS command ypcat: use Net::NIS; 152 | Chapter 5: TCP/IP Name and Configuration Services # get our default NIS domain name my $domain = Net::NIS::yp_get_default_domain(); # grab the map my ( $status, $info ) = Net::NIS::yp_all( $domain, 'hosts.byname' ); foreach my $name ( sort keys %{$info} ) { print "$name => $info->{$name}\n"; } First we query the local host for its default domain name. With this info, we can call Net::NIS::yp_all() to retrieve the entire host map. The function call returns a status variable and a reference to a hash table containing the contents of that map. We print this information using Perl’s usual dereference syntax. If we want to look up the IP address of a single host, it is more efficient to query the server specifically for that value: use Net::NIS; my $hostname = ''; my $domain = Net::NIS::yp_get_default_domain(); my ( $status, $info ) = Net::NIS::yp_match( $domain, 'hosts.byname', $hostname ); print "$info\n"; Net::NIS::yp_match() returns a status variable and the appropriate value (as a scalar) for the info being queried. If the Net::NIS module does not compile or work for you, there’s always the “call an external program” method. For example: @hosts=`/ypcat hosts` or: open my $YPCAT, '-|', '/ypcat hosts'); while (){...} Let’s wind up this section with a useful example of both this technique and Net::NIS in action. This small but handy piece of code will query NIS for the list of NIS servers currently running and then query each of them in turn using the yppoll program. If any of the servers fails to respond, it complains loudly: use Net::NIS; my $yppollex = '/usr/sbin/yppoll'; # full path to the yppoll executable my $domain = Net::NIS::yp_get_default_domain(); # our NIS domain my ( $status, $info ) = Net::NIS::yp_all( $domain, 'ypservers' ); foreach my $server ( sort keys %{$info} ) { my $answer = `$yppollex -h $server hosts.byname`; NIS, NIS+, and WINS | 153 if ( $answer !~ /has order number/ ) { print STDERR "$server is not responding properly!\n"; } } There are a number of ways we could improve upon this code (e.g., we could check if the order number returned matched amongst the servers), but that’s left as an exercise for you-know-who. NIS+ Sun included the next version of NIS, called NIS+, with the Solaris operating system. NIS+ addresses many of the most serious problems of NIS, such as security. Unfortu- nately (or fortunately, since NIS+ can be a bit difficult to administer), NIS+ has not caught on in the Unix world nearly as well NIS did. Until recently, there was virtually no support for it on machines not manufactured by Sun. Thorsten Kukuk’s work to bring it to Linux ( has ceased, and even Sun has aban- doned it for LDAP. Given its marginal status, we’re not going to look at NIS+ in this book. If you do need to work with NIS+ from Perl, Harris has a Net::NISPlus module that’s up to the task. Windows Internet Name Server (WINS) Let’s look at one more dying protocol, for historical reasons. When Microsoft began to run its proprietary networking protocol NetBIOS over TCP/IP (NetBT), it also found a need to handle the name-to–IP address mapping question. The first shot was the lmhosts file, modeled after the standard host file. This was quickly supplemented with an NIS-like mechanism. Since NT version 3.5, Microsoft has offered a centralized scheme called the Windows Internet Name Server (WINS). WINS differs in several ways from NIS: • WINS is specialized for the distribution of host-to-IP address mappings. Unlike NIS, it is not used to centralize distribution of other information (e.g., passwords, networks, port mappings, and user groups). • WINS servers receive most of the information they distribute from preconfigured client registrations (they can be preloaded with information). Once they’ve re- ceived an IP address either manually or via DHCP, WINS clients are responsible for registering and re-registering their information. This is different from NIS in that client machines ask the server for information that has been preloaded and, with only one exception (passwords), do not update the information on that server. WINS, like NIS, offers the ability to have multiple servers available for reliability and load sharing through the use of a push/pull partner model. As of Windows 2000, WINS was deprecated (read “killed off”) in favor of the Dynamic Domain Name Service (DDNS), an extension to the basic DNS system we’re just about to discuss. 154 | Chapter 5: TCP/IP Name and Configuration Services Given that WINS, like NIS+, is about to go to the Great Protocol Graveyard to die, we’re not going to explore Perl code to work with it. There is currently very little support for working directly with WINS from Perl (I know of no Perl modules designed spe- cifically to interact with WINS). If you need to do this, your best bet may be to call some of the command-line utilities found in the Windows resource kits, such as WINSCHK and WINSCL. Domain Name Service (DNS) As useful as they may be, NIS and WINS suffer from flaws that make them unsuitable for “entire-Internet” uses. There are two main issues: Scale Even though these schemes allow for multiple servers, each server must have a complete copy of the entire network topology.* This topology must be duplicated to every other server, which can become a time-consuming process. WINS also suffers because of its dynamic registration model: a sufficient number of WINS clients could melt down any set of Internet-wide WINS servers with registration requests. Administrative control We’ve been talking about strictly technical issues up until now, but that’s not the only side of administration. NIS, in particular, requires a single point of adminis- tration; whoever controls the master server controls the entire NIS domain led by that machine, and any changes to the network namespace must pass through that administrative gatekeeper. This doesn’t work for a namespace the size of the Internet. The Domain Name Service (DNS) was invented to deal with the flaws inherent in maintaining host files or NIS/NIS+/WINS-like systems. Under DNS, the network namespace is partitioned into a set of somewhat arbitrary “top-level domains.” Each top-level domain can then be subdivided into smaller domains, each of which can in turn be partitioned, and so on. At each dividing point it is possible to designate a different party to retain authoritative control over that portion of the namespace. This handles our administrative control concern. Network clients that reside in the individual parts of this hierarchy consult the name server closest to them in the hierarchy. If the information the client is looking for can be found on that local server, it is returned to the client. On most networks, the majority of name-to-IP address queries are for machines on that network, so the local servers handle most of the local traffic. This satisfies the scale problem. Multiple DNS servers * NIS+ offered mechanisms for a client to search for information outside of the local domain, but they were not as flexible as those in DNS. Domain Name Service (DNS) | 155 (also known as secondary or slave servers) can be set up for redundancy and load- balancing purposes. If a DNS server is asked about a part of the namespace that it does not control or know about, it can either instruct the client to look elsewhere (usually higher up in the tree) or fetch the required information on behalf of the client by contacting other DNS servers. In this scheme, no single server needs to know the entire network topology, most quer- ies are handled locally, local administrators retain local control, and everybody is happy. DNS offers such an advantage compared to other systems that most other sys- tems, including NIS and WINS, offer a way to integrate DNS. For instance, a Solaris NIS server can be instructed to perform a DNS query if a client asks it for a host it does not know. The results of this query are returned as a standard NIS query reply, so the client has no knowledge that any magic has been performed on its behalf. Microsoft DNS servers have similar functionality: if a client asks a Microsoft DNS server for the address of a local machine that it does not know about, the server can be configured to consult a WINS server on the client’s behalf. Generating DNS (BIND) Configuration Files Production of DNS configuration files for the popular BIND DNS server (https://www follows the same procedure that we used to generate host and NIS source files: • We store data in a separate database (the same database can and probably should be the source for all of the files we’ve been discussing). • We convert data to the output format of our choice, checking for errors as we go. • We use RCS (or an equivalent source control system) to store old revisions of files. For DNS, we have to expand the second step because the conversion process is more complicated. As we launch into these complications, you may find it handy to have DNS and BIND by Paul Albitz and Cricket Liu (O’Reilly) on hand for information on the DNS configuration files we’ll be creating. Creating the administrative header DNS configuration files begin with an administrative header that provides information about the server and the data it is serving. The most important part of this header is the Start of Authority (SOA) resource record. The SOA contains: • The name of the administrative domain served by this DNS server • The name of the primary DNS server for that domain • Contact info for the DNS administrator(s) • The serial number of the configuration file (more on this in a moment) 156 | Chapter 5: TCP/IP Name and Configuration Services • Refresh and retry values for secondary servers (i.e., when they synchronize with the primary server) • Time to Live (TTL) settings for the data being served (i.e., how long the information being provided can safely be cached) Here’s an example header: @ IN SOA ( 2007052900 ; serial 10800 ; refresh 3600 ; retry 604800 ; expire 43200) ; TTL @ IN NS Most of this information is just tacked on the front of a DNS configuration file verbatim each time it is generated. The one piece we need to worry about is the serial number. Once every X seconds (where X is determined by the refresh value), secondary name servers contact their primary servers looking for updates to their DNS data. Modern secondary DNS servers (like BIND v8+ or Microsoft DNS) can also be told by their master servers to check for updates when the master data has changed. In both cases, each secondary server queries the primary server for its SOA record. If the SOA record contains a serial number higher than that server’s current serial number, a zone transfer is initiated (that is, the secondary downloads a new data set). Consequently, it is im- portant to increment this number each time a new DNS configuration file is created. Many DNS problems are caused by failures to update the serial number. There are at least two ways to make sure the serial number is always incremented: • Read the previous configuration file and increment the value found there. • Compute a new value based on an external number source “guaranteed” to incre- ment over time (like the system clock or the RCS version number of the file). Here’s some example code that uses a combination of these two methods to generate a valid header for a DNS zone file. It creates a serial number formatted as recommended in Albitz and Liu’s book (YYYYMMDDXX, where YYYY=year, MM=month, DD=day, and XX=a two-digit counter to allow for more than one change per day): # get today's date in the form of YYYYMMDD my @localtime = localtime; my $today = sprintf( "%04d%02d%02d", $localtime[5] + 1900, $localtime[4] + 1, $localtime[3] ); # get username on either Windows or Unix my $user = ( $^O eq 'MSWin32' ) ? $ENV{USERNAME} : ( getpwuid($<) )[6] . ' (' . ( getpwuid($<) )[0] . ')'; Domain Name Service (DNS) | 157 sub GenerateHeader { my ($olddate,$count); # open old file if possible and read in serial number # (assumes the format of the old file) if ( open( my $OLDZONE, '<', $target ) ) { while (<$OLDZONE>) { last if ( $olddate, $count ) = /(\d{8})(\d{2}).*serial/; } close $OLDZONE; } # If $count is defined, we did find an old serial number. # If the old serial number was for today, increment last 2 digits; # else start a new number for today. my $count = ( defined $count and $olddate eq $today ) ? $count + 1 : 0; my $serial = sprintf( "%8d%02d", $today, $count ); # begin the header my $header = "; dns zone file - GENERATED BY $0\n"; $header .= "; DO NOT EDIT BY HAND!\n;\n"; $header .= "; Converted by $user on " . scalar( (localtime) ) . "\n;\n"; # count the number of entries in each department and then report foreach my $entry ( keys %entries ) { $depts{ $entries{$entry}->{department} }++; } foreach my $dept ( keys %depts ) { $header .= "; number of hosts in the $dept department: " . "$depts{$dept}.\n"; } $header .= '; total number of hosts: ' . scalar( keys %entries ) . "\n;\n\n"; $header .= <<"EOH"; @ IN SOA ( $serial ; serial 10800 ; refresh 3600 ; retry 604800 ; expire 43200) ; TTL @ IN NS EOH return $header; } Our code attempts to read in the previous DNS configuration file to determine the last serial number in use. This number then gets split into date and counter fields. If the date we’ve read is the same as the current date, we need to increment the counter. If not, we create a serial number based on the current date with a counter value of 00. 158 | Chapter 5: TCP/IP Name and Configuration Services Once we have our serial number, the rest of the code concerns itself with writing out a pretty header in the proper form. Generating multiple configuration files Now that we’ve covered the process of writing a correct header for our DNS configu- ration files, there is one more complication we need to address. A well-configured DNS server has both forward (name-to-IP address) and reverse (IP address-to-name) map- ping information available for every domain, or zone, it controls. Thus, two configu- ration files are required per zone. The best way to keep these synchronized is to create them both at the same time. This is the last file-generation script we’ll see in this chapter, so let’s put together everything we’ve done so far. Our script will take a simple database file and generate the necessary DNS zone configuration files. To keep this script simple, I’ve made a few assumptions about the data, the most im- portant of which has to do with the topology of the network and namespace. This script assumes that the network consists of a single class-C subnet with a single DNS zone. As a result, we only create a single forward mapping file and its reverse mapping sibling file. Code to handle multiple subnets and zones (with separate files for each) would be a worthwhile addition. Here’s a quick walkthrough of what we’ll do in this code: 1. Read the database file into a hash of hashes, checking the data as we go. 2. Generate a header. 3. Write out the forward mapping (name-to-IP address) file and check it into RCS. 4. Write out the reverse mapping (IP address-to-name) file and check it into RCS. Here is the code and its output: use Rcs; my $datafile = 'database'; # our host database my $outputfile = "zone.$$"; # our temporary output file my $target = 'zone.db'; # our target output my $revtarget = 'rev.db'; # our target output for the reverse mapping my $defzone = ''; # the default zone being created my $rcsbin = '/usr/local/bin'; # location of our RCS binaries my $recordsep = "-=-\n"; # get today's date in the form YYYYMMDD my @localtime = localtime; my $today = sprintf( "%04d%02d%02d", $localtime[5] + 1900, $localtime[4] + 1, $localtime[3] ); # get username on either Windows or Unix my $user = Domain Name Service (DNS) | 159 ( $^O eq 'MSWin32' ) ? $ENV{USERNAME} : ( getpwuid($<) )[6] . ' (' . ( getpwuid($<) )[0] . ')'; # read in the database file open my $DATAFILE, '<', "$datafile" or die "Unable to open datafile:$!\n"; my %addrs; my %entries; { local $/ = $recordsep; # read in the database file one record at a time while (<$DATAFILE>) { chomp; # remove the record separator # split into key1,value1 my @record = split /:\s*|\n/; my $record = {}; # create a reference to empty hash %{$record} = @record; # populate that hash with @record # check for bad hostname if ( $record->{name} =~ /[^-.a-zA-Z0-9]/ ) { warn '!!!! ' . $record->{name} . " has illegal host name characters, skipping...\n"; next; } # check for bad aliases if ( $record->{aliases} =~ /[^-.a-zA-Z0-9\s]/ ) { warn '!!!! ' . $record->{name} . " has illegal alias name characters, skipping...\n"; next; } # check for missing address if ( !$record->{address} ) { warn '!!!! ' . $record->{name} . " does not have an IP address, skipping...\n"; next; } # check for duplicate address if ( defined $addrs{ $record->{address} } ) { warn '!!!! Duplicate IP addr:' . $record->{name} . ' & ' . $addrs{ $record->{address} } . ", skipping...\n"; next; } else { $addrs{ $record->{address} } = $record->{name}; } 160 | Chapter 5: TCP/IP Name and Configuration Services $entries{ $record->{name} } = $record; # add this to a hash of hashes } close $DATAFILE; } my $header = GenerateHeader(); # create the forward mapping file open my $OUTPUT, '>', "$outputfile" or die "Unable to write to $outputfile:$!\n"; print $OUTPUT $header; foreach my $entry ( sort byaddress keys %entries ) { print $OUTPUT "; Owned by ", $entries{$entry}->{owner}, ' (', $entries{$entry}->{department}, "): ", $entries{$entry}->{building}, '/', $entries{$entry}->{room}, "\n"; # print A record printf $OUTPUT "%-20s\tIN A %s\n", $entries{$entry}->{name}, $entries{$entry}->{address}; # print any CNAMES (aliases) if ( defined $entries{$entry}->{aliases} ) { foreach my $alias ( split( ' ', $entries{$entry}->{aliases} ) ) { printf $OUTPUT "%-20s\tIN CNAME %s\n", $alias, $entries{$entry}->{name}; } } print $OUTPUT "\n"; } close $OUTPUT; Rcs->bindir($rcsbin); my $rcsobj = Rcs->new; $rcsobj->file($target); $rcsobj->co('-l'); rename( $outputfile, $target ) or die "Unable to rename $outputfile to $target:$!\n"; $rcsobj->ci( '-u', '-m' . "Converted by $user on " . scalar(localtime) ); # now create the reverse mapping file open my $OUTPUT, '>', "$outputfile" or die "Unable to write to $outputfile:$!\n"; print $OUTPUT $header; foreach my $entry ( sort byaddress keys %entries ) { print $OUTPUT '; Owned by ', $entries{$entry}->{owner}, ' (', $entries{$entry}->{department}, '): ', $entries{$entry}->{building}, '/', $entries{$entry}->{room}, "\n"; # this uses the default zone we defined at the start of the script printf $OUTPUT "%-3d\tIN PTR %s$defzone.\n\n", ( split /\./, $entries{$entry}->{address} )[3], $entries{$entry}->{name}; Domain Name Service (DNS) | 161 } close $OUTPUT; $rcsobj->file($revtarget); $rcsobj->co('-l'); # assumes target has been checked out at least once rename( $outputfile, $revtarget ) or die "Unable to rename $outputfile to $revtarget:$!\n"; $rcsobj->ci( "-u", "-m" . "Converted by $user on " . scalar(localtime) ); sub GenerateHeader { my ( $olddate, $count ); # open old file if possible and read in serial number # (assumes the format of the old file) if ( open( my $OLDZONE, '<', $target ) ) { while (<$OLDZONE>) { last if ( $olddate, $count ) = /(\d{8})(\d{2}).*serial/; } close $OLDZONE; } # If $count is defined, we did find an old serial number. # If the old serial number was for today, increment last 2 digits; # else start a new number for today. my $count = ( defined $count and $olddate eq $today ) ? $count + 1 : 0; my $serial = sprintf( "%8d%02d", $today, $count ); # begin the header my $header = "; dns zone file - GENERATED BY $0\n"; $header .= "; DO NOT EDIT BY HAND!\n;\n"; $header .= "; Converted by $user on " . scalar( (localtime) ) . "\n;\n"; # count the number of entries in each department and then report my %depts; foreach my $entry ( keys %entries ) { $depts{ $entries{$entry}->{department} }++; } foreach my $dept ( keys %depts ) { $header .= "; number of hosts in the $dept department: " . "$depts{$dept}.\n"; } $header .= '; total number of hosts: ' . scalar( keys %entries ) . "\n;\n\n"; $header .= <<"EOH"; @ IN SOA ( $serial ; serial 10800 ; refresh 3600 ; retry 604800 ; expire 43200) ; TTL @ IN NS 162 | Chapter 5: TCP/IP Name and Configuration Services EOH return $header; } sub byaddress { my @a = split( /\./, $entries{$a}->{address} ); my @b = split( /\./, $entries{$b}->{address} ); ( $a[0] <=> $b[0] ) || ( $a[1] <=> $b[1] ) || ( $a[2] <=> $b[2] ) || ( $a[3] <=> $b[3] ); } Here’s the forward mapping file (zone.db) that gets created: ; dns zone file - GENERATED BY createdns ; DO NOT EDIT BY HAND! ; ; Converted by David N. Blank-Edelman (dnb); on Fri May 29 15:46:46 2007 ; ; number of hosts in the design department: 1. ; number of hosts in the softwaredepartment: 1. ; number of hosts in the IT department: 2. ; total number of hosts: 4 ; @ IN SOA ( 2007052900 ; serial 10800 ; refresh 3600 ; retry 604800 ; expire 43200) ; TTL @ IN NS ; Owned by Cindy Coltrane (marketing): west/143 bendir IN A ben IN CNAME bendir bendoodles IN CNAME bendir ; Owned by David Davis (software): main/909 shimmer IN A shim IN CNAME shimmer shimmy IN CNAME shimmer shimmydoodles IN CNAME shimmer ; Owned by Ellen Monk (design): main/1116 sulawesi IN A sula IN CNAME sulawesi su-lee IN CNAME sulawesi ; Owned by Alex Rollins (IT): main/1101 sander IN A sandy IN CNAME sander Domain Name Service (DNS) | 163 micky IN CNAME sander mickydoo IN CNAME sander And here’s the reverse mapping file (rev.db): ; dns zone file - GENERATED BY createdns ; DO NOT EDIT BY HAND! ; ; Converted by David N. Blank-Edelman (dnb); on Fri May 29 15:46:46 2007 ; ; number of hosts in the design department: 1. ; number of hosts in the softwaredepartment: 1. ; number of hosts in the IT department: 2. ; total number of hosts: 4 ; @ IN SOA ( 2007052900 ; serial 10800 ; refresh 3600 ; retry 604800 ; expire 43200) ; TTL @ IN NS ; Owned by Cindy Coltrane (marketing): west/143 3 IN PTR ; Owned by David Davis (software): main/909 11 IN PTR ; Owned by Ellen Monk (design): main/1116 12 IN PTR ; Owned by Alex Rollins (IT): main/1101 55 IN PTR This method of creating files opens up many more possibilities. Up to now, we’ve generated files using content from a single text-file database. We read a record from the database and wrote it out to our file, perhaps with a dash of nice formatting. Only data that appeared in the database found its way into the files we created. Sometimes, however, it is useful to have the script itself add content during the con- version process. For instance, in the case of DNS configuration file generation, you may wish to embellish the conversion script so it inserts MX (Mail eXchange) records point- ing to a central mail server for every host in your database. A trivial code change from: # print A record printf $OUTPUT "%-20s\tIN A %s\n", $entries{$entry}->{name},$entries{$entry}->{address}; to: # print A record printf $OUTPUT "%-20s\tIN A %s\n", $entries{$entry}->{name},$entries{$entry}->{address}; 164 | Chapter 5: TCP/IP Name and Configuration Services # print MX record print $OUTPUT " IN MX 10 $mailserver\n"; will configure DNS so that mail destined for any host in the domain is received by the machine $mailserver instead. If that machine is configured to handle mail for its do- main, you’ve activated a really useful infrastructure component (i.e., centralized mail handling) with just a single line of Perl code. DNS Checking: An Iterative Approach We’ve spent considerable time in this chapter on the creation of the configuration information to be served by network name services, but that’s only one side of the coin for system and network administrators. Keeping a network healthy also entails check- ing these services once they’re up and running to make sure they are behaving in a correct and consistent manner. For instance, for a system/network administrator, a great deal rides on the question, “Are all of my DNS servers up?” In a troubleshooting situation, it’s equally valuable to ask yourself “Are they all serving the same information?” or, more specifically, “Are the servers responding to the same queries with the same responses? Are they in sync as intended?” We’ll put these questions to good use in this section. In Chapter 2 we saw an example of the Perl motto “There’s more than one way to do it” in action. Perl’s TMTOWTDI-ness makes it an excellent prototype language in which to do “iterative development.” Iterative development is one way of describing the evolutionary process that takes place when writing system administration (and other) programs to handle particular tasks. With Perl, it’s all too easy to bang out a quick-and-dirty hack that gets a job done. Later, you may return to that script and rewrite it so it’s more elegant. There’s even likely to be yet a third iteration of the same code, this time taking a different approach to solving the problem. In this section, we’ll look at three different approaches to the same problem of DNS consistency checking. These approaches will be presented in the order you might re- alistically follow while trying to solve the problem and refine your solution. This or- dering reflects one view on how a solution to a problem can evolve in Perl; your take on this may differ. The third approach, using the Net::DNS module, is probably the easiest and most error-proof of the bunch, but Net::DNS may not address every situation, so we’re going to walk through some “roll your own” approaches first. Be sure to note the pros and cons listed after each solution has been presented. Here’s the task: write a Perl script that takes a hostname and checks a list of DNS servers to see if they all return the same information when queried about this host. To make this task simpler, we’re going to assume that the host has a single, static IP address (i.e., does not have multiple interfaces or addresses associated with it). Domain Name Service (DNS) | 165 Before we look at each approach in turn, let me show you the “driver” code we’re going to use: use Data::Dumper; my $hostname = $ARGV[0]; my @servers = qw(nameserver1 nameserver2 nameserver3); # name servers my %results; foreach my $server (@servers) { $results{$server} = LookupAddress( $hostname, $server ); # populates %results } my %inv = reverse %results; # invert the result hash if (scalar keys %inv > 1) { # see how many elements it has print "There is a discrepancy between DNS servers:\n"; print Data::Dumper->Dump( [ \%results ], ['results'] ), "\n"; } For each of the DNS servers listed in the @servers list, we call the LookupAddress() subroutine. LookupAddress() queries a specific DNS server for the IP address of a given hostname and returns the result so it can be stored in a hash called %results. Each DNS server has a key in %results with the IP address returned by that server as its value. There are many ways to determine if all of the values in %results are the same (i.e., if all the DNS servers returned the same thing in response to our query). Here, we choose to invert %results into another hash table, making all of the keys into values, and vice versa. If all values in %results are the same, there should be exactly one key in the inverted hash. If not, we know we’ve got a situation on our hands, so we call Data::Dumper->Dump() to nicely display the contents of %results for the system admin- istrator to puzzle over. Here’s a sample of what the output looks like when something goes wrong: There is a discrepancy between DNS servers: $results = { nameserver1 => '', nameserver2 => '', nameserver3 => '', }; Let’s take a look at the contestants for the LookupAddress() subroutine. Using nslookup If your background is in Unix, or you’ve done some programming in other scripting languages besides Perl, your first attempt might look a great deal like a shell script. An external program called from the Perl script does the hard work in the following code: use Data::Dumper; my $hostname = $ARGV[0]; 166 | Chapter 5: TCP/IP Name and Configuration Services my @servers = qw(nameserver1 nameserver2 nameserver3 nameserver4); my $nslookup = '/usr/bin/nslookup'; my %results; foreach my $server (@servers) { $results{$server} = LookupAddress( $hostname, $server ); # populates %results } my %inv = reverse %results; # invert the result hash if ( scalar keys %inv > 1 ) { # see how many elements it has print "There is a discrepency between DNS servers:\n"; print Data::Dumper->Dump( [ \%results ], ['results'] ), "\n"; } sub LookupAddress { my ( $hostname, $server ) = @_; my @results; open my $NSLOOK, '-|', "$nslookup $hostname $server" or die "Unable to start nslookup:$!\n"; while (<$NSLOOK>) { next until (/^Name:/); # ignore until we hit "Name: " chomp( $result = <$NSLOOK> ); # next line is Address: response $result =~ s/Address(es)?:\s+//; # remove the label push( @results, $result ); } close $NSLOOK; return join( ', ', sort @results ); } The benefits of this approach are: • It’s a short, quick program to write (perhaps even translated line by line from a real shell script). • We didn’t have to write any messy network code. • It takes the Unix approach of using a general-purpose language to glue together other smaller, specialized programs to get a job done, rather than creating a single monolithic program. • It may be the only approach for times when you can’t code the client/server com- munication in Perl; for instance, when you have to talk with a server that requires a special client and there’s no alternative. The drawbacks of this approach are: • It’s dependent on another program outside the script. What if this program is not available, or its output format changes? Domain Name Service (DNS) | 167 • It’s slower, because it has to start up another process each time it wants to make a query. We could reduce this overhead by opening a two-way pipe to an nslookup process that stays running while we need it. This would take a little more coding skill, but it would be the right thing to do if we were going to continue down this path and further enhance this code. • We have less control. We’re at the external program’s mercy for implementation details. For instance, here nslookup (more specifically, the resolver library nslookup is using) is handling server timeouts, query retries, and appending a domain search list for us. Working with raw network sockets If you are a “power sysadmin,” you may decide calling another program is not accept- able. You might want to implement the DNS queries using nothing but Perl. This entails constructing network packets by hand, shipping them out on the wire, and then parsing the results returned from the server. The code in this section is probably the most complicated code you’ll find in this entire book; it was written by looking at the reference sources described later, along with several examples of existing networking code (including the module by Michael Fuhr/ Olaf Kolkman described in the next section). Here is a rough overview of what is going on in this approach: we query a DNS server by constructing a specific network packet header and packet contents, sending the packet to a DNS server, and then receiving and parsing the response from that server.† Each and every DNS packet (of the sort we are interested in) can have up to five distinct sections: Header Contains flags and counters pertaining to the query or answer (always present). Question Contains the question being asked of the server (present for a query and echoed in a response). Answer Contains all the data for the answer to a DNS query (present in a DNS response packet). Authority Contains information on the location from which an authoritative response may be retrieved. † For the nitty-gritty details, I highly recommend that you open RFC 1035 to the section entitled “Messages” and read along. 168 | Chapter 5: TCP/IP Name and Configuration Services Additional Contains any information the server wishes to return in addition to the direct an- swer to a query. Our program will concern itself strictly with the first three of these sections. We’ll be using a set of pack() commands to create the necessary data structure for a DNS packet header and packet contents. We’ll pass this data structure to the IO::Socket module that handles sending this data out as a packet. The same module will also listen for a response on our behalf and return data for us to parse (using unpack()). Conceptually, this process is not very difficult. There’s one twist to this process that should be noted before we look at the code. RFC 1035‡ (Section 4.1.4) defines two ways of representing domain names in DNS packets: uncompressed and compressed. The uncompressed representation places the full do- main name (for example, in the packet, and is nothing special. However, if the same domain name is found more than once in a packet, it is likely that a com- pressed representation will be used for all but the first mention. A compressed repre- sentation replaces the domain information (or part of it) with a two-byte pointer back to the first uncompressed representation. This allows a packet to mention host1, host2, and host3 in, without having to include the bytes for each time. We have to handle both representations in our code, hence the decompress() routine. Without further fanfare, here’s the code: use IO::Socket; use Data::Dumper; my $hostname = $ARGV[0]; my @servers = qw(nameserver1 nameserver2 nameserver3); # name of the name servers my $defdomain = ''; # default domain if not present my %results; foreach my $server (@servers) { $results{$server} = LookupAddress( $hostname, $server ); # populates %results } my %inv = reverse %results; # invert the result hash if ( scalar keys %inv > 1 ) { # see how many elements it has print "There is a discrepency between DNS servers:\n"; print Data::Dumper->Dump( [ \%results ], ['results'] ), "\n"; } sub LookupAddress { my ( $hostname, $server ) = @_; my $id = 0; my ( $lformat, @labels, $count, $buf ); ‡ RFC 1035 has been updated by RFC 1101, but not in a way that impacts this discussion. Domain Name Service (DNS) | 169 ### ### Construct the packet header ### my $header = pack( 'n C2 n4', ++$id, # query id 1, # qr, opcode, aa, tc, rd fields (only rd set) 0, # ra, z, rcode 1, # one question (qdcount) 0, # no answers (ancount) 0, # no ns records in authority section (nscount) 0 ); # no additional rr's (arcount) # if we do not have any separators in the name of the host, # append the default domain if ( index( $hostname, '.' ) == −1 ) { $hostname .= $defdomain; } # construct the qname section of a packet (domain name in question) for ( split( /\./, $hostname ) ) { $lformat .= 'C a* '; $labels[ $count++ ] = length; $labels[ $count++ ] = $_; } ### ### construct the packet question section ### my $question = pack( $lformat . 'C n2', @labels, 0, # end of labels 1, # qtype of A 1 ); # qclass of IN ### ### send the packet to the server and read the response ### my $sock = new IO::Socket::INET( PeerAddr => $server, PeerPort => 'domain', Proto => 'udp' ); $sock->send( $header . $question ); # we know the max packet size $sock->recv( $buf, 512 ); close($sock); ### ### unpack the header section 170 | Chapter 5: TCP/IP Name and Configuration Services ### my ( $id, $qr_opcode_aa_tc_rd, $ra_z_rcode, $qdcount, $ancount, $nscount, $arcount ) = unpack( 'n C2 n4', $buf ); if ( !$ancount ) { warn "Unable to lookup data for $hostname from $server!\n"; return; } ### ### unpack the question section ### # question section starts 12 bytes in my ( $position, $qname ) = decompress( $buf, 12 ); my ( $qtype, $qclass ) = unpack( '@' . $position . 'n2', $buf ); # move us forward in the packet to end of question section $position += 4; ### ### unpack all of the resource record sections ### my ( $rtype, $rclass, $rttl, $rdlength, $rname, @results ); for ( ; $ancount; $ancount-- ) { ( $position, $rname ) = decompress( $buf, $position ); ( $rtype, $rclass, $rttl, $rdlength ) = unpack( '@' . $position . 'n2 N n', $buf ); $position += 10; # this next line could be changed to use a more sophisticated # data structure - it currently concatenates all of the answers push( @results, join( '.', unpack( '@' . $position . 'C' . $rdlength, $buf ) ) ); $position += $rdlength; } # we sort results to deal with round-robin DNS responses # # we probably should use a custom sort routine to sort # them semantically, but in this case we're just looking for # the presence of different results from each DNS server return join( ', ', sort @results ); } # handle domain information that is "compressed" as per RFC 1035 # # we take in the starting position of our packet parse and return # the place in the packet we left off at the end of the domain name # (after dealing with the compressed format pointer) and the name we found sub decompress { my ( $buf, $start ) = @_; my ( $domain, $i, $lenoct ); Domain Name Service (DNS) | 171 # get the size of the response, since we're going to have to keep track of # where we are in that data my $respsize = length($buf); for ( $i = $start; $i <= $respsize; ) { $lenoct = unpack( '@' . $i . 'C', $buf ); # get length of label if ( !$lenoct ) { # 0 signals we are done with this section $i++; last; } if ( $lenoct == 192 ) { # we've been handed a pointer, so recurse $domain .= ( decompress( $buf, ( unpack( '@' . $i . 'n', $buf ) & 1023 ) ) )[1]; $i += 2; last; } else { # otherwise, we have a plain label $domain .= unpack( '@' . ++$i . 'a' . $lenoct, $buf ) . '.'; $i += $lenoct; } } return ( $i, $domain ); } Note that this code is not precisely equivalent to that from the previous example, be- cause we’re not trying to emulate all of the nuances of nslookup’s behavior (timeouts, retries, searchlists, etc.). When looking at the three approaches discussed here, be sure to keep a critical eye out for these subtle differences. The benefits of this approach are: • It isn’t dependent on any other programs. You don’t need to know the particulars of another programmer’s work. • It may be as fast as or faster than calling an external program. • It is easier to tweak the parameters of the situation (timeouts, etc.). The drawbacks of this approach are: • It’s likely to take longer to write and is more complex than the previous approach. • It requires more knowledge external to the direct problem at hand (i.e., you may have to learn how to put together DNS packets by hand, something we did not need to know when we called nslookup). • Our code does not deal with truncated DNS replies (if the reply is too large, most implementations fail over to giving a TCP response). 172 | Chapter 5: TCP/IP Name and Configuration Services • You may have to handle OS-specific issues yourself (these are hidden in the pre- vious approach by the work already done by the external program’s author). Using Net::DNS As mentioned in Chapter 1, one of Perl’s real strengths is the support of a large com- munity of developers who churn out code for others to reuse. If there’s something you need to do in Perl that seems universal, chances are good that someone else has already written a module to handle it. In our case, we can make use of Michael Fuhr’s excellent Net::DNS module (now maintained by Olaf Kolkman) to make our job simpler. For this task, we simply have to create a new DNS resolver object, configure it with the name of the DNS server we wish to use, ask it to send a query, and then use the supplied methods to parse the response: use Net::DNS; my $hostname = $ARGV[0]; my @servers = qw(nameserver1 nameserver2 nameserver3 nameserver4); my %results; foreach my $server (@servers) { $results{$server} = LookupAddress( $hostname, $server ); # populates %results } my %inv = reverse %results; # invert the result hash if ( scalar keys %inv > 1 ) { # see how many elements it has print "There is a discrepency between DNS servers:\n"; use Data::Dumper; print Data::Dumper->Dump( [ \%results ], ['results'] ), "\n"; } # only slightly modified from the example in the Net::DNS manpage sub LookupAddress { my ( $hostname, $server ) = @_; my $res = new Net::DNS::Resolver; $res->nameservers($server); my $packet = $res->query($hostname); if ( !$packet ) { warn "Unable to lookup data for $hostname from $server!\n"; return; } my (@results); foreach my $rr ( $packet->answer ) { push( @results, $rr->address ); } return join( ', ', sort @results ); } Domain Name Service (DNS) | 173 The benefits of this approach are: • The code is legible. • It is often faster to write. • Depending on how the module you use is implemented (is it pure Perl or is it glue to a set of C or C++ library calls?), the code you write using this module may be just as fast as calling an external compiled program. • It is potentially portable, depending on how much work the author of the module has done for you. Any place this module can be installed, your program can run. • As in the first approach we looked at, writing code can be quick and easy if someone else has done the behind-the-scenes work for you. You don’t have to know how the module works; you just need to know how to use it. • Code reuse. You are not reinventing the wheel each time. The drawbacks of this approach are: • You’re back in the dependency game. You need to make sure the module will be available for your program to run, and you need to trust that the module writer has done a decent job. • If there is a bug in the module and the original author goes missing, you may wind up becoming the maintainer of the module. • There may not be a module to do what you need, or it may not run on your oper- ating system of choice. More often than not, using a prewritten module is my preferred approach. However, any of the approaches discussed here will get the job done. TMTOWTDI, so go forth and do it! DHCP The Dynamic Host Configuration Protocol isn’t a TCP/IP name service, but it is enough of a kissing cousin that it belongs in this chapter next to DNS. DNS lets us find the IP address associated with a hostname (or hostname associated with an IP address, in the case of a reverse lookup). DHCP lets a machine dynamically retrieve its network con- figuration information (including its IP address) given its Ethernet address. It is a more general and robust successor to the BOOTP and RARP protocols that used to serve a similar purpose in a more limited way. I glossed over the complex parts of how DNS works in the previous section,* but I’m not going to be able to hand wave past the slightly more involved interaction that takes * For example, we won’t talk about zone transfers, non-ASCII names, or even what happens if the answer to a question can’t fit in a single UDP packet. Paul Vixie wrote an excellent article on this topic for the April 2007 issue of ACM Queue magazine called “DNS Complexity.” It’s well worth a read. 174 | Chapter 5: TCP/IP Name and Configuration Services place between a DHCP server and a DHCP client. Let’s get that out of the way right now, before we even think about bringing Perl into the picture. In the previous section, we could pretty much say, “DNS client asks a DNS server a question and gets an answer. Done.” The worst-case scenario for the sort of queries we did might have been “DNS client asks a question, gets an answer, then has to ask another server the same question.” The DHCP dance is more interesting. Here’s roughly the conversation that goes on: DHCP Client (to everyone): Hey, is anyone out there? I need an address and some other configuration information. DHCP Server directly to DHCP Client: Request acknowledged by server at IP address {blah}. I can let you use the following IP address and other information: {blah, blah, blah}. DHCP Client (to everyone): OK, I’d like to use the IP address and other infor- mation that the server at IP address {blah} offered. DHCP Server to DHCP Client: OK, acknowledged. You are welcome to use that configuration for {some amount of time}. [half that amount of time passes] DHCP Client to Server: Hi, I’m using this IP address and configuration informa- tion: . Can I continue to use it? DHCP Server to Client: OK, acknowledged. You are welcome to use that con- figuration for {some amount of time}. [client prepares to leave the network] (optionally) DHCP Client to Server: OK, done with this IP address and configuration. There are a few variations on this conversation. For example, a client can remember its last address and lease duration (the amount of time the server said it could use the information) and ask to use it again in a broadcasted version of the third step. In that case, the server can quickly acknowledge the request, and the client is off and running with its previous address. Other variations occur when either the server or the client isn’t in as sunny a mood as in the example conversation. Sometimes either side will decline a request or an offer, and then a new negotiation round occurs. For all of the gripping details, you should read the clearly written DHCP RFC (2131 as of this writing) or the The DHCP Handbook mentioned at the end of this chapter. The interaction between a DHCP server and a DCHP client differs from our previous name server examples in a number of ways. Besides the number of steps, we also see the signs of: • A negotiated handshake. The server and client have to agree that they are going to talk to each other and then agree on the configuration the client will use. Both sides need to let each other know that they agree. DHCP | 175 • Persistent state. Once the handshake has taken place, the server keeps track of the agreement for the duration of the server-specified time (the client’s lease). Halfway into the lease, the client will (under most circumstances) attempt to renew it. Of the protocols mentioned in this chapter, only WINS has a similar notion. Clients register with a WINS server and are required to re-register periodically so the server can maintain correct mapping in- formation for them. DHCP is used for more than just name-to-address mapping and the rules of interaction are a little different, but DHCP and WINS are pretty similar in this regard. These two factors make our decisions around the programming of DHCP scripts a little more interesting. Most of these decisions come down to questions of how “nice” or compliant we want our scripts to be. Let’s look at two example tasks so you can see what I mean. Active Probing for Rogue DHCP Servers In Chapter 13 we will work on a passive detector for rogue (i.e., unofficial/unwanted) DCHP servers using network sniffing. We’ll construct something similar here, using a more active approach. The key to this approach is the first part of the sample DHCP conversation outlined previously. The first step for a client that wishes to obtain con- figuration information is a “hey, anybody out there?” broadcast. Any DHCP server that can hear this request is supposed to respond with a DHCP offer for the information it is configured to provide. Before we go much further with this example, I have to jump in with a qualification about servers being “supposed to respond” with certain information. DHCP servers—at least the good ones—are highly con- figurable. They can be told to respond to broadcasts only from known Ethernet addresses, network subnets, vendor classes (more on those later), and so on. All other requests are ignored, with maybe just a note left in the log file. The code we’re about to see is designed to catch servers that aren’t con- figured properly. We’re looking for the servers that will answer requests indiscriminately (because after all, they are the ones that can cause the most trouble). These are the rogue servers likely to still have bad default settings. Though this code is probably too blunt to catch more subtle configu- ration errors, you could certainly sharpen it so it could find the more specific problems in almost-properly-configured severs as well. 176 | Chapter 5: TCP/IP Name and Configuration Services For this task we’re just interested in who answers our broadcast. We mostly don’t care what they say, though that information can be helpful when tracking down the host in question. I mention this because some of the information we may send (like a fake Ethernet address) will obviously be bogus. Given how little we care about the contents of the query we make or the response we receive, you can probably guess that we’re going to be equally laissez-faire about having the script comply with the rest of the DHCP protocol. We’re not going to bother to continue negotiating for a lease with the server that responds, because we don’t need one. All we want to do is provoke a response from any server willing to talk to us. As a side goal, we’re also going to attempt to avoid gumming any legitimate DHCP servers we run, because denial of service (DOS) attacks are seldom a good idea. Now let’s get to the code. The code we need is quick to write, thanks to the Net::DHCP::Packet module by Stephan Hadinger. This module lets us construct and deconstruct DHCP packets using an easy OOP syntax. We’ll also be using the IO::Socket::INET module by Graham Barr to make the UDP sending and receiving code simpler (with one gotcha that I’ll point out later). Here’s our code, with explication to follow: use IO::Socket::INET; use Net::DHCP::Packet; use Net::DHCP::Constants; my $br_addr = sockaddr_in( '67', inet_aton('') ); my $xid = int( rand(0xFFFFFFFF) ); my $chaddr = '0016cbb7c882'; my $socket = IO::Socket::INET->new( Proto => 'udp', Broadcast => 1, LocalPort => '68', ) or die "Can't create socket: $@\n"; my $discover_packet = Net::DHCP::Packet->new( Xid => $xid, Chaddr => $chaddr, Flags => 0x8000, DHO_DHCP_MESSAGE_TYPE() => DHCPDISCOVER(), DHO_HOST_NAME() => 'Perl Test Client', DHO_VENDOR_CLASS_IDENTIFIER() => 'perl', ); $socket->send( $discover_packet->serialize(), 0, $br_addr ) or die "Error sending:$!\n"; my $buf = ''; $socket->recv( $buf, 4096 ) or die "recvfrom() failed:$!"; my $resp = new Net::DHCP::Packet($buf); DHCP | 177 print "Received response from: " . $socket->peerhost() . "\n"; print "Details:\n" . $resp->toString(); close($socket); Let’s take a quick walk through the code. We start by defining the destination for our packets (a broadcast address†), and a couple of other constants I’ll mention in a moment. We then create the socket we’ll be using to send and receive packets. This socket definition specifies the protocol, the source port, and a request to set the broadcast flag on that socket. You’ll note that it doesn’t specify the destination for the packets ($br_addr) defined earlier in the constructor. You can’t use connect() to receive a broadcast response, as detailed in the sidebar “Thank You, Lincoln Stein (Listening to Broadcasts Using IO::Socket)”, so this is broken out as a separate step. You should also note that, thanks to the source and destination port numbers in the code, we’re going to have to run our DHCP examples with some sort of administrative privileges (e.g., via sudo on Unix/Mac OS X). If we don’t, we’re in for the Permission Denied blues. Thank You, Lincoln Stein (Listening to Broadcasts Using IO::Socket) When writing this example, I spent a good part of a day banging my head against a particular problem. For the life of me, I couldn’t get IO::Socket::INET to let me send to a broadcast address and then listen to the responses using the same socket, even though I knew this was possible. I had code that could listen for packets and code that could send broadcasts (as verified by Wireshark, a network sniffer), but I couldn’t get the pieces to play nicely together. In desperation, I did two things: I re-read the appropriate sections of Stein’s excellent book Network Programming with Perl (Addison-Wesley) and traced the IO::Socket::INET module’s operations in a debugger really carefully. In Stein’s book, I found this paragraph about the connect() call being used for UDP sockets (p. 532): A nice side effect of connecting a datagram socket is that such a socket can receive messages only from the designated peer. Messages sent to the socket from other hosts or from other ports on the peer host are ignored.... Servers that typically must receive messages to multiple clients should generally not connect their sockets. My trace of IO::Socket::INET showed that it indeed did connect() its socket if a remote address was specified (i.e., using the PeerAddr argument in the new() call), even if the remote address was the broadcast address. That connected socket refused to listen to packets from any other host. The code presented in this section intentionally does not specify PeerAddr when it wants to listen to responses to broadcasted packets, even though it seems like the obvious † Eagle-eyed network administrators will note that we’re wimping out here by sending to the “all-ones” broadcast address rather than the one specific to our subnet. Figuring out that address would add a bit of distraction in this example. There are a number of modules (such as NetAddr::IP, Net::Interface, and IO::Interface) that can help if you want to be more accurate in your target. 178 | Chapter 5: TCP/IP Name and Configuration Services approach. My thanks to Lincoln Stein’s book for helping me figure this out before I went batty. Once we have the socket to be used for transiting data, it’s time to bake some packets. The next line constructs a DHCP packet to our specification. Let’s look at each flag in that constructor call, because they’re all pretty important for understanding how things work: Xid This is the transaction ID for the discussion we’re about to have. The client picks a random ID for both it and the server to include in each packet being sent. This lets them both know that they are both referring to the same conversation. Chaddr This is the client hardware address (i.e., Ethernet or MAC address). In this example I’ve done the simple but potentially dangerous thing of taking my laptop’s Ethernet address and incrementing some of the digits. It would probably be better to either use a known idle address (like that broken Ethernet card you have sitting on your shelf) or determine the current address of your machine (assuming it isn’t using DHCP at the moment).‡ Flags The DHCP specification allows you to set a flag to request that the response to the packet you’re sending be sent as a broadcast (as opposed to a unicast reply) as well. In this case we’re asking all servers that reply to do so using a broadcast response just to be safe. DHO_DHCP_MESSAGE_TYPE This option sets the type of DHCP packet being sent. Here’s where we say we’re sending out the initial DHCP DISCOVER packet. DHO_HOST_NAME Though not necessary for this particular task, this option lets our client self-identify using a name of our choice. Using this option can help make network sniffer traces easier to follow (because we can clearly identify our packets). Later, in a more complex example, we’ll actually reserve a lease. In that case, it will be handy when we look at the lease database of a server (e.g., in a wireless router). It’s much easier to tell which client is yours in the list if the client has supplied a recognizable name. DHO_VENDOR_CLASS_IDENTIFIER Another optional flag that isn’t strictly necessary, the vendor class is a concept detailed in the DHCP RFC that can help make server configuration easier. It allows ‡ This ostensibly simple task isn’t as easy as it sounds. There is no truly portable way to get the Ethernet address of an interface on your machine. The closest you can come is using a module like Net::Address::Ethernet or Net::Ifconfig::Wrapper (which basically call ifconfig or ipconfig for you), IO::Interface::Simple (which attempts to use some of the system libraries to find the information), or even Net::SNMP::Interfaces pointed at the local host. None of these solutions works all of the time, so caveat implementor. DHCP | 179 you to break up your network devices into different vendor classes for the purpose of assigning configurations. For example, you might want all of your Windows machines to receive a configuration that points them to your Windows Server 2003 DNS servers. If you assign them all to the same vendor class, the better DHCP servers will allow you to apply configuration directives to that set. If specifying a DHCP packet is easy thanks to Net::DHCP::Packet, actually sending it is even easier. In the next line of code, we just call send() on a massaged (serialized) version of that packet and send it to the broadcast address and port previously defined. The middle argument of the send() call is for optional flags—we set it to 0, which means we’ll take the defaults. Immediately after sending the packet, we call recv() to listen for a response. There are two important things to note here: • To keep the example code as simple as possible we call recv() only once, which means we’re only going to listen for a single packet in response to our broadcast. Ideally we would loop so we could return to listening for packets. • A simple recv() call like this will block (i.e., hang) forever until it receives data. Again, we didn’t do anything fancy here for simplicity’s sake. There are ways to get around this limitation, such as using alarm() or IO::Select. See Lincoln Stein’s book Network Programming with Perl (Addison-Wesley) or the examples in Perl Cookbook (O’Reilly) for more details. For the purposes of illustration, we’ll assume that everything goes as planned and we do receive a packet. We then take that packet apart for display using Net::DHCP::Packet and print it. Here’s some sample output showing what we might see at this point: Received response from: Details: op = BOOTREPLY htype = HTYPE_ETHER hlen = 6 hops = 0 xid = 912c020f secs = 0 flags = 8000 ciaddr = yiaddr = siaddr = giaddr = chaddr = 0016cbb7c882 sname = file = Options : DHO_DHCP_MESSAGE_TYPE(53) = DHCPOFFER DHO_DHCP_SERVER_IDENTIFIER(54) = DHO_DHCP_LEASE_TIME(51) = 86400 DHO_SUBNET_MASK(1) = DHO_ROUTERS(3) = 180 | Chapter 5: TCP/IP Name and Configuration Services DHO_DOMAIN_NAME_SERVERS(6) = padding [270] = 000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000 That’s a lovely offer for a lease. The next logical thing for our code to do (that it doesn’t do yet) would be to check to make sure the DHCP server offering this lease is one of our authorized servers. If not, the hunt is on! Monitoring Legitimate DHCP Servers We can develop this last piece of code even further into yet another useful script. In the previous example we started the DHCP handshake process with a DHCPDISCOVER and received a DHCPOFFER, but we didn’t consummate the transaction with the last step in the process. Leaving off at this point means we didn’t hang in there long enough to see if the server would have actually given us a lease. What if we wanted to see whether our legitimate DHCP server was properly handing out leases and other configuration information? To do that, we’d have to follow the handshake to its natural conclusion. Here’s some example code that will request a lease, print out the final packet when it receives it, and then (because we’re well-mannered programmers) release that lease. I’ve highlighted the interesting bits that we’ll talk about after you get a chance to see the new code: use IO::Socket::INET; use Net::DHCP::Packet; use Net::DHCP::Constants; my $socket = IO::Socket::INET->new( Proto => 'udp', Broadcast => 1, LocalPort => '68', ) or die "Can't create socket: $@\n"; my $br_addr = sockaddr_in( '67', inet_aton('') ); my $xid = int( rand(0xFFFFFFFF) ); my $chaddr = '0016cbb7c882'; my $discover_packet = Net::DHCP::Packet->new( Xid => $xid, Chaddr => $chaddr, Flags => 0x8000, DHO_DHCP_MESSAGE_TYPE() => DHCPDISCOVER(), DHO_HOST_NAME() => 'Perl Test Client', DHO_VENDOR_CLASS_IDENTIFIER() => 'perl', DHCP | 181 ); $socket->send( $discover_packet->serialize(), 0, $br_addr ) or die "Error sending:$!\n"; my $buf = ''; $socket->recv( $buf, 4096 ) or die "recvfrom() failed:$!"; my $resp = new Net::DHCP::Packet($buf); my $request_packet = Net::DHCP::Packet->new( Xid => $xid, Chaddr => $chaddr, DHO_DHCP_MESSAGE_TYPE() => DHCPREQUEST(), DHO_VENDOR_CLASS_IDENTIFIER() => 'perl', DHO_HOST_NAME() => 'Perl Test Client', DHO_DHCP_REQUESTED_ADDRESS() => $resp->yiaddr(), DHO_DHCP_SERVER_IDENTIFIER() => $resp->getOptionValue( DHO_DHCP_SERVER_IDENTIFIER() ), ); $socket->send( $request_packet->serialize(), 0, $br_addr ) or die "Error sending:$!\n"; $socket->recv( $buf, 4096 ) or die "recvfrom() failed:$!"; $resp = new Net::DHCP::Packet($buf); print $resp->toString(); my $dhcp_server = $resp->getOptionValue( DHO_DHCP_SERVER_IDENTIFIER() ); close($socket); my $socket = IO::Socket::INET->new( Proto => 'udp', LocalPort => '68', PeerPort => '67', PeerAddr => $dhcp_server, ) or die "Can't create socket: $@\n"; my $release_packet = Net::DHCP::Packet->new( Xid => $xid, Chaddr => $chaddr, DHO_DHCP_MESSAGE_TYPE() => DHCPRELEASE(), ); $socket->send( $release_packet->serialize() ) or die "Error sending:$!\n"; You probably have the gist of what’s going on here, thanks to the last example. Let’s look at the ways this code expands on those ideas. The first chunk of highlighted code shows two flags we haven’t seen before: 182 | Chapter 5: TCP/IP Name and Configuration Services DHO_DHCP_REQUESTED_ADDRESS This is the address our client will request from the server. As part of the handshake, we’re just parroting back the address provided to us in the reply from the DHCP server. The DHCP server sends us a packet with the yiaddr field set to a suggested address that it could give us (i.e., “your Internet address could be...”). In our packet, we say, “OK, we will use the address you’re suggesting.” DHO_DHCP_SERVER_IDENTIFIER When there are multiple DHCP servers on the network that can hear broadcasts and respond to them, it is important for the client to be able to indicate which server’s response it is choosing to answer. It does this using the identifier (i.e., the IP address) of the server in this field. The second piece of interesting code demonstrates the use of a new socket for the release of the lease. We need a new socket because we’re done broadcasting our replies and instead need to send a unicast response to the server. We get the address of that server from the DHO_DHCP_SERVER_IDENTIFIER option set in the server’s previous response to us. The final addition in this example is the snippet that sends a request to relinquish the previously assigned lease. You might note that this packet has the fewest options/flags of any we’ve seen to date: our client just has to reference the transaction ID we’ve been using and the client’s hardware (Ethernet) address in order to let the server know which lease to release. With that we’ve seen one way to test a full DISCOVER-OFFER-REQUEST-RELEASE cycle. Clearly we could get even fancier and test lease renewals or closely check over the results we’re receiving from our servers to make sure they make sense. If you’d like to do the former, where acting like a real DHCP client is important, I should mention in the interest of full disclosure that there is a module, Net::DHCPClientLive by Ming Zhang, that can make your life easier. It basically lets you test the various states a DHCP client moves through (e.g., “lease expired, attempt to renew lease”) and will automatically send and receive the necessary packets to reach that state. Use it when you really need to beat up on your DHCP servers for testing purposes. Module Information for This Chapter Module CPAN ID Version Rcs CFRETER 1.05 Net::NIS RIK 0.34 Data::Dumper (ships with Perl) ILYAM 2.121 IO::Socket::INET (ships with Perl) GBARR 1.2301 Net::DNS OLAF 0.59 Net::DHCP::Packet SHADINGER 0.66 Module Information for This Chapter | 183 References for More Information DNS and BIND, Fifth Edition (, by Paul Al- bitz and Cricket Liu (O’Reilly) Managing NFS and NIS, Second Edition (, by Mike Eisler et al. (O’Reilly) The DHCP Handbook, Second Edition, by Ralph Droms and Ted Lemon (Sams) Network Programming with Perl, by Lincoln Stein (Addison-Wesley) Perl Cookbook, Second Edition (, by Tom Christiansen and Nathan Torkington (O’Reilly) RFC 849: Suggestions For Improved Host Table Distribution, by Mark Crispin (1983) RFC 881: The Domain Names Plan and Schedule, by J. Postel (1983) RFC 882: Domain Names: Concepts And Facilities, by P. Mockapetris (1983) RFC 1035: Domain Names: Implementation And Specification, by P. Mockapetris (1987) RFC 1101: NS Encoding of Network Names and Other Types, by P. Mockapetris (1989) RFC 2131: Dynamic Host Configuration Protocol, by R. Droms (1997) 184 | Chapter 5: TCP/IP Name and Configuration Services CHAPTER 6 Working with Configuration Files Let us consider the lowly config file. For better or worse, config files are omnipresent not just in the lives of sysadmins, but in the lives of anyone who ever has to configure software before using it. Yes, GUIs and web-based point-and-click festivals are becom- ing more prevalent for configuration, but even in those cases there’s often some piece of configuration information that has to be twiddled before you get to the nifty GUI installer. From the Perl programmer’s point of view, the evolutionary stages of a program usually go like this. First, the programmer writes the roughest and simplest of scripts. Take for example the following script, which reads a file and adds some text to lines that begin with the string hostname: before writing the data out to a second file: open my $DATA_FILE_H, '<', '/var/adm/data' or die "unable to open datafile: $!\n"; open my $OUTPUT_FILE_H, '>', '/var/adm/output' or die "unable to write to outputfile: $!\n"; while ( my $dataline = <$DATA_FILE_H> ) { chomp($dataline); if ( $dataline =~ /^hostname: / ) { $dataline .= ''; } print $OUTPUT_FILE_H $dataline . "\n"; } close $DATA_FILE_H; close $OUTPUT_FILE_H; That’s quickly replaced by the next stage, the arrival of variables to represent parts of the program’s configuration: my $datafile = '/var/adm/data'; # input data filename my $outputfile = '/var/adm/output'; # output data filename my $change_tag = 'hostname: '; # append data to these lines my $fdqn = ''; # domain we'll be appending open my $DATA_FILE_H, '<', $datafile or die "unable to open $datafile: $!\n"; 185 open my $OUTPUT_FILE_H, '>', $outputfile or die "unable to write to $outputfile: $!\n"; while ( my $dataline = <$DATA_FILE_H> ) { chomp($dataline); if ( $dataline =~ /^$change_tag/ ) { $dataline .= $fdqn; } print $OUTPUT_FILE_H $dataline . "\n"; } close $DATA_FILE_H; close $OUTPUT_FILE_H; Many Perl programs happily remain at this stage for their whole lives. However, more experienced programmers will recognize that code like this is fraught with potential peril: problems can arise as development continues and the program gets bigger and bigger, perhaps being handed off to other people to maintain. These problems will manifest the first time someone naïvely adds code deep within the program that modifies $change_tag or $fdqn. All of a sudden, the program output changes in an unexpected and unwanted way. In a small code snippet it is easy to spot the connection between $change_tag or $fdqn and the desired results, but it can be much trickier in a program that scrolls by for screen after screen. One approach to fixing this problem would be to rename variables like $fdqn to some- thing more obscure, like $dont_change_this_value_yesiree_bob, but that’s a bad idea. Besides consuming far too many of the finite number of keystrokes you’ll be able to type in your lifetime, it wreaks havoc on code readability. There are a number of data- hiding tricks we could play instead (closures, symbol table manipulation, etc.), but they don’t help with readability either and are more complex than necessary. The best idea is to use something similar to the use constants pragma (see the sidebar “Constant As the Northern Star” on page 187) to make the variables read-only: use Readonly; # we've upcased the constants so they stick out # note: this is the Perl 5.8.x syntax, see the Readonly docs for using # Readonly with versions of Perl older than 5.8 Readonly my $DATAFILE => '/var/adm/data'; # input data filename Readonly my $OUTPUTFILE => '/var/adm/output'; # output data filename Readonly my $CHANGE_TAG => 'hostname: '; # append data to these lines Readonly my $FDQN => ''; # domain we'll be appending open my $DATA_FILE_H, '<', $DATAFILE or die "unable to open $DATAFILE: $!\n"; open my $OUTPUT_FILE_H, '>', $OUTPUTFILE or die "unable to write to $OUTPUTFILE: $!\n"; while ( my $dataline = <$DATA_FILE_H> ) { chomp($dataline); 186 | Chapter 6: Working with Configuration Files if ( $dataline =~ /^$CHANGE_TAG/ ) { $dataline .= $FDQN; } print $OUTPUT_FILE $dataline . "\n"; } close $DATA_FILE_H; close $OUTPUT_FILE_H; Constant As the Northern Star Why not actually “use constants” instead? The Readonly module’s documentation points out a number of reasons, the three most compelling of which are: 1. The ability to interpolate Readonly variables into strings (e.g., print "Constant set to $CONSTANT\n") 2. The ability to lexically scope Readonly variables (e.g., Readonly my $constant => "fred") so they can be present in only the scope you desire 3. Unlike with use constant, once a Readonly variable is defined, attempts to redefine it are rebuffed Now that we’ve seen the nec plus ultra of storing configuration information within the script,* we’ve hit a wall: what happens when we decide to write a second or third script that needs similar configuration information? Copy and paste is the wrong answer. Simply duplicating the same information into a second script might seem harmless, but this would lead to Multiple Sources of Truth™.† That is, someday, something will change, and you’ll forget to update one of these files, and then you’ll spend hours trying to figure out what’s gone wrong. This is the first step on to the road away from Oz and toward an unpleasant encounter with the flying monkeys and an unhappy lady with a broomstick. Don’t do it. The right answer is probably to create some sort of config file (or something more sophisticated, which we will touch on later in the chapter). The next choice once you’ve reached the decision to use a config file is: what format? The answer to that question is similar to the old joke, “The wonderful thing about standards is there are so many to choose from!” Discussions of which formats are best are usually based on some mishmash of religion, politics, and personal aesthetic taste. Because I’m a flaming pluralist, we’re going to take a look at how to deal with several of the most common formats, and I’ll leave you to choose the best one for your application. * There are some games we could play with __DATA__, but in general, keeping the configuration information at the beginning of the script is better form. † To use Hugh Brown’s special phrase for the situation. Working with Configuration Files | 187 Configuration File Formats We’ll look at four ways of storing configuration data in this chapter: in binary format, as naked delimited data, as key/value pairs, and using a markup format such as XML or YAML. I’ll try to give you my humble opinion about each to help you with your decision process. Binary The first kind of configuration file we’re going to look at is my least favorite, so let’s get it out of the way quickly. Some people choose to store their configuration data on disk as basically a serialized memory dump of their Perl data structures. There are several ways to write this data structure to disk, including the old warhorse Storable: use Storable; # write the config file data structure out to $CONFIG_FILE store \%config, $CONFIG_FILE; # use nstore() for platform independent file # later (perhaps in another program), read it back in for use my $config = retrieve($CONFIG_FILE); If you need something that is pure Perl, DBM::Deep is another good choice. It has the benefit of producing data files that aren’t platform-specific by default (though Storable’s nstore() method can help with that): use DBM::Deep; my $configdb = new DBM::Deep 'config.db'; # store some host config info to that db $configdb->{hosts} = { 'agatha' => '', 'gilgamesh' => '', 'tarsus' => '', }; # (later) retrieve the names of the hosts we've stored print join( ' ', keys %{ $configdb->{hosts} } ) . "\n"; Files in a binary format are typically really fast to read, which can be quite helpful if performance is a concern. Similarly, there’s something elegant about having the infor- mation stay close to the native format (i.e., a Perl data structure you’re going to traverse in memory) for its entire lifespan, rather than transcoding it to and from another rep- resentation through a myriad of parsing/slicing/dicing steps. So why is this my least favorite kind of config file? To me, the least palatable aspect is the opaque nature of the files created. I much prefer my config files to be human- readable whenever possible. I don’t want to have to rely on a special program to decode the information (or encode it, when the data gets written in the first place). Besides that 188 | Chapter 6: Working with Configuration Files visceral reaction, the use of binary formats also means you can’t operate on the data using other standard tools at your disposal, like grep.‡ Luckily, if you’re looking for speed, there are other alternatives, as you’ll see in a moment. Naked Delimited Data Also in the category of formats I tend to dislike are those that simply present a set of data in fields delimited by some character. The /etc directory on a Unix box is lousy with them: passwd, group, and so on. Comma- or Character-Separated Value (CSV, take your pick of expansions) files are in the same category. Reading them in Perl is pretty easy because of the built-in split() operator: use Readonly; Readonly my $DELIMITER => ':'; Readonly my $NUMFIELDS => 4 ; # open your config file and read in a line here # now parse the data my ( $field1, $field2, $field3, $field4, $excess ) = split $DELIMITER, $line_of_config, $NUMFIELDS; For CSV files, a number of helpful modules can handle tricky situations like escaped characters (e.g., when commas are used in the data itself). Text::CSV::Simple, a wrapper around Text::CSV_XS, works well: use Text::CSV::Simple; my $csv_parser = Text::CSV::Simple->new; # @data will then contain a list of lists, one entry per line of the file my @data = $csv_parser->read_file($datafile); This data format is also on my “least favored” list. Unlike the binary format, it has the benefit of being human-readable and parsable by standard tools; however, it also has the drawback of being easily human-misunderstandable and mangle-able. Without a good memory or external documentation, it is often impossible to understand the con- tents of the file (“what was the 7th field again?”), making it susceptible to fumble- fingering and subtle typos. It is also field-order fragile. And if that doesn’t convince you not to use this format, the more you work with it, the more you’ll find that individual parsers in different applications have different ideas on how to handle commas, quotes, and carriage returns within the values. CSV data seems interoperable on the surface, but it often requires a deeper understanding of the quirks of the programs producing or consuming it. ‡ As Mark Pilgrim once said, “Never trust a format you can’t edit in Emacs or vi” ( 11/tools-affect-co.html#comment-2725). Configuration File Formats | 189 Key/Value Pairs The most common format is the key {something} value style, where {something} is usually whitespace, a colon, or an equals sign. Some key/value pair formats (e.g., .ini files) throw in other twists, like section names that look like this: [server section] {setting}={value} {setting}={value} or configuration scopes (as in Apache’s configuration file): {setting} {value} {setting} {value} ... Dealing with key/value pair formats using Perl modules is initially difficult, because there are so many choices. As of this writing, there are at least 26 modules in this category on CPAN. How do you pick which module to use? The first step is to ask yourself a number of questions that will help define your needs and winnow down the contenders. To start, you’ll want to consider just how complex you want the configuration file to be: Will simple .ini files work for you? More complex .ini files? Apache style? Extended Apache style? Do you need sections? Do you need scoped directives? Want to write your own grammar representing the format? Next, consider how you would like to interact with the configuration information: Want the module to hand you back a simple data structure or an object represent- ing the information? Prefer to treat things like magical tied hashes or Perl constants? Does the information you get back have to come back in the same order as it is listed in the config file? Would you be happy if the module figured out the config file format for you? Finally, think about what else is important to you: Do you care how quickly the configuration is parsed or how much memory the parsing process takes? Should it handle caching of the config for fast reload? Do you want to be able to cascade the configs (i.e., have a global config that can include other config files)? Should the configuration data be validated on parsing? Should the module both read and write config files for you? The answers to each of these questions will point at a different module or set of modules available for your use. I don’t have space to discuss all of the modules out there, so let’s look at three of particular merit. Config::Std is Damian Conway’s config parsing module. It has the distinction of being able to read the configuration file and then update it with section order preserved and 190 | Chapter 6: Working with Configuration Files comments still intact. The file format it uses looks much like that of an .ini file, so it should be pretty easy for most people to understand on first sight. Here’s an example of the module in action (note that the examples in this section will be very simple— read: boring—because the modules are all designed to make the process of dealing with config files simple): use Config::Std; read_config 'config.cfg' => my %config; # now work with $config{Section}{key}... ... # and write the config file back out again write_config %config; In Conway’s book Perl Best Practices ( (O’Reilly), he suggests that if you need something more sophisticated than his simple Config::Std format can provide, Thomas Linden’s Config::General can oblige. It han- dles files in the Apache config file family and has a much richer syntax. Actual use of the module isn’t any more complex than use of Config::Std: use Config::General; my %config = ParseConfig( -ConfigFile => 'rcfile' ); # now work with the contents of %config... ... # and then write the config file back out again SaveConfig( 'configdb', \%config ); If Config::General doesn’t give you enough bells and whistles, there is always Config::Scoped, by Karl Gaissmaier. This module parses a similarly complex format that includes scoped directives (essentially the ones used by BIND or the ISC DHCP server), can check the data being parsed, will check the permissions of the config file itself, and includes caching functionality. This caching functionality allows your pro- gram to parse the more complex format once and then quickly load in a binary repre- sentation of the format on subsequent loads if the original file hasn’t changed. This gives us the speed we coveted from the first kind of file we looked at and the readability of the file formats discussed in this section. It doesn’t, however, offer an easy way to programmatically update an existing configuration file, like some of the other modules we’ve seen. Here’s a small snippet that shows how to use the caching functionality: use Config::Scoped; my $parser = Config::Scoped->new( file => 'config.cfg' ); my $config = $parser->parse; # store the cached version on disk for later use $parser->store_cache( cache => 'config.cfg.cache' ); # (later, in another program... we load the cached version) my $cfg = Config::Scoped->new( file => 'config.cfg' )->retrieve_cache; Configuration File Formats | 191 If you are the type of person that likes to smelt your own bits, there are also a number of other modules, such as Config::Grammar by David Schweikert, that allow you to define your own grammar to represent the configuration file format. I tend not to like creating custom formats if I can help it for maintainability purposes, but if this is a requirement, modules like Schweikert’s can oblige. Markup Languages Each of the formats we’ve seen so far is limited in some way. Some lack readability, while others lack extensibility. Most have a certain level of unpredictability in their parsing: there’s either a lack of precision in the specification or a level of rigidity to the format that can make it hard to tell whether you have parsed the data correctly. For example, with CSV files, the quoting of delimiters is handled differently depending on the parser. With binary files, one wrong element in the unpack() template of your pro- gram can cause it to happily start reading in garbage without any indication to the parser that something has gone wrong. Markup languages, when implemented and used intelligently, can overcome these concerns. XML When I started work on the first edition of this book it was clear that XML was an up- and-coming technology that deserved sysadmins’ attention, and hence it found its way into the book.* In the intervening years XML has worked its way into many of the nooks and crannies of system administration and data handling, to the point where having some facility with it is probably essential. Given the importance of XML, we’re going to spend a considerable part of this chapter discussing how to work with it. One place where XML has started to become more prevalent is in configuration files. There are in fact a variety of XML dialects (that we won’t discuss here), including DCML, NetML, and SAML, that are hoping to become the de facto formats for different parts of the configuration management space. Before we dig into the mechanics of using XML it will be worthwhile to look at why it works well in this context. XML has a few properties that make it a good choice for configuration files: • XML is a plain-text format, which means we can use our usual Perl bag o’ tricks to deal with it easily. • When kept simple (because a complex/convoluted XML document is as inscrut- able as one in any other format), XML is self-describing and practically self- documenting. With a character-delimited file like /etc/passwd, it is not always * Most of the XML discussion in the first edition was presented using account maintenance as a backdrop, but as the use of XML matured in the world of system administration it became clear that this chapter would be a better home for this discussion. 192 | Chapter 6: Working with Configuration Files easy to determine which part of a line represents which field. With XML, this is never a problem because an obvious tag can surround each field. • With the right parser, XML can also be self-validating. If you use a validating parser, mistakes in an entry’s format will be caught right away, since the file will not parse correctly according to its document type definition (DTD) or schema. Even without a validating parser, any XML parser that checks for well-formedness will catch many errors.† • XML is flexible enough to describe virtually any set of text information you would ever desire to keep. The freedom to define almost arbitrary tags lets it be as de- scriptive as you’d like. This flexibility means you can use one parser library to get at all of your data, rather than having to write a new parser for each different format. Here’s an example of what I mean when I say that XML can be self-describing and self- documenting. If I write a simple XML file like this, you can probably understand the gist of it without needing a separate manual page: agatha ... We’ll use a config file like this one, but with a slightly more complicated format, in the examples to come. Writing XML from Perl XML is a textual format, so there are any number of ways to write XML files from Perl. Using ordinary print statements to write XML-compliant text would be the simplest method, but we can do better. Perl modules like XML::Generator by Benjamin Holzman and XML::Writer by David Megginson can make the process easier and less error-prone by handling details like start/end tag matching and escaping special characters (<, >, &, etc.) for us. To keep the example simple, let’s say we wanted to write the teeny XML config file that we just saw.‡ The XML::Writer way to do this would be: use XML::Writer; use IO::File; † One error that is easily caught by the well-formedness check is truncated files. If the file is missing its last closing tag because some data has been lost from the end, the document will not parse correctly. This is one property YAML, which we’ll look at later in this chapter, does not have. ‡ As a quick aside, the XML specification recommends that every XML file begin with a version declaration (e.g., ). It is not mandatory, but if we want to comply, XML::Writer offers the xmlDecl() method to create one for us. Configuration File Formats | 193 my %hosts = ( 'name' => 'agatha', 'addr' => '', ); my $FH = new IO::File('>netconfig.xml') or die "Unable to write to file netconfig.xml: $!\n"; my $xmlw = new XML::Writer( OUTPUT => $FH ); $xmlw->startTag('network'); print $FH "\n "; $xmlw->startTag('host'); # note that we're not guaranteed any specific ordering of the # subelements using this code foreach my $field ( keys %hosts ) { print $FH "\n "; $xmlw->startTag($field); $xmlw->characters( $hosts{$field} ); $xmlw->endTag; } print $FH "\n "; $xmlw->endTag; print $FH "\n"; $xmlw->endTag; $xmlw->end; $FH->close(); Using XML::Writer gives us a few perks: • The code is quite legible; anyone with a little bit of markup language experience will instantly understand the names startTag(), characters(), and endTag(). • Though our data didn’t need this, characters() is silently performing a bit of pro- tective magic for us by properly escaping reserved entities like the greater-than symbol (>). • Our code doesn’t have to remember the last start tag we opened for later closing. XML::Writer handles this matching for us, allowing us to call endTag() without specifying which end tag we need. Two drawbacks of using XML::Writer in this case are: • We certainly typed more than we would have if we had just used print statements. • XML::Writer is oriented more toward generating XML that will be parsed by a machine rather than something pretty that a human might like to read. It does have a couple of initialization-time options that allow it to provide slightly prettier output, but they still don’t produce output that looks like our original example 194 | Chapter 6: Working with Configuration Files file—hence all of the icky print $FH statements scattered throughout the code to add whitespace. Survey of best-practice tools to parse and manipulate XML from Perl We’re going to look at several ways to parse XML from Perl, because each way has its strengths that make it well suited for particular situations or programming styles. Knowing about all of them will allow you to pick the right tool for the job. To make the parsing tools easier to compare and understand, we’re going to use a common example XML file as input. Let’s look at that now so we understand just what the data we’re going to chew on looks like. Here’s the full file. We’ll take it apart in a second: This is the configuration of our network in the Boston office. SMTP POP3 IMAP4 HTTP HTTPS DNS NTP LDAP LDAPS Configuration File Formats | 195 firewall This file represents a very small network consisting of three servers and two clients. Each element represents a machine. The first is a server that provides mail services: SMTP POP3 IMAP4 Each interface has an associated with it to provide its DNS name (in DNS parlance, it has an A resource record). The server in the example excerpt provides three mail-related services and has a DNS CNAME element reflecting that. Other serv- ers provide other services, and their CNAME or CNAMEs are listed accordingly. Here’s a sample client: 196 | Chapter 6: Working with Configuration Files It is different from our example server not only because it has a different type attribute and has no services listed, but also because it has multiple interfaces (it is probably a laptop, since it has both wireless and Ethernet interfaces), though that could be changed if we decided to multihome any of the servers. Each interface has an address and a DNS hostname (A resource record) listed for it. There’s an interesting decision ingrained in this file. Unlike our first XML example, this file shows the use of both attributes (name, type, os) and subelements (interface, service). Choosing between the two is always a fun intellectual exercise and a great way to start a debate. In this case I was guided by one of the canonical discussions on this topic ( and chose to make the pieces of information that describe a machine be attributes and the things we add to or remove from a machine (interfaces, services) be subelements. This was strictly my choice; you should do what makes sense for you. So, let’s get into the game of parsing XML files. We’re going to look at the process using three different modules/approaches with roughly increasing complexity. Each module has pluses and minuses that make it better than the others for certain situations. I’ll list them right at the beginning of each section so you go into each passage with the right expectations. Having at least a cursory understanding of all three will give you a com- plete toolkit that can tackle virtually all of your XML-related needs.* Working with XML using XML::Simple XML::Simple Pros and Cons Benefits: • It’s the simplest module (at least on the surface) to use. • It maps XML into a standard Perl data structure. • It will actually use one of the other modules that we’ll see in a moment (XML::LibXML) for pretty fast parsing. • It’s well documented. • The module’s maintainer is very responsive to questions, issues, etc. * As an aside, you could also write your own code to parse XML (perhaps using some fancy regexp footwork). If you attempt this, however, you’ll spend more time on the parser and less on your actual goal, with little return. If you do need a super-simple XML parser made out of regexps, modules like that also exist, though we won’t be looking at them here. Configuration File Formats | 197 Drawbacks: • It doesn’t preserve element order or format (i.e., if you read in an XML file and spit it out again, the elements won’t necessarily be in the same order or have the same format). • It can’t handle “mixed content” situations (where both text and elements are embedded in another element), as in: This is our devel network • All of the data gets slurped into memory by default. • Some people are philosophically opposed to a direct transformation of XML to Perl data structures. So when should you use this module? XML::Simple is perfect for small XML-related jobs like configuration files. It’s great when your main task includes reading in an XML file or writing an XML file (although it may get a little trickier if you have to read and then write, depending on the situation). I use it for the majority of situations where XML is just a small part of the actual task at hand and not the main point. The easiest way to read an XML config file from Perl is to use the XML::Simple module, by Grant McLean. It allows you to write simple code like this to slurp an XML file into a Perl data structure: use XML::Simple; my $config = XMLin('config.xml'); # work with $config->{stuff} Turning that data structure back into XML for writing after you’ve made a change to it is equally as easy: ... (data structure already in place) XMLout($config, OutputFile => $configfile ); I know what you’re thinking: it can’t be that easy, right? Well, let’s see what happens if we feed our sample XML file to the module using its defaults. If we ran the first of our XML::Simple code samples and then dumped the resulting data structure using the debugger’s x command, we’d get this output:† 0 HASH(0xa36fc8) 'description' => HASH(0x97f5e8) 'content' => ' This is the configuration of our network in the Boston office. ' 'name' => 'Boston' 'host' => HASH(0xa5d6ac) † To debug XML::Simple code, it is best to use a good data-structure-dumping module like Data::Dumper, Data::Dump::Streamer, YAML, or the Perl debugger, as demonstrated here. 198 | Chapter 6: Working with Configuration Files 'agatha' => HASH(0xa5d424) 'interface' => HASH(0xa38f98) 'addr' => '' 'arec' => '' 'cname' => '' 'name' => 'eth0' 'type' => 'Ethernet' 'os' => 'linux' 'service' => ARRAY(0xa5da6c) 0 'SMTP' 1 'POP3' 2 'IMAP4' 'type' => 'server' 'baron' => HASH(0xa51390) 'interface' => HASH(0xa3e228) 'addr' => '' 'arec' => '' 'cname' => ARRAY(0xa5d874) 0 '' 1 '' 2 '' 'name' => 'eth0' 'type' => 'Ethernet' 'os' => 'linux' 'service' => ARRAY(0xa5d994) 0 'DNS' 1 'NTP' 2 'LDAP' 3 'LDAPS' 'type' => 'server' 'gil' => HASH(0xa5d61c) 'interface' => HASH(0xa3de44) 'addr' => '' 'arec' => '' 'cname' => '' 'name' => 'eth0' 'type' => 'Ethernet' 'os' => 'linux' 'service' => ARRAY(0xa5d964) 0 'HTTP' 1 'HTTPS' 'type' => 'server' 'krosp' => HASH(0xa5d754) 'interface' => HASH(0xa5d664) 'en0' => HASH(0xa5d5ec) 'addr' => '' 'arec' => '' 'type' => 'Ethernet' 'en1' => HASH(0xa5d604) 'addr' => '' 'arec' => '' 'type' => 'AirPort' 'os' => 'osx' 'type' => 'client' 'mr-tock' => HASH(0xa4ee28) Configuration File Formats | 199 'interface' => HASH(0xa2fc50) 'addr' => '' 'arec' => '' 'cname' => '' 'name' => 'fxp0' 'type' => 'Ethernet' 'os' => 'openbsd' 'service' => 'firewall' 'type' => 'server' 'zeetha' => HASH(0xa5d4b4) 'interface' => HASH(0xa5d4cc) 'en0' => HASH(0xa5d454) 'addr' => '' 'arec' => '' 'type' => 'Ethernet' 'en1' => HASH(0xa5d46c) 'addr' => '' 'arec' => '' 'type' => 'AirPort' 'os' => 'osx' 'type' => 'client' Using just the defaults, we get a pretty workable data structure. There’s a hash with a key called host. This points to another hash that contains the hosts, each keyed by its name. This means we could get each host’s information with a simple $config->{host}->{hostname}. That’s all well and good, but if you peer a little harder at the data structure, a few interesting things start to stick out (in order of least to most important): 1. XML::Simple has preserved much of the whitespace that was included in the file just to make things prettier. When we actually want to operate on the data, we could strip the leading and trailing whitespace, but it would be much more convenient to have the parser do it for us. If we change: my $config = XMLin('config.xml'); to: my $config = XMLin('config.xml', NormalizeSpace => 2); we get our wish. 2. If you look at how the elements from the file have been trans- lated into data structures for all of the server hosts, you’ll notice something inter- esting. Take a peek at the results for, say, agatha and mr-tock. The service section for agatha looks like this: 'service' => ARRAY(0xa4f2d8) 0 'SMTP' 1 'POP3' 2 'IMAP4' while the same section for mr-tock looks like this: 'service' => 'firewall' 200 | Chapter 6: Working with Configuration Files In one case it’s an array, and in the other it’s a simple scalar. This difference will make coding harder, because it means we have to have two conventions for data retrieval. Why did XML::Simple do two different things in this case? By default XML::Simple converts nested elements (like those in the ele- ments) differently based on whether there is one or more than one subelement present. We can actually tune that behavior by using the ForceArray argument. It can take either a simple 1 to force all nested elements into arrays or, better yet, a list of element names to force only the listed elements into array form. If we instead write: my $config = XMLin('config.xml', NormalizeSpace => 2, ForceArray => ['service']); those two sections will look much more uniform: 'service' => ARRAY(0xa4f2d8) 0 'SMTP' 1 'POP3' 2 'IMAP4' ... 'service' => ARRAY(0xa31fac) 0 'firewall' When to Use XML::Simple’s ForceArray You could make a good argument that one should always use ForceArray => 1 because it provides the maximum amount of consistency. That’s true, but using that setting also ensures the maximum amount of syntactic hassle. You’ll quickly become annoyed at having to use an array index every time you want to get to the contents of even single subelements in the original XML file. I’d like to suggest that you use ForceArray with a list of element names in the following judicious manner: if you have an element that could even conceivably contain more than one instance of a subelement (e.g., multiple subelements), include it in the ForceArray list. If an element definitely‡ will only have one subelement instance, you can leave it out. Also, if you plan to use the KeyAttr option we’ll discuss shortly, any elements listed for that option need to be listed in ForceArray as well. 3. There’s another section of the XMLin() data structure that looks a little awry. If you look at how the elements from the file have been trans- lated into data structures for all of the hosts, you’ll notice something interesting if you compare the results for agatha and zeetha. The interface section for agatha looks like this: ‡ Experienced old-timers may snicker at the notion that you could say anything “definite” about user- supplied data (“oh, that will never happen...”), and they’d be right. To get the best assurance possible that your expectations won’t be violated, you should provide a way to validate the XML file using a DTD or XML schema. Configuration File Formats | 201 'interface' => HASH(0xa38f98) 'addr' => '' 'arec' => '' 'cname' => '' 'name' => 'eth0' 'type' => 'Ethernet' while the same section for zeetha looks like this: 'interface' => HASH(0xa5d4cc) 'en0' => HASH(0xa5d454) 'addr' => '' 'arec' => '' 'type' => 'Ethernet' 'en1' => HASH(0xa5d46c) 'addr' => '' 'arec' => '' 'type' => 'AirPort' In one case it is a hash whose keys are the various components that make up the interface (address, type, etc.), while in the other it is a hash of a hash whose keys are the interface names and whose components are the sub-hash’s keys. When it comes time to work with the imported data structure, we’ll be forced into coding two different ways to get at basically the same kind of data. Here, we’re running into another example of single and multiple instances with the same subelement name () being treated differently. Let’s try to apply the ForceArray option to this case as well: my $config = XMLin('config.xml', NormalizeSpace => 2, ForceArray => ['interface']); Great, now those two sections also have become much more uniform: 'interface' => HASH(0xa32f28) 'eth0' => HASH(0xa32e80) 'addr' => '' 'arec' => '' 'cname' => '' 'type' => 'Ethernet' ... 'interface' => HASH(0xa2593c) 'en0' => HASH(0xa257bc) 'addr' => '' 'arec' => '' 'type' => 'Ethernet' 'en1' => HASH(0xa257d4) 'addr' => '' 'arec' => '' 'type' => 'AirPort' But wait a second; didn’t we just set something called ForceArray? The elements have been converted into hash of hashes with no explicit arrays in sight. There’s some magic afoot here that we really should 202 | Chapter 6: Working with Configuration Files discuss, and that leads us to our fourth comment on the default Perl data structure XMLin() created for us. 4. XML::Simple notices when nested subelements (like those we’ve been dealing with) have certain attributes and does something special with them. By default, if they have the attribute name, key, or id, XML::Simple will turn the usual array created by nested subelements into a hash keyed by that attribute. In the case we just saw, once the elements were converted to an array, further logic kicked in because each element has a name attribute. On the surface this may appear like a little too much “Do What I Mean” meddling, but it turns out that it yields some very usable data structures, especially if you understand how/when it works. In our example, we can now reference: # can remove last two arrows $config->{hostname}->{interface}->{interface_name} The code reads well as a result. As with the ForceArray option, we can tweak this behavior easily. To turn it off entirely, we can add KeyAttr => {}. However, the resulting data structure will have to be accessed by looping through the array of elements to find the one we need. That’s a little cumbersome, so most of the time we’ll instead want to write code like this: # can remove last two arrows $config->{hostname}->{interface}->{ip_addr} to access a data structure with an inner part like this: 'interface' => HASH(0xa31c50) '' => HASH(0xa31b18) 'arec' => '' 'name' => 'en0' 'type' => 'Ethernet' '' => HASH(0xa31b30) 'arec' => '' 'name' => 'en1' 'type' => 'AirPort' We’d then use a parse line like this: my $config = XMLin( 'config.xml', NormaliseSpace => 2, ForceArray => ['interface'], # uses square brackets KeyAttr => { 'interface' => 'addr' }, # uses curly braces ); The KeyAttr option highlighted earlier says to turn the subelements of interface elements (only) into a hash keyed on the interface’s IP address. If for some reason you wanted to have the addr field also appear in the contents of that hash (instead of appearing only in the key name), you could add a + at the front of that attribute name. One last thing to point out here: we left the ForceArray argument in place Configuration File Formats | 203 to make sure even single elements get turned into arrays for KeyAttr to transform. The lesson to learn from that little exploration of XML::Simple’s default parsing behavior is that although the module is simple on the surface, you may have to do a little unex- pected argument tweaking to get the results you want. XML::Simple has a “strict” mode you can turn on (like use strict;) to help guide you in the right direction, but it still takes a little work get things right sometimes. This issue becomes painfully clear when we try to round-trip the XML (i.e., read it in, modify the data, and then write it back out again). Modifying the data we’re working with is easy—we use the standard Perl data structure semantics to add/delete/modify elements of the data structure in memory. But how about writing the data? If we use the previous parsing code example and then add this to the bottom: print XMLout($config); we get XML output that looks like this in part: ... SMTP POP3 IMAP4 ... This is hardly what we started with, and we’ve lost data (the interface address)! If we add a KeyAttr option (matching the one for XMLin()), as recommended by the module’s strict mode, we get back the data, but not the subelement/attribute changes: ... SMTP POP3 IMAP4 ... Unpleasant situation, no? We have a few choices at this point if we want to stick with XML::Simple, including: • Changing the format of our data file. This seems a bit extreme. • Changing the way we ask XML::Simple to parse our file by working hard to fine- tune our options. For example, the XML::Simple documentation recommends using KeyAttr => {} for both reading and writing in this situation. But when we tailor 204 | Chapter 6: Working with Configuration Files the way we read in the data to make for easy writing, we lose our easy hash se- mantics for data lookup and manipulation. • Performing some data manipulation after reading but before writing. We could read the data into a structure we like (just as we did before), manipulate the data to our heart’s content, and then transform that data structure into one XML::Simple “likes” before writing it out. This isn’t terribly hard, but you do need a good grasp of how to manipulate moderately complex data structures. It usually involves a map() call or two and a pile of punctuation. Your situation will dictate which (if any) of these options is best. I’ve picked from all three choices in the past, depending what seemed to make the most sense. And some- times, to be quite frank about it, the best way to win is not to play at all. Sometimes you need to leave the comfy harbors of XML::Simple and use another module entirely. That’s just what we’ll do in the next section. Grant McLean, XML::Simple’s author, says that he recommends people: • Use XML::Simple’s strict mode to avoid common mistakes. • Switch to using XML::LibXML if they haven’t gotten the results they want from XML::Simple within 5–10 minutes. He’s written an interesting article on the subject, available at http://www Working with XML using XML::LibXML XML::LibXML Pros and Cons Benefits: • It provides a very fast parser. • It has an emphasis on standards compliance (although it currently supports only XPath 1.0; XPath 2.0 support is not planned as of this writing). • It supports both XPath and DOM, making navigating through and operating on a document pretty easy. • It’s the current default recommendation for XML parsing from Perl (as per http:// • The module maintainer is very responsive to questions, issues, etc. • As a bonus, it can also parse HTML and (largely) operate on it as if it were XML. Drawbacks: • A working and compatible version of the libxml2 library must be present on the system to be called by the module. • The documentation is centered on describing what is supported in the module but is very light on how to go about using it. If you already know what you want to do Configuration File Formats | 205 (perhaps because you are familiar with XML, DOM, XPath, and other related standards), this is fine, but if you’re just starting out and want to understand the basics, you’ll have a very hard time getting what you need from the documentation. This is by far the module’s biggest drawback. • All of the data gets slurped into memory by default. When should you use this module? XML::LibXML is good for the vast majority of cases where you need to process reasonably sized XML documents as the main thrust of what you are trying to get done. It’s fast, it behaves the way the standards suggest it should, and it’s pretty easy to use, provided that you understand the basics of XPath or DOM and can figure how to use it. XML::LibXML, maintained by Petr Pajas, has much to recommend it, including many powerful features and options. I’ll just skim the surface here, to show you the most useful stuff. You’ll want to consult the resources listed at the end of the chapter for the rest of the story. Like XML::Simple, XML::LibXML slurps your entire XML document into memory by default and gives you the tools to work with that in-memory representation. Unlike XML::Simple, the interface for doing so in XML::LibXML is not your native Perl data struc- ture semantics. Instead, the data is represented as a tree with several methods for ma- nipulating it. If the idea of representing XML data in a tree structure doesn’t make immediate sense to you, you should reach for a bookmark and insert it here. Pause reading this chapter, go read Appendix B, and then come right back. This is important because the very next paragraph is going to assume that you have at least the back- ground provided in that appendix at your disposal. XML::LibXML supports the two most common ways to navigate XML data represented in tree form: the W3C Document Object Model (DOM) and XPath.* Though I tend to favor XPath over DOM in my programming because I like XPath’s concision and ele- gance, we’ll look at examples using both approaches. It’s good to know both methods because XML::LibXML allows you to use virtually any combination of the two that makes sense to you. Let’s start with the DOM method of getting around a tree of XML data. To make a comparison between DOM and XPath easier, we’ll stick to the same sample XML document introduced in the XML::Simple section. To keep the tree mortality rate down, I won’t reprint that document here. XML::LibXML programs that use the DOM method and those that use XPath begin the same way (load the module, create a parser instance, parse the XML data): use XML::LibXML; my $prsr = XML::LibXML->new(); * DOM Level 2 Core and XPath 1.0, to be precise (as of this writing). 206 | Chapter 6: Working with Configuration Files my $doc = $prsr->parse_file('config.xml'); To use the DOM method of walking our data, we start by retrieving the root element of the XML document: my $root = $doc->documentElement(); From this element we can either start to explicitly walk the tree by hand or ask the module to search down the tree on our behalf. To walk the tree, we request the child nodes and iterate over them explicitly: my @children = $root->childNodes; foreach my $node (@children){ print $node->nodeName(). "\n"; } If you run the XML::LibXML code we’ve written so far, you’ll get some very peculiar output: #text description #text host #text host #text host #text host #text host #text host #text The description and host lines make sense (those are the and elements from our document), but what’s with all of the #text nodes (nodes with a default name of #text)? If you were to look carefully at the contents of one of the #text nodes when the program was running, you would see: x $node->data 0 ' ' or, to make this even clearer: x split(//,$node->data); 0 ' ' 1 ' ' 2 ' ' 3 ' ' 4 ' ' Configuration File Formats | 207 The node is holding one carriage return character and four space characters. Unlike XML::Simple, which strips whitespace by default, XML::LibXML tries to preserve any whitespace it encounters when parsing a document (because it isn’t always clear when the whitespace itself could be significant). It preserves the whitespace by storing it in generic text nodes in the tree. If you find the “empty” text nodes distracting and you don’t need the whitespace kept around, you can ask the parser to drop all of the nodes that hold only whitespace. To do this, you set a parser option before parsing the file: $prsr->keep_blanks(0); my $doc = $prsr->parse_file('config.xml'); Adding this option gives us the output we’d expect (the names of all of the child nodes of , the root element of the document): description host host host host host host Just to keep things simpler for the rest of this section, you can assume that keep_blanks(0) has been set. The actual mechanics of walking a tree are pretty basic: you iterate over the child nodes of the node you are at, and if any of those nodes have children (you can check with hasChildNodes()), walk those as well. Most often people write recursive programs for this sort of thing; see the tree-walking code in Chapter 2 for the basic idea. To navigate by hand to a specific node someplace deeper in the tree, access the node you want at each level in the tree and descend from there: my $root = $doc->documentElement(); my @children = $root->childNodes; my $current = $children[2]; # second element @children = $current->childNodes(); $current = $children[1]; # first element print $current->textContent(); # 'HTTP' # or, chain the steps together in a punctuation-heavy fashion (yuck): print STDOUT (($root->childNodes())[2]->childNodes())[1]->textContent(); In addition to walking down the tree, we can also use nextSibling() to go sideways: my $root = $doc->documentElement(); my @children = $root->childNodes; my $current = $children[2]; # second element $current = $current->nextSibling; # move to third element If all this manual tree-walking code looks like a pain, there is another DOM-flavored alternative: XML::LibXML can do some of the work for us. If we know we only care about 208 | Chapter 6: Working with Configuration Files certain subelements of the element we’ve focused on, we can ask for just those elements using getChildrenByTagName(). This function takes the name of an element and returns only the nodes containing that element. For example, in our document we might want to only retrieve the interface definition(s) of a host: my $root = $doc->documentElement(); my @children = $root->childNodes; my $current = $children[5]; # element for krosp my @interface_nodes = $current->getChildrenByTagName('interface'); This grep()-like function saves us the effort of iterating over all of the children of a node looking for the elements of interest. If you have a sufficiently large tree, the reduction in effort can make a real difference. A more exciting method related to getChildrenByTagName() is getElementsByTagName(). getElementsByTagName() will search not only the children of the node it is called from, but everything below that node in the tree. If we wanted to retrieve all of the interface definitions for all of the hosts, we could write something like this: my $root = $doc->documentElement(); my @interface_nodes = $root->getElementsByTagName('interface'); Once we’ve found the node or nodes we want in the tree, we can retrieve the child text nodes that store what you would probably consider the “contents” of an element’s node (if you didn’t know that that info is actually kept in a separate text sub-node): foreach my $node ( @interface_nodes ) { $node->textContent(); # returns the contents of all child text nodes } If that node has attributes, we can list them or get their values: foreach my $attribute ($node->attributes()){ print $attribute->nodeName . ":" . $attribute->getValue() . "\n"; } # or to retrieve a specific attribute: print $node->getAttribute('name') if $node->hasAttribute('name'); Attributes of elements are stored in an associated attribute node, which is why we can call nodeName() here to get the attribute’s name. Attribute nodes are associated with their elements but are not children of those element nodes in the same way the text nodes that hold the “con- tents”† of the elements are. For instance, if we call childNodes(), they are not listed as children. This is true in both the DOM and XPath specs. † I’m using snarky quotes around the word “contents” here and elsewhere to indicate that an element node doesn’t actually have data in it. It has one or more child text nodes that hold its data. But when you see baa-la-la it is hard not to think of “baa-la-la” as the contents of the element node. Configuration File Formats | 209 To change the “contents” of an element node, we change the data in the appropriate child text node: # If we know that a node has a single child, and that child is a text # node, we can go right to it. To test whether it is a text node, we # could do something like the first line below. # # If we don't know which child (or children) of the node holds the data we # want, we can iterate over the list returned by childNodes(), testing # nodeType() and textContent() as we go along. my $textnode = $node->firstChild if ($node->firstChild->nodeType == XML_TEXT_NODE); $textnode->setData('new information'); There are various methods, such as insertData(), appendData(), and replaceData(), that let us operate on the text node as we would expect. Attribute modification takes place on the element node directly using the analogous call to getAttribute(): setAttribute(). Operating on the data associated with nodes is the most common task, but sometimes we need to manipulate the node tree itself. If we want to add or delete elements (and/or their subelements) to or from an XML document, we’ll need to mess with its nodes. Let’s start with the second operation, deletion, because it is the easier of the two. To delete a node (perhaps deleting a whole branch of the tree at the same time), we locate that node’s parent and tell it to remove the node in question: my $parent = $node->parentNode; $parent->removeChild($node); Alternatively, we can chain these two steps. In case you are curious, $node gets a new parent (a XML::LibXML::DocumentFragment node) after the following is executed: $node->parentNode->removeChild($node) Adding an element node to a tree is a little trickier because we have to construct everything about that node before adding it to the tree. That is, we have to make the node itself, set any attributes, create text nodes and any other sub-nodes, and give those nodes values; only then can we finally connect it to the tree. For example, let’s say we wanted to add a new element. In this element, we’ll place other elements that describe the network. One such element could be to help us distinguish development (or staging) networks from produc- tion networks. The XML in question would look like this (whitespace added for readability): This is a production network To create just that much, we’d write code like this: 210 | Chapter 6: Working with Configuration Files # let's build the XML elements from the inside out my $type = $doc->createElement('type'); $type->setAttribute( 'name', 'production' ); $type->appendTextNode('This is a production network'); my $meta = $doc->createElement('meta'); # make a subelement, or child, of $meta->appendChild($type); If you are suitably anal retentive (a good quality in a network administrator), you are probably bothered by the element being a separate ele- ment in the document (rather than a subelement of , where it rightfully belongs). To fix this, let’s move it into the meta tag: my $root = $doc->documentElement(); # find the element that is a child of the # root element and make it the last child of the # element instead $meta->appendChild($root->getChildrenByTagName('description')); Now that we have the XML fragment, let’s place it into our document’s node tree. If we wanted to be lazy, we could simply add it to the end of the document by making it a child of the root element ($root->appendChild($meta)), but what we really want is to have it come first in the document, since it describes the data that will follow: my $root = $doc->documentElement(); # place it before the root element's current first child $root->insertBefore($meta,$root->firstChild); If you plan to insert lots of nodes, crafting them by hand can be a bit tedious. Fortu- nately, XML::LibXML provides a very nice shortcut called parse_balanced_chunk() that takes in XML data and returns a document fragment that can be linked into your node tree. Let’s use our first example to demonstrate this technique: my $root = $doc->documentElement(); my $xmltoinsert = <<'EOXML'; This is a production network EOXML my $meta = $prsr->parse_balanced_chunk($xmltoinsert); $root->insertBefore($meta,$root->firstChild); Once you have the tree you want in memory, with whatever changes you made to the nodes or the tree itself, writing it out is a snap: open my $OUTPUT_FILE, '>', $filename or die "Can't open $filename for writing: $!\n"; Configuration File Formats | 211 print $OUTPUT_FILE $doc->toString; close $OUTPUT_FILE; That’s basically how to work with a document via the DOM model. Once you get this far into understanding XML::LibXML, the documentation may start to make more sense to you, so be sure to give the XML::LibXML::{Element, Node, Attr and Text} docu- mentation another read for the countless methods left out of this brief introduction. If you are one of the people who took my earlier suggestion and read the XPath ap- pendix, you may be curious about how you can put your newfound XPath knowledge into practice. Let’s look at that now. As I mentioned before, XML::LibXML programs that use XPath start the same way as those that are DOM-based: use XML::LibXML; my $prsr = XML::LibXML->new(); $prsr->keep_blanks(0); my $doc = $prsr->parse_file('config.xml'); The difference begins at the point where you want to start navigating the node tree or querying nodes from it. Instead of manually walking nodes, you can instead bring location paths into the picture using findnodes(). For comparison’s sake, let’s go back and redo some of the DOM examples using XPath. The first example we saw with DOM was a list of the root’s child nodes. Here’s the XPath equivalent: my @children = $doc->findnodes('/network/*'); foreach my $node (@children){ print "$node->nodeName()\n"; } Not so exciting, I know. But now let’s look at the very next example, where we had to do a lot of work to get the data associated with the first service provided by the second host in our XML config file. With XPath, all of that code gets whittled down to a line or two: # we ask for the single node we're going to get back using a # list context (the parens around $node) because findnodes() # returns a NodeList object in a scalar context my ($tnode) = $doc->findnodes('/network/host[2]/service[1]/text()'); print $tnode->data . "\n"; # or, if you'd like to do this in a way that allows for # a query that could return multiple text nodes: foreach my $tnode ($doc->findnodes('/network/host[2]/service[1]/text()')){ print $tnode->data . "\n"; } With one findnodes() call, we can locate the correct nodes and return their associated text nodes. findnodes() gives us all of the power of XPath 1.0’s location paths. This power includes things like the descendant operator (//), which can easily replicate the 212 | Chapter 6: Working with Configuration Files functionality of the DOM getChildrenByTagName() and getElementByTagName() calls and add a whole new level of sophistication at the same time (thanks to XPath predicates): # find all of the hosts that currently provide more than one service my @multiservers = $doc->findnodes('//host[count(service) > 1]'); # find their names (name attribute values) instead and print them foreach my $anode ($doc->findnodes('//host[count(service) > 1]/@name')){ print $anode->value . "\n"; } Here we’ve used the XPath descendant operator to find all the host element nodes in the document and then filtered that set using the XPath function count() in a predicate. Now let’s look at a simple XPath example that demonstrates the programmatic flexi- bility XML::LibXML offers: @nodes = $doc->findnodes('/network/host[@type = "server"]//addr'); This XPath expression will find all servers and return their element nodes. We probably don’t want the nodes themselves, though; we more likely want the actual information stored in their text node children (i.e., the addresses they “hold”). There are three things you could do at this point, depending on your pro- gramming style and perhaps the larger context of the program: 1. Change the XPath expression to look more like the previous example (i.e., add a text() step): @nodes = $doc->findnodes('/network/host[@type = "server"]//addr/text()'); This gets you all of the text nodes that hold the address values. We’ve already seen how to iterate over a list of text nodes, extracting their contents with a data() method call as we go, so I won’t repeat that foreach() loop here. 2. Make additional XPath evaluation calls from each of the nodes found: foreach my $node (@nodes){ print $node->find('normalize-space(./text())') . "\n"; } Here we’ve called find() and not findnodes() because we’re going to evaluate an XPath expression that will yield a string (not a node or node set). The expression we’re evaluating says, “Start at the current node, find its associated text node, and normalize its value (i.e., strip the leading/trailing whitespace).” We could have left out the normalize-space() XPath function call and kept it like the other examples in this list, but this way helps show how breaking the task into two XPath calls can lead to more legible location paths in your code. 3. Switch to using DOM methods at this point: foreach my $node (@nodes) { print $node->textContent() . "\n"; } Configuration File Formats | 213 The last choice may seem the least sexy of the three, but it is actually one of the more important options at this point in our XPath-related discussion. XPath is superb for navigating a document or querying certain information from it, but it doesn’t address how to modify that document at all. Once we’ve found the information we want to modify, or if we want to make some change to the tree starting at a node we’ve located, XPath steps out of the picture and we’re back in DOM-land again. Everything we saw a little earlier in this section about how to modify the information stored in the node tree or how to mess with the tree itself now comes into play. If we want to change the data stored in a text node, we call setData(). If we want to remove a node, we call removeChild() from its parent, and so on. Even the use of to_string() to write out the tree is the same. XPath and XHTML Here’s a tip that Petra Pajas, the current maintainer of XML::LibXML, recommended I share with you: Beginners using XPath to parse an XHTML document (e.g., via XML::LibXML) often get stymied because simple XPath location paths like /html/body don’t appear to match anything. Questions about this come up time and time again on the perl-XML mailing list because it certainly looks like it should work. Here’s the trick: XHTML has a default namespace of its own predefined (). See the sidebar “XML Namespa- ces” on page 224 for a more complete explanation, but if we were to use Perl terms, you could think of the and elements as living in a separate package from the default one the XPath parser would normally search. To get around this, we have to give the XPath implementation a mapping that assigns a prefix for that namespace. Once we’ve done this, we can successfully use location paths that include the prefix we defined. For example, /x:html/x:body will now do the right thing. To create this mapping in XML::LibXML, we create a new XPathContext (a context in which we’re going to do XPath work) and then register a prefix for the XHTML name- space in it. Here’s a code snippet that demonstrates how this is done. The code extracts the textual contents of all paragraph nodes in a document: use XML::LibXML; use XML::LibXML::XPathContext; my $doc = XML::LibXML->new->parse_file('index.xhtml'); my $xpath = XML::LibXML::XPathContext->new($doc); $xpath->registerNs( x => '' ); for my $paragraph ($xpath->findnodes('//x:p')) { print $paragraph->textContent,"\n"; } Hope this tip saves you a bit of frustration. 214 | Chapter 6: Working with Configuration Files To drive this point home and reinforce what you learned earlier, let’s look at a more extended example of some XPath/DOM interactions used to do real work. For this example, we’ll generate a DNS zone file for the wired network portion of the XML config file we’ve been using. To keep the focus on XML, we’ll use the GenerateHeader code from Chapter 5 to generate a correct and current zone file header: use XML::LibXML; use Readonly; Readonly my $domain => ''; # from the programs we wrote in Chapter 5 print GenerateHeader(); my $prsr = XML::LibXML->new(); $prsr->keep_blanks(0); my $doc = $prsr->parse_file('config.xml'); # find all of the interface nodes of machines connected over Ethernet foreach my $interface ( $doc->findnodes('//host/interface[@type ="Ethernet"]') ) { # print a pretty comment for each machine with info retrieved via # DOM methods my $p = $interface->parentNode; print "\n; " . $p->getAttribute('name') . ' is a ' . $p->getAttribute('type') . ' running ' . $p->getAttribute('os') . "\n"; # print the A record for the host # # yes, we could strip off the domain and whitespace using # a Perl regexp (and that might make more sense), but this is just # an example so you can see how XPath functions can be used my $arrname = $interface->find( " substring-before( normalize-space( arec / text() ), '$domain' ) "); print "$arrname \tIN A \t \t " . $interface->find('normalize-space(addr/text())') . " \n "; # find all of the CNAME RR and print them as well # # an example of using DOM and XPath methods in the same for loop # note: XPath calls can be computationally expensive, so you would # (in production) not want to place them in a loop in a loop foreach my $cnamenode ( $interface->getChildrenByTagName('cname') ) { print $cnamenode->find( " substring-before(normalize-space(./text()),'$domain')") . "\tIN CNAME\t$arrname\n"; } Configuration File Formats | 215 # we could do more here, e.g., output SRV records ... } Now let’s shift gears entirely and leave tree-based XML parsing behind for a bit. Working with XML using SAX2 via XML::SAX SAX2 via XML::SAX Pros and Cons Benefits: • Data can be parsed as it is received (you don’t have to wait for the entire document to begin processing). • SAX2 has become a multilingual standard. (SAX started out in the Java world but was quickly adopted by all of the major scripting languages as well. This means your Perl SAX2 code, at least at a conceptual level, will be easy for your Java, Python, Ruby, and other colleagues to understand.) • XML::SAX makes it easy to use different parser backends with the same basic code. • XML::SAX is object-oriented through and through. • SAX2 has some very cool advanced features, like pipelining (multiple XML filter routines connected to each other) and easy ways to consume data from non-XML sources or export data from XML. Drawbacks: • You snooze, you lose. The parser will send you information a single time. If you don’t save that information or you realize you should have kept the data in a dif- ferent data structure for later retrieval, you’re out of luck! • XML::SAX is object-oriented through and through. If your programming experience isn’t particularly oriented toward this approach, the learning curve can be steep. • Sometimes you have to do more coding because certain operations require manual labor. Examples include collecting textual data and finding specific elements. If you want to store anything from the document being parsed (e.g., if you need a tree), you have to do that by hand. So when is it appropriate to use XML::SAX? This module is good for large XML data sets or conditions where collecting all of the data first into an in-memory tree isn’t practical. XML::SAX works well if the idea of an event-based parsing model fits the way you think about your task at hand. If you are already using XML::Parser, this would be a good next step. So far everything we’ve seen for handling XML requires us to slurp all of the data into some in-memory representation before we can begin to operate on it. Even if memory prices drop, at a certain point this doesn’t scale. If you have a really huge XML data set, trying to keep it all in memory probably won’t work. There are also issues of timing and efficiency. If you have to bring all the data into memory before you can proceed, 216 | Chapter 6: Working with Configuration Files the actual work can’t take place until the parsing is totally complete. You can’t start processing if your data hasn’t entirely arrived yet (e.g., if it’s coming over a network pipe). Finally, this model can yield a lot of unnecessary work, especially in those cases where your program is acting as a filter to modify data (e.g., renaming all elements to elements or some such transformation). With a tree-based model, the parser treats every element it reads the same, even though most (in this case, everything that isn’t a element) aren’t relevant to the task at hand. We’re going to look at a standard model for XML processing that uses an approach without these disadvantages: SAX. SAX stands for Simple API for XML and is currently in its second major revision (SAX2). It provides a processing model that treats the data in an XML document as a stream of events to be handled. To understand what this means, let’s take a small digression into some Perl/XML history that is still relevant to this day. Once upon a time, James Clark, the technical lead for the XML Working Group, created a really spiffy XML parser library in C called expat. expat was a well-respected piece of code, and as the popularity of XML increased, various developers started calling it from within their code to handle the work of parsing XML documents (as of this writing, important software projects such as Apache HTTP Server and Mozilla’s Firefox browser still do). Larry Wall himself actually wrote the first module for calling expat from Perl. This module, XML::Parser, was subsequently maintained by Clark Cooper, who sub- stantially revamped it and shepherded it for quite a number of years. It is now in the capable hands of Matt Sergeant. XML::Parser provides several interfaces for working with XML data. Let’s take a really quick look at its stream style (i.e., parsing mode), because it will allow us to slide back into talking about SAX2 with considerable ease. First, some technical background. XML::Parser is an event-based module, which can be described using a stockbroker analogy. Before trading begins, you leave a set of instructions with the broker for actions she should take should certain triggers occur (e.g., sell a thousand shares should the price drop below 3¼ dollars per share, buy this stock at the beginning of the trading day, and so on). With event-based programs, the triggers are called events and the instructions for what to do when an event happens are called event handlers. Handlers are usually just special subroutines designed to deal with particular events. Some people call them callback routines, since they are run when the main program “calls us back” after a certain condition is established. With the XML::Parser module, our events will be things like “started parsing the data stream,” “found a start tag,” and “found an XML comment,” and our handlers will do things like “print the contents of the element you just found.” Before we begin to parse our data, we need to create an XML::Parser object. When we create this object, we’ll specify which parsing mode, or style, to use. XML::Parser pro- vides several styles, each of which behaves a little differently. The style of a parse will Configuration File Formats | 217 determine which event handlers it calls by default and the way data returned by the parser (if any) is structured. Certain styles require that we specify an association between each event we wish to manually process and its handler. No special actions are taken for events we haven’t chosen to explicitly handle. This association is stored in a simple hash table with keys that are the names of the events we want to handle, and values that are references to our handler subroutines. For the styles that require this association, we pass in the hash using a named parameter called Handlers (for example, Handlers => {Start => \&start_handler}) when we create a parser object. We’ll be using the stream style, which does not require this initialization step: it simply calls a set of predefined event handlers if certain subroutines are found in the program’s namespace. The stream event handlers we’ll be using are simple: StartTag, EndTag, and Text. All but Text should be self-explanatory. Text, according to the XML::Parser doc- umentation, is “called just before start or end tags with accumulated non-markup text in the $_ variable.” We’ll use it when we need to know the contents of a particular element. Let’s take a look at the code first, and then we’ll explore a few of the interesting points it demonstrates: use strict; use XML::Parser; use YAML; # needed for display, not part of the parsing my $parser = new XML::Parser( ErrorContext => 3, Style => 'Stream', Pkg => 'Config::Parse' ); $parser->parsefile('config.xml'); print Dump( \%Config::Parse::hosts ); package Config::Parse; our %hosts; our $current_host; our $current_interface; sub StartTag { my $parser = shift; my $element = shift; my %attr = %_; # not @_, see the XML::Parser doc if ( $element eq 'host' ) { $current_host = $attr{name}; $hosts{$current_host}{type} = $attr{type}; $hosts{$current_host}{os} = $attr{os}; } 218 | Chapter 6: Working with Configuration Files if ( $element eq 'interface' ) { $current_interface = $attr{name}; $hosts{$current_host}{interfaces}{$current_interface}{type} = $attr{type}; } } sub Text { my $parser = shift; my $text = $_; my $current_element = $parser->current_element(); $text =~ s/^\s+|\s+$//g; if ( $current_element eq 'arec' or $current_element eq 'addr' ) { $hosts{$current_host}{interfaces}{$current_interface} {$current_element} = $text; } if ( $current_element eq 'cname' ) { push( @{ $hosts{$current_host}{interfaces}{$current_interface}{cnames} }, $text ); } if ( $current_element eq 'service' ) { push( @{ $hosts{$current_host}{services} }, $text ); } } sub StartDocument { } sub EndTag { } sub PI { } sub EndDocument { } The StartTag() and Text() subroutines do all the work in this code. If we see a start tag, we create a new hash key for the host (found in the tag’s attributes) and store the information found in the attributes in a sub-hash keyed by its name. We also set a global variable to keep the name of the host found in that tag in play for the subelements nested in that element. One such element is the element. If we see its starting tag, we add a nested hash for the interface to the hash being kept for the current host and similarly set a global variable so we can use the current interface name when we subsequently parse its subelements. This use of global variables to maintain state in a nested set of elements is a common idiom when working with XML::Parser, although it’s not particularly elegant from a programming or program maintenance perspective (for all the “unprotected global variables are icky” reasons). The tutorial for XML::SAX points out that it would be better Configuration File Formats | 219 to use a closure to maintain state when using XML::Parser, but that would make our code more complex than we really need given that this is just a stepping-stone example. The Text() subroutine deals with the elements we care about that have data in them. For and , which appear only once in an interface, we store the values in the appropriate interface’s sub-hash. We can tell which is the appropriate interface by consulting the global variables StartTag() sets. The code that handles and tags is a hair more complex, because there can be more than one instance of these tags in an interface or host element. To handle the possibility of multiple values, their contents get pushed onto an anonymous array that will be stored in the host record. The two other interesting parts of this code are the empty subroutines at the end and the way the data structure that gets generated by StartTag() and Text() is displayed. The empty subroutines are there because XML::Parser in stream style will print the data from any event that doesn’t have a subroutine defined to handle it. We don’t want any output from those events, so we define empty subroutines for them. The data structure we create is displayed using YAML. Here’s an excerpt of the pro- gram’s output: agatha: interfaces: eth0: addr: arec: cnames: - type: Ethernet os: linux services: - SMTP - POP3 - IMAP4 type: server ... zeetha: interfaces: en0: addr: arec: type: Ethernet en1: addr: arec: type: AirPort os: osx type: client 220 | Chapter 6: Working with Configuration Files We’ll be looking at YAML a little later in the chapter, so consider this a foreshadowing of some good stuff to come. Now let’s get to SAX2, because we’re practically there. Similar to XML::Parser’s stream style, SAX2 is an event-based API that requires us to provide the code to handle events as the XML parser generates them. One of the main differences between XML::Parser and XML::SAX is that the latter is object-oriented through and through. This can be a bit of a stumbling block for people without an OOP background, so I will try to keep the XML::SAX example as simple as possible from an OOP perspective. If you really want a good grasp of how OOP in Perl functions, Damian Conway’s Object Oriented Perl: A Comprehensive Guide to Concepts and Programming Techniques (Man- ning) is your best resource. The only other caveat is that we’ll only be skimming the surface of the subject in this fly-by. There are further SAX2 pointers in the references section at the end of the chapter that can help you go deeper into the subject. Enough preface; let’s see some code. We need to write two kinds of code to use XML::SAX: the parser initialization code and the event handlers. The parser initialization for a simple parse consists of asking XML::SAX::ParserFactory to hand us back a parser instance: use XML::SAX; use YAML; # needed for display, not part of the parsing use HostHandler; # we'll define this in a moment my $parser = XML::SAX::ParserFactory->parser( Handler => HostHandler->new ); There are two things about this code snippet that aren’t obvious at first glance. First, it includes HostHandler, which is the module we’ll construct in a moment that imple- ments the event handling class. I called it HostHandler because it provides the handler object the parser will use to handle the SAX2 events as they come in from parsing our host definition.‡ The class’s new() method returns the object used to encapsulate that code. If this seems a bit confusing, hang tight. When we return to this subject in a moment with some concrete code examples, it should all gel. Let’s get back to the parser initialization code. The second unobvious feature of this code is the module being called with the huge name of XML::SAX::ParserFactory. This module’s purpose (I’m intentionally avoiding using the OOP parlance here) is to return a parser object from an appropriate parser-providing module. Examples of parser- providing modules include XML::LibXML and XML::SAX::PurePerl, the pure-Perl parser packaged with XML::SAX. XML::SAX::ParserFactory provides a generic way to request a parser, so you can write the same code independently of which XML::SAX-friendly parser module you intend to use. In this case we’re letting XML::SAX::ParserFactory pick one for us, though there are ways of being more picky (see the documentation). ‡ The name was arbitrary. It could have been BobsYourUncle, but I’d recommend sticking to something at least vaguely understandable to someone reading your code. Configuration File Formats | 221 Once we have a parser ready to go, we aim it at our XML document just as we did with every other parser we’ve used to date: open my $XML_DOC, '<', 'config.xml' or die "Could not open config.xml:$!"; # parse_file takes a filehandle, not a filename $parser->parse_file($XML_DOC); close $XML_DOC; print Dump( \%HostHandler::hosts ); Now let’s see where the real action in SAX lives—the event handling code. We’ll take it in bite-sized pieces and use our previous XML::Parser example for comparison. As with XML::Parser, we’re going to need to write a few subroutines that will fire based on what the parser finds as it moves through the document. The names are a little different, though: StartTag() becomes start_element(), EndTag() becomes end_element(), and Text() (mostly) becomes characters(). There is one big difference between the two sets of subroutines: the XML::Parser sub- routines were unaffiliated subroutines that lived in a specific package, but the XML::SAX subroutines need to be class methods. If your lack of an OOP background makes you break out into a cold sweat when you hear terms like “class method,” don’t panic! XML::SAX makes it really easy. All you need to do is include two lines like these ahead of your subroutines, and presto, you have class methods (or more precisely, you are now overriding the default methods XML::SAX::Base provides): package HostHandler; use base 'XML::SAX::Base'; XML::SAX::Base handles all of the scut work associated with the parser object, including defining the new() method we called in our parser initialization code. If you haven’t already done so, now would be a good time to shift your mental model so you are thinking solely (even on a very basic level) in terms of objects. Nothing fancy is required. Keep it as simple as this: there’s a parser object, and it will encapsulate code and data for us. The code in the object (the object’s method calls) consists of subroutines that the parser will call when it finds something of interest. For example, if the parser finds a start tag for an element, the object’s start_element() method is called. Other code, such as little utility rou- tines, will also reside in this object. We can even use the object to hold data for us (e.g., the name of the host whose record we’re parsing), instead of using global variables like we did in the previous section. That’s it—that’s all the OOP knowledge you’ll need for the rest of this section. 222 | Chapter 6: Working with Configuration Files Let’s look at the first of those method definitions. Here’s the method that gets called when the parser finds the start tag for an element: # %hosts is used to collect all of the parsed data # (yes, we could keep this in the object itself) my %hosts; sub start_element { my ( $self, $element ) = @_; $self->_contents(''); # these weird '{}something' hash keys are using James Clark notation; # we'll address this convention in a moment when we talk about # XML namespaces if ( $element->{LocalName} eq 'host' ) { $self->{current_host} = $element->{Attributes}{'{}name'}{Value}; $hosts{ $self->{current_host} }{type} = $element->{Attributes}{'{}type'}{Value}; $hosts{ $self->{current_host} }{os} = $element->{Attributes}{'{}os'}{Value}; } if ( $element->{LocalName} eq 'interface' ) { $self->{current_interface} = $element->{Attributes}{'{}name'}{Value}; $hosts{ $self->{current_host} }{interfaces} { $self->{current_interface} }{type} = $element->{Attributes}{'{}type'}{Value}; } $self->{current_element} = $element->{LocalName}; $self->SUPER::start_element($element); } This subroutine has obviously been modified from its equivalent in the XML::Parser example, so let’s look at the differences. The first change is in the arguments passed to the event handler. XML::SAX passes to its handlers a reference to the parser object as the first argument and handler-specific data in the rest of the arguments. start_element() gets information in its second argument about the element the parser has just seen via a reference to a data structure that looks like this: 0 HASH(0xa30624) 'LocalName' => 'host' 'Name' => 'host' 'NamespaceURI' => undef 'Prefix' => '' 'Attributes' => HASH(0xa30768) '{}name' => HASH(0xa3033c) 'LocalName' => 'name' 'Name' => 'name' 'NamespaceURI' => '' 'Prefix' => '' 'Value' => 'agatha' Configuration File Formats | 223 '{}os' => HASH(0xa307f8) 'LocalName' => 'os' 'Name' => 'os' 'NamespaceURI' => '' 'Prefix' => '' 'Value' => 'linux' '{}type' => HASH(0xa30678) 'LocalName' => 'type' 'Name' => 'type' 'NamespaceURI' => '' 'Prefix' => '' 'Value' => 'server' It’s a hash with the fields described in Table 6-1. Table 6-1. Contents of the hash passed to start_element() Hash key Contents LocalName The name of the element, without any namespace prefix (see the sidebar “XML Namespaces” for more info on what that means) Name The name of the element, including the namespace prefix Prefix The namespace prefix for this element (if it has one) NamespaceURI The URI for the element’s namespace (if it has one) Attributes A hash of hashes containing information about the element’s attributes XML Namespaces Up to now, I’ve intentionally avoided any mention of the concept of XML namespaces. They don’t usually show up in smallish XML documents (like config files), and I didn’t want to add an extra layer of complexity to the rest of the material. But XML::SAX pro- vides namespace information to its event handlers, so we should give them at least a passing glance before moving on. If you’d like more detail about XML namespaces, the best place to start is the official W3C recommendation on the subject at http://www.w3 .org/TR/REC-xml-names/. XML namespaces are a way of making sure that elements in a document are unique and partitioned. If our document had an element called , its contents or subelements could refer to either a color or a fruit. For a contrived case, imagine the situation where a design firm needs to provide information about a new juice box to a citrus grower’s organization. The file could easily use elements for both senses of the word. With namespaces, you can add an extra attribute (xmlns) to disambiguate an element: ... 224 | Chapter 6: Working with Configuration Files Now everything in the element has a namespace associated with that URI* and it’s clear just what kind of orange we’re talking about.† A slightly more complex XML namespace syntax lets you define multiple namespaces in the same element, each with its own identifying string (called a prefix): #ffa500 Citrus sinensis In this case, we’ve defined two different namespaces with the prefixes color and fruit. We can then use these prefixes to label the two subelements appropriately with namespace: orange, as in the preceding code, so there is no confu- sion. I did say the example was contrived.... One last related note: James Clark, source of much impressive work in the XML world (including the expat parser we discussed earlier in the chapter) invented an informal syntax for displaying namespaces that has become known as “James Clark notation.” It uses the form <{namespace}element_name>. In this notation, our first example from earlier would be written as: <{}orange> ... This syntax isn’t accepted by any XML parser, but it is used in places like XML::SAX’s representation of attributes. If an element has attributes (as in the sample data we just saw), the attributes are stored in their own hash of hashes data structure. The keys of that hash are the attribute names, represented in James Clark notation (see the previous sidebar). The content of each key is a hash whose keys are described in Table 6-2. Table 6-2. Contents of the hash used to store attribute information Hash key Contents LocalName The name of the attribute without any namespace prefix Name The name of the attribute including the namespace prefix (if it has one) Prefix The namespace prefix for this element (if it has one) NamespaceURI The URI for the attribute’s namespace (if it has one and the attribute was prefixed) Value The attribute’s value * The URI here is just used as a convenient unique string that will describe the namespace. It doesn’t have to be real—the parser never opens a network connection to attempt to reach the URI. It is considered cool to have something at that URI for documentation purposes (e.g., Transform), but this isn’t required. † If it helps you understand the concept, think of XML namespaces like package statements in Perl. package foo puts all of the subsequent code (until another package statement comes along) into the foo namespace. This lets you have two scalars called $orange in the same program, each in its own namespace. Configuration File Formats | 225 Our configuration file didn’t use namespaces, so the attributes in our data structure all start out with empty prefixes ({}). This is what makes their hash keys look so funny. Now that you understand how information about an element is passed into start_element(), hopefully the code shown earlier will start to make more sense. If you ignore the _content() and SUPER::start_element() methods (we’ll get to those in a few moments), all the code is doing is either copying information out of the $element data structure into our %hosts hash or squirreling away information from $element (like the current element name) into the parser object‡ for later use. That’s what happens when the parse encounters a new start tag. Let’s see what it does for the textual contents (as opposed to another subelement) of the element: sub characters { my ( $self, $data ) = @_; $self->_contents( $self->_contents() . $data->{Data} ); $self->SUPER::characters($data); } You’ll notice this is much smaller than the Text() subroutine in our XML::Parser ex- ample. All it does is use a separate _contents() method* to collect the data it receives (ignore the second mysterious SUPER:: line, I’ll explain it soon). That method looks like this: # stash any text passed to us in the parser object or return the # current contents of that stash sub _contents { my ( $self, $text ) = @_; $self->{'_contents'} = $text if defined $text; return $self->{'_contents'}; } The characters() method is much smaller than the Text() subroutine because of a subtle but important difference in how the two work. With Text(), the module author guaranteed that it would receive (to quote the docs) “accumulated non-markup text.” That’s not the way it works for characters(). The XML::SAX tutorial says: “A SAX parser has to make no guarantees whatsoever about how many times it may call characters for a stretch of text in an XML document—it may call once, or it may call once for every character in the text.” As a result, we can’t make the same assumptions that we did before in our XML::Parser code about when we have the entire text contents of the element to be stored. Instead, we have to push that work into end_element(), because ‡ OOP purists will probably stomp on me with their steel-toed boots because the code isn’t using “getters” and “setters” for that squirreling. I’m trying to keep the amount of code in the example down to keep the focus on XML::SAX, but point taken, so you can stop kicking me now. * If just to placate the OOP thugs from the last footnote just a little bit.... 226 | Chapter 6: Working with Configuration Files by then we’re certain we’ve collected the contents of an element. The first thing the end_element() handler does is retrieve the current contents of the collected data and strip the leading/following whitespace, just in case we want to store it for posterity: sub end_element { my ( $self, $element ) = @_; my $text = $self->_contents(); $text =~ s/^\s+|\s+$//g; # remove leading/following whitespace if ( $self->{current_element} eq 'arec' or $self->{current_element} eq 'addr' ) { $hosts{ $self->{current_host} }{interfaces} { $self->{current_interface} }{ $self->{current_element} } = $text; } if ( $self->{current_element} eq 'cname' ) { push( @{ $hosts{ $self->{current_host} }{interfaces} { $self->{current_interface} }{cnames} }, $text ); } if ( $self->{current_element} eq 'service' ) { push( @{ $hosts{ $self->{current_host} }{services} }, $text ); } $self->SUPER::end_element($element); } 1; # to make sure the HostHandler module will load properly One quick warning about this code: it makes no attempt to handle mixed content situations like this: This is some text in the element. This is some text in a subelement This is some more text in the element. You can handle mixed content using XML::SAX, but it increases the complexity of the event handlers beyond what I wanted to show for a basic SAX2 example. We’re practically done with our exploration of SAX2-based XML reading. There are a number of more advanced SAX techniques that we won’t have room to explore. One of those holds the secret to the lines of code in our example that began $self–>SUPER::, so I want to at least mention it. SAX2-based coding makes it very easy to construct multistage pipelines, like Unix-style pipes. A piece of SAX2 code can take Configuration File Formats | 227 in a stream of SAX2 events, transform/filter them in some fashion, and then pass the events on to the next handler. XML::SAX makes it relatively easy to hook up handlers (XML::SAX::Machine by Barrie Slaymaker makes it very easy). The $self->SUPER:: calls in each of our methods makes sure that the events get passed on correctly should our code be placed somewhere before the end of a pipeline. Even if you don’t think it will happen to your code, it is good practice to include those lines. Working with XML using a hybrid approach (XML::Twig) XML::Twig Pros and Cons Benefits: • It offers a very Perl-centric approach. • It’s engineered to handle very large data sets in a memory/CPU-efficient and gran- ular manner. It is especially good in those scenarios where you need to operate on a small portion of a much larger document. You can instruct XML::Twig to process only a particular element and its subelements, and it will create an in-memory representation of just that part of your data. You can then flush this document fragment from memory and replace it with the next instance of the desired element. • It has the ability to use XPath-like selectors when choosing what data to process. These selectors make it easy to construct callbacks (i.e., give it an XPath selector and it will run a piece of code when it finds something in the document that matches the selector). • The module offers a nice compromise between tree-based processing (similar to XML::LibXML’s DOM features) and stream-based processing (like the SAX2 pro- cessing model). • It can also read HTML (it uses HTML::TreeBuilder’s XML export, so it needs to read the entire doc into memory). • It has options to maintain attribute order and to pretty-print in a format that’s easy to read. • Its emphasis is on DWIM (do what I mean). • It has superb documentation ( and author support. Drawbacks: • It’s not particularly standards-compliant (in the way XML::SAX follows SAX2 and XML::LibXML implements the W3C DOM model), but that may not matter to you. • It implements only a subset of the XPath 1.0 standard (albeit a very useful subset). • Depending on the situation, it can be slower than XML::LibXML. • It uses expat as its underlying parser (probably not an issue because it’s so solid, but expat doesn’t see much active maintenance). When should you use this module? XML::Twig is especially good for situations where you are processing a large data set but only need to operate on a smaller subset of that 228 | Chapter 6: Working with Configuration Files data. Once you grok its basic way of thinking about the world (as “twigs”), it can be a pleasure for someone with Perl and a dash of XPath experience to use. There’s considerable overlap between XML::Twig’s functionality and the functionality of the modules we’ve seen so far. Like the others, Michel Rodriguez’s XML::Twig can create and manipulate an in-memory tree representation of an XML document (DOM- like) or parse the data while providing event-based callbacks. To keep this section short and sweet, I’m going to focus on the unique features XML::Twig provides. The excellent documentation and the module’s website ( can provide details on the rest of its functionality. XML::Twig’s main premise is that an XML document should be processed as a bunch of subtrees. In Appendix B, I introduce the notion that you can represent an XML docu- ment as a big tree structure starting from the root element of the document. XML::Twig takes this one step further: it allows you to select certain subtrees of that structure (“twigs”) as you parse the document and operate on those twigs while ignor- ing the rest of the data whizzing by. This selection takes place using a subset of the XPath 1.0 specification. Before parsing, you provide a set of XPath selectors and their callbacks (the Perl code to run when the selector matches). This is similar to some of the callback-based code we’ve seen earlier in this chapter, except now we’re thinking about firing off code based on finding subtrees of a document rather than just certain elements or parse events. Let’s see how this works in practice by looking at two simple examples. We’ll use the same sample XML data file for these examples as well. First, here’s a simple example of data extraction from an XML document. If we wanted to extract just the elements and their contents, we’d write:† use XML::Twig; my $twig = XML::Twig->new( twig_roots => { # $_ gets set to the element here 'host/interface' => sub { $_->print }, }, pretty_print => 'indented', ); $twig->parsefile('config.xml'); and the output would begin like this: † If we didn’t want to write any code at all, XML::Twig comes with an xml_grep utility that would allow us to write xml_grep 'host/interface' config.xml. There is an XML::LibXML-based version of this utility at http: // Configuration File Formats | 229 ... The key here is the twig_roots option, which lets XML::Twig know that we only care about subtrees/twigs in the data found in each element. For each twig found matching that specification, we ask the module to (pretty-)print its contents. Let’s follow that extraction example with a slightly more complex transformation ex- ample. If we wanted to modify our sample document so that all of the elements became elements instead (complete with port num- bers as attributes), we would write something like this: use XML::Twig; use LWP::Simple; my %port_fix = ( 'DNS' => 'domain', 'IMAP4' => 'imap', 'firewall' => 'all' ); my $port_list_url = ''; my %port_list = &grab_iana_list; my $twig = XML::Twig->new( twig_roots => { 'host/service' => \&transform_service_tags }, twig_print_outside_roots => 1, ); $twig->parsefile('config.xml'); # change -> and add that service's port number # as an attribute sub transform_service_tags { my ( $twig, $service_tag ) = @_; my $port_number = ( $port_list{ lc $service_tag->trimmed_text } or $port_list{ lc $port_fix{ $service_tag->trimmed_text } } or $port_fix{ lc $service_tag->trimmed_text } ); $service_tag->set_tag('port'); $service_tag->set_att( number => $port_number ); 230 | Chapter 6: Working with Configuration Files $twig->flush; } # retrieve the IANA allocated port list from its URL and return # a hash that maps names to numbers sub grab_iana_list { my $port_page = get($port_list_url); # each line is of the form: # service port/protocol explanation # e.g.: # http 80/tcp World Wide Web HTTP my %ports = $port_page =~ /([\w-]+)\s+(\d+)\/(?:tcp|udp)/mg; return %ports; } Let’s take this apart step by step. First, we (somewhat gratuitously, I admit) grab the IANA-allocated port number list and return it as a hash for further lookups. Some of the service names we’ve used in our example won’t be found in that assignment list, so we also load up a hash with the information we’ll need to fix up any lookups that fail. Then we load XML::Twig with the selector we need and a reference to the subroutine that it will run when it finds that selector. In the same step, we also set twig_print_outside_roots, which tells XML::Twig to pass along any data from the document that doesn’t match the twig_roots selector verbatim (as opposed to simply dropping it, as in our first example). With this defined, we pull the trigger and the parse commences on our sample config file. The parse will hum along, passing input data to output data untouched until it finds a twig that matches the selector. When this happens, the entire twig, plus the element that was parsed to yield the twig, will be sent to the handler associated with that selector. In this case, the element in question is and it contains a single piece of text: the name of the service. We request the whitespace-“trimmed” version of that text and use it to look up the port number in the hash we built from the IANA data. If we don’t find it in the first lookup, we try again with a fixed-up version of the name (e.g., we look up “domain” if “DNS” wasn’t found). If this second attempt fails, we give up on the IANA list and pull the value we need from the fixed-up hash itself (e.g., for the service “firewall,” which isn’t a network service with an assigned port). XML::Twig makes it very simple to perform the actual transformation. set_tag changes the tag name and set_att lets us insert a new attribute with the port number we just retrieved. The final step for the handler is to instruct XML::Twig to print out the contents of the twig and remove it from memory before moving on in the document. This flush step is optional, but it is one of the keys to XML::Twig’s memory efficiency. Once flushed (or purged if you don’t need to print that twig), the subtree you were working on no longer resides in memory, so each new subtree found takes up essentially the same space instead of accumulating in memory, like it would in a DOM-based representation. Configuration File Formats | 231 XML::Twig has a ton of other methods available that make working with XML pretty easy for a Perl programmer. This section has just presented some of the essential pieces that differentiate it from the other approaches we explored; be sure to consult the documentation for more details. With that, we can conclude our tour of the top three best-practice approaches (as of this writing) for dealing with XML from Perl. Now that you have some best-of-breed tools in your toolkit, you should be able to handle any XML challenge that comes your way using an approach well suited to that situation. As a final note for this section, there are a number of up-and-coming modules that will also deserve your attention as they mature. Two of the more interesting ones I’d rec- ommend you check out if you are going to work with XML are XML::Rules by Jenda Krynicky and XML::Compile by Mark Overmeer. But what if, after all of that, you decide XML itself is close, but not exactly the best format for your particular needs? Well.... YAML Some people think that XML has too much markup for each piece of content and would prefer something with fewer angle brackets. For these people, there is a lighter-weight format called YAML (which stands for YAML Ain’t Markup Language). It’s trying to solve a different problem than XML, but it often gets pressed into service for similar reasons. YAML tries to strike a balance between structure and concision, so it looks a little cleaner to the average eye. Here’s a fairly literal translation from the sample XML config file we rubbed raw in our discussion of XML: --- network: description: name: Boston text: This is the configuration of our network in the Boston office. hosts: - name: agatha os: linux type: server interface: - name: eth0 type: Ethernet addr: arec: cname: - service: - SMTP - POP3 - IMAP4 232 | Chapter 6: Working with Configuration Files - name: gil os: linux type: server interface: - name: eth0 type: Ethernet addr: arec: cname: - service: - HTTP - HTTPS - name: baron os: linux type: server interface: - name: eth0 type: Ethernet addr: arec: cname: - - - service: - DNS - NTP - LDAP - LDAPS - name: mr-tock os: openbsd type: server interface: - name: fxp0 type: Ethernet addr: arec: cname: - service: - firewall - name: krosp os: osx type: client interface: - name: en0 type: Ethernet addr: arec: - name: en1 type: AirPort Configuration File Formats | 233 addr: arec: - name: zeetha os: osx type: client interface: - name: en0 type: Ethernet addr: arec: - name: en1 addr: type: AirPort arec: Already this is probably looking a little easier on the eyes. It’s a fairly literal translation because it attempts to preserve all of the XML attribute names (YAML doesn’t have tag attributes per se, so all of the attributes and the contents of each element are listed in the same way). If direct conversion weren’t a priority, we’d definitely want to write our config file in an even more straightforward way. For example, here’s a repeat of the YAML file we generated earlier in the chapter while mucking about with XML::Parser: agatha: interfaces: eth0: addr: arec: cnames: - type: Ethernet os: linux services: - SMTP - POP3 - IMAP4 type: server ... zeetha: interfaces: en0: addr: arec: type: Ethernet en1: addr: arec: type: AirPort os: osx type: client 234 | Chapter 6: Working with Configuration Files There’s not a big difference, but hopefully you’ll get a sense that it is possible to simplify your data file even further by eliminating extraneous labels. The Perl module to parse YAML‡ is called, strangely enough, YAML and is used like this: use YAML qw(DumpFile); # finds and loads an appropriate YAML parser my $config = YAML::LoadFile('config.yml'); # (later...) dump the config back out to a file YAML::DumpFile( 'config.yml' , $config ); The YAML module itself is just a frontend to other YAML parsers that provides a common interface similar to what we saw with XML::SAX. By default it provides simple Load/ Dump procedure calls that operate on in-memory data, though you can also use LoadFile and DumpFile to work with files. That’s almost all there is to it: you either Load YAML data from some place or Dump a YAML representation of the data. If you’d prefer a more object-oriented way of working with YAML, Config::YAML can provide it. There is also a screamingly fast parser/dumper for YAML built on the libyaml library called YAML::XS. If you don’t need a pure-Perl parser, that’s the recom- mended module to use (the YAML module will attempt to use it by default if it is available). And with that last simple but very powerful config file format, we can start to wrap up the chapter. There are an infinite number of possible formats for config files, but at least now we’ve hit the highlights. All-in-One Modules If all this talk about picking the right module for config parsing has started to wear on you, let me ease us toward the end of this chapter with a quick look at a set of modules that can help you sidestep the choice. Config::Context is Michael Graham’s wrapper around the Config::General, XML::Simple, and Config::Scoped modules that allows you to use a single module for each of the formats those modules handle. On top of this, it also adds contexts (as in Apache), so you can use tags in those file formats. If you crave a module that supports a larger menu of config file formats, Config::Auto by Jos Boumans can handle colon/space/equals-separated key/value pairs, XML formats, Perl code, .ini formats, and BIND9-style and .irssi config file for- mats. Not only that, but it will (by default) guess the format it is parsing for you without further specification. If that’s too magical for you, you can specify the format yourself. ‡ One nice property of YAML is that it is language-independent. There are YAML parsers and emitters for Ruby, Python, PHP, Java, OCaml, and even JavaScript. All-in-One Modules | 235 Advanced Configuration Storage Mechanisms You’re probably sick of talking about config files at this point (I don’t blame you), so let’s end this chapter with a brief mention of some of the more advanced alternatives. There are a number of other reasonable places to stash configuration information.* Shared memory segments can work well when performance is the key criterion. Many systems are now keeping their configuration info in databases via DBI (see Chap- ter 7). Others have specific network servers to distribute configuration information. These are all interesting directions to explore, but beyond the scope of this book. Module Information for This Chapter Modules CPAN ID Version Readonly ROODE 1.03 Storable (ships with Perl) AMS 2.15 DBM::Deep RKINYON 1.0013 Text::CSV::Simple TMTM 1.00 Text::CSV_XS JWIED 0.23 Config::Std DCONWAY 0.0.4 Config::General TLINDEN 2.31 Config::Scoped GAISSMA 0.11 Config::Grammar DSCHWEI 1.02 XML::Writer JOSEPHW 0.606 XML::Simple GRANTM 2.18 XML::LibXML PAJAS 1.69 XML::SAX GRANTM 0.96 XML::Parser MSERGEANT 2.36 XML::Twig MIROD 3.32 LWP::Simple (ships with Perl) GAAS 5.810 YAML INGY 0.68 Config::YAML MDXI 1.42 YAML::XS NUFFIN 0.29 Config::Context MGRAHAM 0.10 Config::Auto KANE 0.16 * There are also a number of other unreasonable places; for example, hidden in image files using Acme::Steganography::Image::Png or in a play via Acme::Playwright. 236 | Chapter 6: Working with Configuration Files References for More Information Some of the material in this chapter is revised and modified from a column that I originally wrote for the February 2006 issue of the USENIX Association’s ;login mag- azine ( Perl Best Practices (, by Damian Conway (O’Reilly), has a good section on config files. XML and YAML and both contain copious information. Microsoft and IBM are very serious about XML. hosts the Perl-XML mailing list. It (along with its archive) is one of the best sources on this topic. is the actual XML 1.0 specification. Anyone who does anything with XML eventually winds up reading the full spec, but for anything but quick reference checks, I recommend reading an annotated version like those mentioned in the next two citations. is a good reference for articles and XML links. It also offers an excellent annotated version of the XML specification created by Tim Bray, one of its authors. XML: The Annotated Specification, by Bob DuCharme (Prentice Hall), is another excellent annotated version of the specification, chock-full of XML code examples. XML Pocket Reference, Third Edition (, by Simon St.Laurent and Michael Fitzgerald (O’Reilly), is a concise but surprisingly com- prehensive introduction to XML for the impatient. Learning XML, Second Edition (, by Erik T. Ray (O’Reilly) and Essential XML: Beyond Markup, by Don Box et al. (Addison- Wesley) are good places to learn the range of XML-based technologies, including XPath. The latter is much more dense and less Perl-friendly but has a level of depth I haven’t found in any other reference. Perl and XML (, by Erik T. Ray and Jason McIntosh (O’Reilly) is worth a look as well, though it was based on the XML modules current at that time. The Perl XML world has changed some since it was published in 2002, but it is a good reference for those modules that are still in use. is a hub for Perl XML-related development. The FAQ and Perl SAX pages at that site are important material you need to read. References for More Information | 237 is the official website for the Gnome libxml library on which XML::LibXML is based. You’ll eventually find yourself here as you try to understand some arcane part of XML::LibXML. is the official website for SAX2. Object Oriented Perl: A Comprehensive Guide to Concepts and Programming Techni- ques, by Damian Conway (Manning), is the best place to learn about OOP in Perl. Understanding OOP in Perl is crucial for using XML::SAX well. is the official website for XML::Twig and is chock-full of good documentation, tutorials, presentations, etc. is the home base for everything YAML-related. 238 | Chapter 6: Working with Configuration Files CHAPTER 7 SQL Database Administration What’s a chapter on database administration doing in a system administration book? There are several strong reasons for people with interests in Perl and system adminis- tration to become database-savvy: • A not-so-subtle thread running through several chapters of this book is the increasing importance of databases to modern-day system administration. We’ve used databases (albeit simple ones) to keep track of user and machine information, but that’s just the tip of the iceberg. Mailing lists, password files, and even the Windows-based operating system registry are all examples of databases you prob- ably interact with every day. All large-scale system administration packages (e.g., offerings from CA, Tivoli, HP, and Microsoft) depend on database backends. If you are planning to do any serious system administration, you are bound to bump into a database eventually. • Database administration is a play within a play for system administrators. Database administrators (DBAs) have to contend with, among other things: — Logins/users — Log files — Storage management (disk space, etc.) — Process management — Connectivity issues — Backups — Security/role-based access control (RBAC) Sound familiar? We can and should learn from both knowledge domains. • Perl is a glue language, arguably one of the best. Much work has gone into Perl/ database integration, thanks mostly to the tremendous energy surrounding web development. We can put this effort to work for us. Though Perl can integrate with several different database formats (Unix DBM, Berkeley DB, etc.), we’re going to 239 pay attention in this chapter to Perl’s interface with large-scale database products. We address other formats elsewhere in this book. • Many applications we use or support require some database for storing information (e.g., Bugzilla, Request Tracker, calendars, etc.). In order to have a good under- standing of the applications we support, we need to be able to mess with the storage beneath the databases and make sure they’re running efficiently. • This is going to sound a bit obvious, but another reason why sysadmins care about databases is that they store information. Sometimes it’s even our information: logs, performance metrics (e.g., for trend analysis and capacity planning), meta- information about users and systems, and so on. In order to be a database-literate system administrator, you have to speak a little Struc- tured Query Language (SQL), the lingua franca of most commercial and several non- commercial databases. Writing scripts in Perl for database administration requires some SQL knowledge because these scripts will contain simple embedded SQL state- ments. See Appendix D for enough SQL to get you started. The examples in this chapter use largely the same data sets that we introduced in previous chapters to keep us from straying from the system administration realm. Interacting with a SQL Server from Perl Once upon a time, there were many Perl modules for interacting with different database systems. Each time you wanted to use a database by a certain vendor, you had to look for the right module for the task and then learn that module’s way of doing things. If you switched databases mid-project, you likely had to rewrite all of your code to use an entirely different module. And then the DataBase Interface (DBI) by Tim Bunce came along, and things got much, much better in the Perl universe. DBI can be thought of as “middleware.” It forms a layer of abstraction that allows the programmer to write code using generic DBI calls, without having to know the specific API of any particular database. It is then up to the DBI software to hand these calls off to a database-specific layer. The DBI module calls a DataBase Dependent (DBD) driver for this. This database-specific driver takes care of the nitty-gritty details necessary for communicating with the server in question. This is a great idea. It is so great that you see it not only in other languages (JDBC, etc.), but also in at least one OS platform: Windows has Open DataBase Connectivity (ODBC) built in. ODBC is not precisely a competitor to DBI, but there’s enough overlap and it’s a big enough presence in the Windows world that we’re going to have to give it some attention. Windows Perl programmers largely interact with ODBC data sources, so for their sake we’ll do a quick comparison. This will still be useful for non- Windows people to see because it’s not uncommon for ODBC to be the only program- matic method for interacting with certain “boutique” databases. 240 | Chapter 7: SQL Database Administration Figure 7-1 shows the DBI and ODBC architectures. In both cases, there is a (at least) three-tiered model: 1. An underlying database (Oracle, MySQL, Sybase, Microsoft SQL Server, etc.). 2. A database-specific layer that makes the actual server-specific requests to the server on behalf of the programmer. Programmers don’t directly communicate with this layer; they use the third tier. In DBI, a specific DBD module handles this layer. For example, when talking with an Oracle database, the DBD::Oracle module will be invoked. DBD modules are usually linked during the building process to a server- specific client library provided by the server vendor. With ODBC, a data source- specific ODBC driver provided by the vendor handles this layer. 3. A database-independent API layer. Soon, we’ll be writing Perl scripts that will communicate with this layer. In DBI, this is known as the DBI layer (i.e., we’ll be making DBI calls). In ODBC, one typically communicates with the ODBC Driver Manager via ODBC API calls. The DBI Architecture Driver manager Application Data source Driver Driver Driver Data source Data source Perl script using DBI API methods DBI API Other drivers MySQL engine MySQL driver XYZ engine XYZ driver Oracle engine Oracle driver The ODBC Architecture Figure 7-1. DBI and ODBC architectures The beauty of this model is that most code written for DBI or ODBC is portable between different servers from different vendors. The API calls made are the same, independent of the underlying database—at least that’s the idea, and it holds true for most database programming. Unfortunately, the sort of code we’re most likely to write (i.e., database Interacting with a SQL Server from Perl | 241 administration code) is bound to be server-specific, since virtually no two servers are administered in even a remotely similar fashion.* Experienced system administrators love portable solutions, but they don’t expect them. With the background in place, let’s move as fast as possible toward writing some code. Interacting with basic DBI will be straightforward because there’s only one DBI module. What about ODBC? That’s an interesting question, as there are two common ways to go about interacting with ODBC in Perl: once upon a time the Win32::ODBC module was the primary conduit, but more recently a DBD module for the DBI framework called DBD::ODBC has become the preferred method interaction method (it is even now rec- ommended by Win32::ODBC’s author). DBD::ODBC essentially subsumes the ODBC world into DBI, making it just one more data source. We’ll see an example of it in action shortly. For our DBI example code, we’ll use the MySQL and Oracle servers; for ODBC, we’ll use the Microsoft SQL Server. Accessing Microsoft SQL Server from Unix Multiplatform system administrators often ask, “How can I talk to my Microsoft SQL Server installation from my Unix machine?” If an environment’s central administration or monitoring system is Unix-based, a new Microsoft SQL Server installation presents a challenge. I know of four ways to deal with this situation. Choices 2 and 3 in the following list are not Microsoft SQL Server-specific, so even if you are not using Microsoft’s RDBMS in your environment you may find that these techniques come in handy some day. Your options are: 1. Build and use DBD::Sybase. DBD::Sybase will require some underlying database communication libraries, and there are two sets of libraries available that will fit the bill. The first one, the Sybase OpenClient libraries, may be available for your platform (e.g., they ship for free with some Linux distributions as part of the Sybase Adaptive Server Enterprise). Your second option is to install the FreeTDS libraries found at See the instructions on this site for building the correct protocol version for the server you will be using. 2. Use a “proxy” driver. There are two DBD proxy modules that ship with DBI: the oldest is called DBD::Proxy, and the more recent addition is DBD::Gofer. Both allow you to run a small network server on your SQL Server machine to transparently proxy requests from your Unix clients to the server. 3. Acquire and use Unix ODBC software via DBD::ODBC. Several vendors, including MERANT ( and OpenLink Software (, will sell such software to you, or you can attempt to use the work of the various open source developers. For more information, see the iODBC (http: // and unixODBC ( home pages. You will * Microsoft SQL Server was initially derived from Sybase source code, so it’s one of the rare counter-examples. 242 | Chapter 7: SQL Database Administration need both an ODBC driver for your Unix platform (provided by the database ven- dor) and an ODBC manager (such as unixODBC or iODBC). 4. Microsoft SQL Server (starting with version 2000) can listen for database queries over HTTP or HTTPS without the need for another web server (such as IIS). The results are returned in an XML format that is easily processed with the methods we saw in Chapter 6. Using the DBI Framework Here are the basic steps for using DBI:† 1. Load the necessary Perl module. There’s nothing special here, we just need to include this line: use DBI; 2. Connect to the database and receive a connection handle. The Perl code to establish a DBI connection to a MySQL database and return a database handle looks like this: # connect to the database named $database using the given # username and password, and return a database handle my $database = 'sysadm'; my $dbh = DBI->connect("DBI:mysql:$database",$username,$pw); die "Unable to connect: $DBI::errstr\n" unless (defined $dbh); DBI will load the low-level DBD driver (DBD::mysql) for us prior to actually con- necting to the server. We then test if the connect() succeeded before continuing. DBI provides RaiseError and PrintError options for connect(), should we want DBI to test the return code of all DBI operations in that session and automatically complain about errors when they happen. For example, if we used this code: $dbh = DBI->connect("DBI:mysql:$database", $username,$pw,{RaiseError => 1}); DBI would call die for us if the connect() failed. 3. Send SQL commands to the server. With our Perl module loaded and a connection to the database server in place, it’s showtime! Let’s send some SQL commands to the server. We’ll use some of the SQL tutorial queries from Appendix D for examples. These queries will use the Perl q convention for quoting (i.e., something is written as q{something}), just so we don’t have to worry about single or double quotes in the actual queries them- selves. Here’s the first of the two DBI methods for sending commands: † For more information on DBI, see Programming the Perl DBI ( by Alligator Descartes and Tim Bunce (O’Reilly). Using the DBI Framework | 243 my $results=$dbh->do(q{UPDATE hosts SET bldg = 'Main' WHERE name = 'bendir'}); die "Unable to perform update:$DBI::errstr\n" unless (defined $results); $results will receive either the number of rows updated, or undef if an error occurs. Though it is useful to know how many rows were affected, that’s not going to cut it for statements like SELECT, where we need to see the actual data. This is where the second method comes in. To use the second method, you first prepare a SQL statement for use and then ask the server to execute it. Here’s an example: my $sth = $dbh->prepare(q{SELECT * from hosts}) or die 'Unable to prep our query:'.$dbh->errstr."\n"; my $rc = $sth->execute or die 'Unable to execute our query:'.$dbh->errstr."\n"; prepare() returns a new creature we haven’t seen before: the statement handle. Just as a database handle refers to an open database connection, a statement handle refers to a particular SQL statement we’ve prepare()d. Once we have this statement handle, we use execute to actually send the query to our server. Later, we’ll be using the same statement handle to retrieve the results of our query. You might wonder why we bother to prepare() a statement instead of just exe- cuting it directly. prepare()ing a statement gives the DBD driver (or more likely, the database client library it calls) a chance to parse and mull over the SQL query. Once a statement has been prepare()d, we can execute it repeatedly via our state- ment handle without parsing it (or deciding how the query will be played out in the server) over and over again. Often this is a major efficiency win. In fact, the default do() DBI method does a prepare() and then an execute() behind the scenes for each statement it is asked to execute. Like the do call we saw earlier, execute() returns the number of rows affected. If the query affects zero rows, the string 0E0 is returned to allow a Boolean test to succeed. −1 is returned if the number of rows affected is unknown by the driver. Before we move on to how the results of a query are retrieved, it is worth mentioning one more twist on the prepare() theme that is supported by most DBD modules: placeholders, also called positional markers, allow you to prepare() a SQL state- ment that has holes in it to be filled at execute() time. This allows you to construct queries on the fly without paying most of the parse-time penalty. The question mark character (?) is used as the placeholder for a single scalar value. Here’s some Perl code to demonstrate the use of placeholders:‡ ‡ This demonstrates the most common case, where the placeholders represent simple strings to be filled into the query. If you’ll be substituting in more complex data types, like SQL datetimes, you’ll need to use the DBI bind_param() method before calling execute(). 244 | Chapter 7: SQL Database Administration my @machines = qw(bendir shimmer sander); my $sth = $dbh->prepare(q{SELECT name, ipaddr FROM hosts WHERE name = ?}); foreach my $name (@machines){ $sth->execute($name); do-something-with-the-results } Each time we go through the foreach loop, the SELECT query is executed with a different WHERE clause. Multiple placeholders are straightforward: $sth->prepare( q{SELECT name, ipaddr FROM hosts WHERE (name = ? AND bldg = ? AND dept = ?)}); $sth->execute($name,$bldg,$dept); The other bonus you get by using placeholders is automatic quoting of the arguments. Now that we know how to retrieve the number of rows affected by non-SELECT SQL queries, let’s look into retrieving the results of our SELECT requests. 4. Retrieve SELECT results. DBI offers three different approaches for retrieving the results of a query. We’re going to look at each of them in turn because they all come in handy at one time or another, depending on the situation and programming context. Two of these mechanisms are similar to the cursors idea we discussed briefly in Appendix D. With these mechanisms we expect to iterate over the results one row at a time, calling some method each time we want the next row of results returned to our program. The first of these mechanisms—using bind_col() or bind_columns() with fetchrow_arrayref()—is often the best tack, because it is both the most efficient and the most “magical” of the choices. Let’s take a look at how it works. After the execute(), we tell DBI to place the answers we get back into the scalar or the collection of scalars (list or hash) of our choosing. That binding between the results and the variables is done like this: # imagine we just finished a query like SELECT first,second,third FROM table my $first; my $second; my $third; $sth->bind_col(1, \$first); # bind first column of search result to $first $sth->bind_col(2, \$second); # bind second column $sth->bind_col(3, \$third); # bind third column, and so on # or perform all of the binds in one shot: $sth->bind_columns(\$first, \$second, \$third); Binding to whole arrays or to elements in a hash is equally as easy using the magical \(...) syntax: $sth->bind_columns( \(@array) ); # $array[0] gets the first column # $array[1] get the second column... Using the DBI Framework | 245 # we can only bind to the hash elements, not to the hash itself $sth->bind_col(1, \$hash{first} ); $sth->bind_col(2, \$hash{second} ); Now, each time we call fetch(), those variables magically get populated with another row from the results of our query: while ($sth->fetch){ # do something with $first, $second and $third # or $array[0], $array[1],... # or $hash{first}, $hash{second} } It turns out that fetch() is actually an alias for the method call fetchrow_arrayref(), giving us a nice segue to the second method of retrieving SELECT results from DBI. If you find the magical nature of binding columns to be a bit too magical or you’d prefer to receive the results back as a Perl data structure so you can manipulate the data, there are a number of methods you can call. In DBI, we call one of the methods in Table 7-1 to return data from the result set. Table 7-1. DBI methods for returning data Name Returns Returns if no more rows fetchrow_arrayref() An array reference to an anonymous array with values that are the columns of the next row in a result set undef fetchrow_array() An array with values that are the columns of the next row in a result set An empty list fetchrow_hashref() A hash reference to an anonymous hash with keys that are the column names and values that are the values of the columns of the next row in a result set undef fetchall_arrayref() A reference to an array of arrays data structure A reference to an empty array fetchall_hashref($key_field) A reference to a hash of hashes. The top-level hash is keyed by the unique values returned from the $key_field column, and the inner hashes are structured just like the ones we get back from fetchrow_hashref() A reference to an empty hash Two kinds of methods are listed: single row (fetchrow_) methods and entire data set (fetchall_) methods. The fetchrow_ methods return a single row from the re- turned results, just like what we’ve seen so far. fetchall_ methods take this one step further and return the entire result set in one fell swoop (essentially by running the appropriate fetchrow_ as many times as necessary to retrieve the data). Be careful to limit the size of your queries when using this method because it does pull 246 | Chapter 7: SQL Database Administration the entire result set into memory. If you have a terabyte-sized result set, this may prove to be a bit problematic. Let’s take a look at these methods in context. For each of these examples, assume the following was executed just previously: $sth = $dbh->prepare(q{SELECT name,ipaddr,dept from hosts}) or die 'Unable to prepare our query: '.$dbh->errstr."\n"; $sth->execute or die "Unable to execute our query: ".$dbh->errstr."\n"; Here’s fetchrow_arrayref() in action: while (my $aref = $sth->fetchrow_arrayref){ print 'name: ' . $aref->[0] . "\n"; print 'ipaddr: ' . $aref->[1] . "\n"; print 'dept: ' . $aref->[2] . "\n"; } Just a quick warning about using fetchrow_arrayref() like this: any time you rely on the order of the elements in an array when you store/retrieve data (i.e., which field is which array element), you’ve created a booby trap in your code that is just waiting to spring on you. All you (or someone else working on your code) have to do is naïvely change the previous SELECT statement, and all bets about what is versus what should be in $aref->[2] are off. The DBI documentation mentions that fetchrow_hashref() is less efficient than fetchrow_arrayref() because of the extra processing it entails, but it can yield more readable and potentially more maintainable code. Here’s an example: while (my $href = $sth->fetchrow_hashref){ print 'name: ' . $href->{name} . "\n"; print 'ipaddr: ' . $href->{ipaddr}. "\n"; print 'dept: ' . $href->{dept} . "\n"; } Finally, let’s look at fetchall_arrayref(). Each reference returned looks exactly like something we’d receive from fetchrow_arrayref(), as shown in Figure 7-2. , , , ... [column1,column2,column3,column..] [column1,column2,column3,column..] [column1,column2,column3,column..] [column1,column2,column3,column..] row ref row ref row ref row ref Figure 7-2. The data structure returned by fetchrow_arrayref() Using the DBI Framework | 247 Here’s some code that will print out the entire query result set: $aref_aref = $sth->fetchall_arrayref; foreach my $rowref (@$aref_aref){ print 'name: ' . $rowref->[0] . "\n"; print 'ipaddr: ' . $rowref->[1] . "\n"; print 'dept: ' . $rowref->[2] . "\n"; print '-'x30,"\n"; } This code sample is specific to our particular data set because it assumes a certain number of columns in a certain order. For instance, we assume the machine name is returned as the first column in the query ($rowref->[0]). We can use some of the magic attributes (often called metadata) of statement han- dles to rewrite our result-retrieval code to make it more generic. Specifically, if we look at $sth->{NUM_OF_FIELDS} after a query, it will tell us the number of fields (columns) in our result set. $sth->{NAME} contains a reference to an array containing the names of each column. Here’s a more generic way to write the last example: my $aref_aref = $sth->fetchall_arrayref; my $numfields = $sth->{NUM_OF_FIELDS}; foreach my $rowref (@$aref_aref){ for (my $i=0; $i < $numfields; $i++){ print $sth->{NAME}->[$i].": ".$rowref->[$i]."\n"; } print '-'x30,"\n"; } Be sure to see the DBI documentation for more metadata attributes. The last method for returning data is through a series of “shortcut” methods, listed in Table 7-2, that prepare a SQL statement, execute it, and then return the data using one of the methods we saw earlier. Table 7-2. DBI shortcut methods Name Combines these methods into a single method selectcol_arrayref($stmnt) prepare($stmnt), execute(), (@{fetchrow_arrayref()})[0] (i.e., returns the first column for each row, though the column number(s) can be changed via an optional Columns argument) selectrow_array($stmnt) prepare($stmnt), execute(), fetchrow_array() selectrow_arrayref($stmnt) prepare($stmnt), execute(), fetchrow_arrayref() selectrow_hashref($stmnt) prepare($stmnt), execute(), fetchrow_hashref() selectall_arrayref($stmnt) prepare($stmnt), execute(), fetchall_arrayref() selectall_hashref($stmnt) prepare($stmnt), execute(), fetchall_hashref() 248 | Chapter 7: SQL Database Administration 5. Close the connection to the server. In DBI, this is simply: # disconnects handle from database $dbh->disconnect; Using ODBC from Within DBI The basic steps for using ODBC from DBI are pretty much identical to the steps we just discussed, with one twist. The hardest part is dealing with the arguments in the initial connect() call. ODBC requires one preliminary step before making a connection: we need to create a data source name (DSN). A DSN is a named reference that stores the configuration information (e.g., server and database name) needed to reach an information source like a SQL server. DSNs come in two flavors, user and system, distinguishing between connections available to a single user on a machine and con- nections available to any user or service.* DSNs can be created either through the ODBC control panel under Windows (see Figure 7-3), or programmatically via Perl. Figure 7-3. The Windows ODBC control panel We’ll take the latter route, if just to keep the snickering down among the Unix folks (see the upcoming note for a better reason). Here’s some code to create a user DSN to our SQL Server database: * There’s a third flavor, file, that writes the DSN configuration information out to a file so it can be shared among several computers, but it isn’t created by the Win32::ODBC method call we’re about to use. Using ODBC from Within DBI | 249 use Win32::ODBC; # we only use this to create DSNs; everything else is # done via DBI through DBD::ODBC # Creates a user DSN to a Microsoft SQL Server # Note: to create a system DSN, substitute ODBC_ADD_SYS_DSN # for ODBC_ADD_DSN - be sure to use a system DSN for # situations where your code will be run as another user # (e.g., in a web application) # if (Win32::ODBC::ConfigDSN( ODBC_ADD_DSN, 'SQL Server', ('DSN=PerlSysAdm', 'DESCRIPTION=DSN for PerlSysAdm', '', # server name 'ADDRESS=', # server IP addr 'DATABASE=sysadm', # our database 'NETWORK=DBMSSOCN', # TCP/IP Socket Lib ))){ print "DSN created\n"; } else { die "Unable to create DSN:" . Win32::ODBC::Error( ) . "\n"; } Should you create your DSNs manually or automatically? This is a su- perb question with no definitive answer. On the one hand, DSNs are compact descriptions of how to access potentially critical or sensitive data. This would lead one to be very cautious about who sets them up and tests them, and how (suggesting that a manual approach would be better). If a DSN is intentionally deleted from a machine, having it automatically created again may be undesirable. On the other hand, manual configuration is easy to get wrong and, in general, doesn’t scale for more than a few servers or applications. The best answer is probably to write and test a set of special setup scripts that can be run either manually or as part of your automated initial configuration process. This should help avoid the pitfalls. Once you have a DSN in place, you can reference it in the connect() call. For example, if we wanted to connect to the database via the DSN created by the previous code, the connect process would look like this: use DBI; $dbh = DBI->connect('DBI:ODBC:PerlSysAdm',$username,$pw); die "Unable to connect: $DBI::errstr\n" unless (defined $dbh); From that point on, you can put the rest of your DBI expertise to work. See the DBD::ODBC documentation for details on the additional features the driver provides and on the few ODBC-specific concerns worth mentioning. You now know how to work 250 | Chapter 7: SQL Database Administration with a database from Perl using both DBI and ODBC, so let’s put your knowledge to the test with some more extended examples from the database administration realm. Server Documentation A great deal of time and energy goes into the configuration of a SQL server and the objects that reside on it. Having a way to document this sort of information can come in handy in a number of situations. If a database gets corrupted and there’s no backup, you may be called upon to recreate all of its tables. You may have to migrate data from one server to another; knowing the source and destination configurations can be important. Even for your own database programming, being able to see a table map can be very helpful. To give you a taste of the nonportable nature of database administration, let me show you an example of the same simple task as written for three different SQL servers using both DBI and ODBC (via Win32::ODBC). Each of these programs does the exact same thing: prints out a listing of all of the databases on a server, their tables, and the basic structure of each table. These scripts could easily be expanded to show more informa- tion about each object. For instance, it might be useful to show which columns in a table had NULL or NOT NULL set. The output of all three programs looks roughly like this: ---sysadm--- hosts name [char(30)] ipaddr [char(15)] aliases [char(50)] owner [char(40)] dept [char(15)] bldg [char(10)] room [char(4)] manuf [char(10)] model [char(10)] ---hpotter--- customers cid [char(4)] cname [varchar(13)] city [varchar(20)] discnt [real(7)] agents aid [char(3)] aname [varchar(13)] city [varchar(20)] percent [int(10)] products pid [char(3)] pname [varchar(13)] city [varchar(20)] quantity [int(10)] price [real(7)] Server Documentation | 251 orders ordno [int(10)] month [char(3)] cid [char(4)] aid [char(3)] pid [char(3)] qty [int(10)] dollars [real(7)] ... It will be to your advantage to look at all three examples, even if you don’t use or plan to ever use the particular database server in question. We’ll be looking at several dif- ferent methods for querying the information in these sections, all of which you will want to know about. MySQL Server via DBI Here’s a DBI way of pulling the information just presented from a MySQL server. MySQL’s SHOW command makes this task pretty easy: use DBI; print 'Enter user for connect: '; chomp(my $user = ); print 'Enter passwd for $user: '; chomp(my $pw = ); my $start= 'mysql'; # connect initially to this database # connect to the start MySQL database my $dbh = DBI->connect("DBI:mysql:$start",$user,$pw, { RaiseError => 1, ShowErrorStatement => 1 }); # find the databases on the server my $sth=$dbh->prepare(q{SHOW DATABASES}); $sth->execute; my @dbs = ( ); while (my $aref = $sth->fetchrow_arrayref) { push(@dbs,$aref->[0]); } # find the tables in each database foreach my $db (@dbs) { print "---$db---\n"; $sth=$dbh->prepare(qq{SHOW TABLES FROM $db}); $sth->execute; my @tables=( ); while (my $aref = $sth->fetchrow_arrayref) { push(@tables,$aref->[0]); } 252 | Chapter 7: SQL Database Administration # find the column info for each table foreach my $table (@tables) { print "\t$table\n"; $sth=$dbh->prepare(qq{SHOW COLUMNS FROM $table FROM $db}); $sth->execute; while (my $aref = $sth->fetchrow_arrayref) { print "\t\t",$aref->[0],' [',$aref->[1],"]\n"; } } } $dbh->disconnect; A few quick comments about this code: • MySQL 5.x (a fairly new release as of this writing) has a special metadata database called INFORMATION_SCHEMA that contains tables that can be queried using ordinary SELECT statements to retrieve the same information as we’re getting from the SHOW commands. If you are using a 5.x version of MySQL, you’ll want to use that mech- anism instead to get the table and column information. Querying this information is slower than querying normal data in your database, however, so be wary of doing so if performance is important to you. • We connect to a starting database only to satisfy the DBI connect semantics; this context is not necessary thanks to the SHOW commands. • If you thought the SHOW TABLES and SHOW COLUMNS prepare and execute statements looked like excellent candidates for placeholders, you’re absolutely right. Unfortu- nately, this particular DBD driver/server combination doesn’t support placehold- ers in this context (at least, not when this book was being written). If you can use placeholders in situations like this, definitely do. They offer some protection against SQL injection attacks, thanks to their automatic quoting property (men- tioned earlier). • We prompt for a database user and password interactively because the alternatives (hard-coding them into the script or passing them on the command line, where they can be found by anyone running a process table dump) are even worse evils. This prompt will echo the password characters as typed. To be really careful, we should use something like Term::Readkey to turn off character echo. • And finally, a tip from Tim Bunce himself. Notice that we’re using RaiseError and ShowErrorStatement in the initial connect to the database. This asks DBI to handle the checking for and reporting of errors, which we would normally have to include with an or die "something" after each DBI call. It helps declutter your code considerably. Server Documentation | 253 Oracle Server via DBI Here’s an Oracle equivalent. This example sparks a whole bunch of commentary, so peruse the code and then we’ll talk about it: use DBI; use DBD::Oracle qw(:ora_session_modes); print 'Enter passwd for sys: '; chomp(my $pw = ); my $dbh = DBI->connect( 'DBI:Oracle:perlsysadm', 'sys', $pw, { RaiseError => 1, AutoCommit => 0, ora_session_mode => ORA_SYSDBA } ); my ( $catalog, $schema, $name, $type, $remarks ); # table_info returns this my $sth = $dbh->table_info( undef, undef, undef, 'TABLE' ); my (@tables); while ( ( $catalog, $schema, $name, $type, $remarks ) = $sth->fetchrow_array() ) { push( @tables, [ $schema, $name ] ); } for my $table ( sort @tables ) { $sth = $dbh->column_info( undef, $table->[0], $table->[1], undef ); # if you encounter an ORA-24345 error from the following fetchrow_arrayref(), # you can set $sth->{LongTruncOk} = 1 here as described in the DBD::Oracle doc print join( '.', @$table ), "\n"; while ( my $aref = $sth->fetchrow_arrayref ) { # [3] = COLUMN_NAME, [5] = TYPE_NAME, [6] = COLUMN_SIZE print "\t\t", $aref->[3], ' [', lc $aref->[5], "(", $aref->[6], ")]\n"; } } $sth->finish; $dbh->disconnect; Here is the promised commentary: • First, a general comment to set the scene: Oracle has a different notion of what the word “database” means than most other servers. The other servers discussed in this chapter each have a model where a user owns a database in which he is per- mitted to create a set of tables. This is why the previous example first found the list of the databases on the server, then stepped into each database, and finally listed the tables inside it. Oracle doesn’t have this additional level of hierarchy. Yes, there are databases in Oracle, but they are more like chunks of storage, often containing many tables owned by many users. The nearest equivalent to our previous usage of the word “database” in Oracle is the schema. A schema is the 254 | Chapter 7: SQL Database Administration collection of objects (tables, indices, etc.) owned by a user. Tables are usually ref- erenced as SCHEMA.TABLENAME. The preceding code connects to a single database instance called “perlsysadm” and shows its contents. • Ideally this code would connect to the database using an account that was specially privileged for this kind of work. To make the code more generic for example pur- poses, it attempts to connect to the database as the standard Oracle systems user sys. This user has permission to look at all tables in the database. To connect to this database as this user, one has to request special SYSDBA privileges, hence the funky parameter ora_session_mode => ORA_SYSDBA in the initial connect. If you have another user with that privilege granted, you will want to change the code to use that user instead of the all-powerful, all-knowing sys. • Besides that connection parameter, the code is surprisingly database server- independent. In contrast to the previous MySQL example, where the SHOW commands did the heavy lifting, here we use the standard DBI table_info() and column_info() calls to retrieve the information we need. Oracle has at least one similar command (DESCR tablename) that returns more information about a table, but sticking with the most generic method possible will improve code portability between separate database servers. • The example code is actually doing more work than it needs to do. To keep the code close in structure to the previous example, it first queries for the list of tables, then iterates over each table in a sorted order, and then queries the column info for that table. It turns out that column_info() is perfectly happy to retrieve information on all of the columns of all of the tables in the database in a single invocation if you just leave out the schema and table name (column_info(undef,undef,undef,undef)); furthermore, the DBI specification says the command should return the information to you in sorted order, so the sort() call also becomes unnecessary. Microsoft SQL Server via ODBC The DBI/DBD::ODBC-based code to show the same database/table/column information from Microsoft SQL Server is basically a combination of the two previous examples. First we use a database-specific query† to get the list of databases, and then we can use the DBI standard calls of table_info() and column_info() to retrieve the information we need. One small but significant set of changes is in the initial connect string: the connect() uses 'dbi:ODBC:{DSN_name_here}' (with some predefined DSN), a different privileged user is entered (see the following note), and the ora_session_mode option is removed. † If your user has the ability to access all databases on the server and you’d prefer not to grovel around in a system table, select catalog_name from information_schema.schemata is another query that can be used to retrieve this information on a relatively recent version of SQL Server. Server Documentation | 255 One of the things that changed between SQL Server 2000 and SQL Server 2005 is the visibility of the metadata (i.e., the list of all objects, etc.). With the 2000 version of the server, virtually any user on the sys- tem could enumerate these objects, but with 2005 this is considerably more locked down: a user must have the VIEW ANY DEFINITION permis- sion to retrieve the same info as before. These changes yield a program that looks like the following: use DBI; # this assumes a privileged user called mssqldba; your # username will probably be different print 'Enter passwd for mssqldba: '; chomp(my $pw = ); # assumes there is a predefined DSN with the name "PerlSys" my $dbh = DBI->connect( 'dbi:ODBC:PerlSys', 'mssqldba', $pw, { RaiseError => 1 }); # fetch the names of all of the databases my (@dbs) = map { $_->[0] } @{ $dbh->selectall_arrayref("select name from master.dbo.sysdatabases") }; my ( $catalog, $schema, $name, $type, $remarks ); # table_info returns this foreach my $db (@dbs) { my $sth = $dbh->table_info( $db, undef, undef, 'TABLE' ); my (@tables); while ( ( $catalog, $schema, $name, $type, $remarks ) = $sth->fetchrow_array() ) { push( @tables, [ $schema, $name ] ); } for my $table ( sort @tables ) { $sth = $dbh->column_info( $db, $table->[0], $table->[1], undef ); print join( '.', @$table ), "\n"; while ( my $aref = $sth->fetchrow_arrayref ) { # [3] = COLUMN_NAME, [5] = TYPE_NAME, [6] = COLUMN_SIZE print "\t\t", $aref->[3], ' [', lc $aref->[5], "(", $aref->[6], ")]\n"; } } } $dbh->disconnect; Just to give you one more way to approach the problem, here’s some code that uses the legacy Win32::ODBC module. This code looks different from our previous two ex- amples in a number of ways. First off, it uses the native-ODBC style of retrieving 256 | Chapter 7: SQL Database Administration information (see the Win32::ODBC docs). It may also look strange because we are relying on a few of the special stored procedures that ship with the server to retrieve the info we need (e.g., sp_columns()), using a really icky calling convention. This particular example is included on the off chance that you’ll find yourself in a situation that requires the use of Win32::ODBC and you’d like an example to help you begin the process. Here’s the code: use Win32::ODBC; print 'Enter user for connect: '; chomp(my $user = ); print 'Enter passwd for $user: '; chomp(my $pw = ); my $dsn='sysadm'; # name of the DSN we will be using # find the available DSNs, creating $dsn if it doesn't exist already die 'Unable to query available DSN's'.Win32::ODBC::Error()."\n" unless (my %dsnavail = Win32::ODBC::DataSources()); if (!defined $dsnavail{$dsn}) { die 'unable to create DSN:'.Win32::ODBC::Error()."\n" unless (Win32::ODBC::ConfigDSN(ODBC_ADD_DSN, "SQL Server", ("DSN=$dsn", "DESCRIPTION=DSN for PerlSysAdm", "", "DATABASE=master", "NETWORK=DBMSSOCN", # TCP/IP Socket Lib ))); } # connect to the master database via the DSN we just defined # # the DSN specifies DATABASE=master so we don't have to # pick it as a starting database explicitly my $dbh = new Win32::ODBC("DSN=$dsn;UID=$user;PWD=$pw;"); die "Unable to connect to DSN $dsn:".Win32::ODBC::Error()."\n" unless (defined $dbh); # find the databases on the server, Sql returns an error number if it fails if (defined $dbh->Sql(q{SELECT name from sysdatabases})){ die 'Unable to query databases:'.Win32::ODBC::Error()."\n"; } my @dbs = ( ); my @tables = ( ); my @cols = ( ); # ODBC requires a two-step process of fetching the data and then # accessing it with a special call (Data) while ($dbh->FetchRow()){ push(@dbs, $dbh->Data("name")); } $dbh->DropCursor(); # this is like DBI's $sth->finish() Server Documentation | 257 # find the user tables in each database foreach my $db (@dbs) { if (defined $dbh->Sql("use $db")){ die "Unable to change to database $db:" . Win32::ODBC::Error() . "\n"; } print "---$db---\n"; @tables=(); if (defined $dbh->Sql(q{SELECT name from sysobjects WHERE type="U"})){ die "Unable to query tables in $db:" . Win32::ODBC::Error() . "\n"; } while ($dbh->FetchRow()) { push(@tables,$dbh->Data("name")); } $dbh->DropCursor(); # find the column info for each table foreach $table (@tables) { print "\t$table\n"; if (defined $dbh->Sql(" {call sp_columns (\'$table\')} ")){ die "Unable to query columns in $table:" . Win32::ODBC::Error() . "\n"; } while ($dbh->FetchRow()) { @cols=$dbh->Data("COLUMN_NAME","TYPE_NAME","PRECISION"); print "\t\t",$cols[0]," [",$cols[1],"(",$cols[2],")]\n"; } $dbh->DropCursor(); } } $dbh->Close(); die "Unable to delete DSN:".Win32::ODBC::Error()."\n" unless (Win32::ODBC::ConfigDSN(ODBC_REMOVE_DSN, "SQL Server","DSN=$dsn")); Database Logins As mentioned earlier, database administrators have to deal with some of the same issues system administrators contend with, like maintaining logins and accounts. For in- stance, at my day job we teach database programming classes. Each student who takes a class gets a login on our Oracle server and her very own (albeit small) database quota on that server to play with. Here’s a simplified version of the code we use to create these databases and logins: use DBI; my $userquota = 10000; # K of user space given to each user my $usertmpquota = 2000; # K of temp tablespace given to each user 258 | Chapter 7: SQL Database Administration my $admin = 'system'; print "Enter passwd for $admin: "; chomp(my $pw = ); my $user=$ARGV[0]; # generate a *bogus* password based on username reversed # and padded to at least 6 chars with dashes # note: this is a very bad algorithm; better to use something # like Crypt::GeneratePassword my $genpass = reverse($user) . '-' x (6-length($user)); my $dbh = DBI->connect("dbi:Oracle:instance",$admin,$pw,{PrintError => 0}); die "Unable to connect: $DBI::errstr\n" unless (defined $dbh); # prepare the test to see if user name exists my $sth = $dbh->prepare(q{SELECT USERNAME FROM dba_users WHERE USERNAME = ?}) or die 'Unable to prepare user test SQL: '.$dbh->errstr."\n"; my $res = $sth->execute(uc $user); $sth->fetchrow_array; die "user $user exists, quitting" if ($sth->rows > 0); if (!defined $dbh->do ( qq { CREATE USER ${LOGIN} PROFILE DEFAULT IDENTIFIED BY ${PASSWORD} DEFAULT TABLESPACE USERS TEMPORARY TABLESPACE TEMP QUOTA $usertmpquota K ON TEMP QUOTA $userquota K ON USERS ACCOUNT UNLOCK })){ die 'Unable to create database:'.$dbh->errstr."\n"; } # grant the necessary permissions $dbh->do("GRANT CONNECT TO ${LOGIN}") or die "Unable to grant connect privs to ${LOGIN}:".$dbh->errstr."\n"; # perhaps a better approach would be to explicity grant the parts of # RESOURCE the users need rather than grant them everything and # removing things like UNLIMITED TABLESPACE later $dbh->do("GRANT RESOURCE TO ${LOGIN}") or die "Unable to grant resource privs to ${LOGIN}:".$dbh->errstr."\n"; # set the correct roles $dbh->do("ALTER USER ${LOGIN} DEFAULT ROLE ALL") or die "Unable to use set correct roles for ${LOGIN}:".$dbh->errstr."\n"; # make sure the quotas are enforced $dbh->do("REVOKE UNLIMITED TABLESPACE FROM ${LOGIN}") or die "Unable to revoke unlimited tablespace from ${LOGIN}:".$dbh->errstr."\n"; $dbh->disconnect; We could use a similar script to delete these accounts and their databases when the class has concluded: Database Logins | 259 use DBI; $admin = 'system'; print "Enter passwd for $admin: "; chomp(my $pw = ); my $user=$ARGV[0]; my $dbh = DBI->connect("dbi:Oracle:instance",$admin,$pw,{PrintError => 0}); die "Unable to connect: $DBI::errstr\n" if (!defined $dbh); die "Unable to drop user ${user}:".$dbh->errstr."\n" if (!defined $dbh->do("DROP USER ${user} CASCADE")); $dbh->disconnect; You might find it useful to code up a variety of login-related functions. Here are a few ideas: Password checker Connect to the server and get a listing of databases and logins. Attempt to connect using weak passwords (login names, blank passwords, default passwords). User mapping Generate a listing of which logins can access which databases. Password control Write a pseudo-password expiration system. Monitoring Space Usage on a Database Server For our final example, we’ll take a look at a way to monitor the storage space of a SQL server. This sort of routine monitoring is similar in nature to the network service mon- itoring we’ll see in Chapter 13. To get technical for a moment, database servers are places to hold stuff. Running out of space to hold stuff is known as either “a bad thing” or “a very bad thing.” As a result, programs that help us monitor the amount of space allocated and used on a server are very useful indeed. Let’s look at a DBI program designed to evaluate the space situation on an Oracle server. Here’s a snippet of output from a program that illustrates graphically each user’s space usage in relationship to her predefined quota. Each section shows a bar chart of the percentage of used versus allocated space in the USERS and TEMP tablespaces. In the following chart, u stands for user space and t stands for temp space. For each bar, the percentage of space used and the total available space are indicated: |uuuuuuu |15.23%/5MB hpotter--------| | | |0.90%/5MB 260 | Chapter 7: SQL Database Administration |uuuuuuu |15.23%/5MB dumbledore-----| | | |1.52%/5MB |uuuuuuuu |16.48%/5MB hgranger-------| | | |1.52%/5MB |uuuuuuu |15.23%/5MB rweasley-------| | |t |3.40%/5MB |uuuuuuuuuuuuuuuuuuuuuuuuuuu |54.39%/2MB hagrid---------| | |- no temp quota | Here’s how we generated this output: use DBI; use DBD::Oracle qw(:ora_session_modes); use POSIX; # for ceil rounding function use strict; print 'Enter passwd for sys: '; chomp( my $pw = ); # connect to the server my $dbh = DBI->connect( 'DBI:Oracle:', 'sys', $pw, { RaiseError => 1, ShowErrorStatement => 1, AutoCommit => 0, ora_session_mode => ORA_SYSDBA } ); # get the quota information my $sth = $dbh->prepare( q{SELECT USERNAME,TABLESPACE_NAME,BYTES,MAX_BYTES FROM SYS.DBA_TS_QUOTAS WHERE TABLESPACE_NAME = 'USERS' or TABLESPACE_NAME = 'TEMP'} ); $sth->execute; # bind the results of the query to these variables, later to be stored in %qdata my ( $user, $tablespace, $bytes_used, $bytes_quota, %qdata ); $sth->bind_columns( \$user, \$tablespace, \$bytes_used, \$bytes_quota ); while ( defined $sth->fetch ) { $qdata{$user}->{$tablespace} = [ $bytes_used, $bytes_quota ]; } $dbh->disconnect; # show this information graphically foreach my $user ( sort keys %qdata ) { graph( $user, Monitoring Space Usage on a Database Server | 261 $qdata{$user}->{'USERS'}[0], # bytes used $qdata{$user}->{'TEMP'}[0], $qdata{$user}->{'USERS'}[1], # quota size $qdata{$user}->{'TEMP'}[1] ); } # print out nice chart given username, user and temp sizes, # and usage info sub graph { my ( $user, $user_used, $temp_used, $user_quota, $temp_quota ) = @_; # line for user space usage if ( $user_quota > 0 ) { print ' ' x 15 . '|' . 'd' x POSIX::ceil( 49 * ( $user_used / $user_quota ) ) . ' ' x ( 49 - POSIX::ceil( 49 * ( $user_used / $user_quota ) ) ) . '|'; # percentage used and total M for data space printf( "%.2f", ( $user_used / $user_quota * 100 ) ); print "%/" . ( $user_quota / 1024 / 1000 ) . "MB\n"; } # some users do not have user quotas else { print ' ' x 15 . '|- no user quota' . ( ' ' x 34 ) . "|\n"; } print $user . '-' x ( 14 - length($user) ) . '-|' . ( ' ' x 49 ) . "|\n"; # line for temp space usage if ( $temp_quota > 0 ) { print ' ' x 15 . '|' . 't' x POSIX::ceil( 49 * ( $temp_used / $temp_quota ) ) . ' ' x ( 49 - POSIX::ceil( 49 * ( $temp_used / $temp_quota ) ) ) . '|'; # percentage used and total M for temp space printf( "%.2f", ( $temp_used / $temp_quota * 100 ) ); print "%/" . ( $temp_quota / 1024 / 1000 ) . "MB\n"; } # some users do not have temp quotas else { print ' ' x 15 . '|- no temp quota' . ( ' ' x 34 ) . "|\n"; } print "\n"; } Writing this code wasn’t particularly hard because Oracle provides a lovely view called SYS.DBA_TS_QUOTAS that contains the tablespace quota information we need in an easy- to-query fashion. This ease is highly database server-specific; other servers can make 262 | Chapter 7: SQL Database Administration you work harder for this information (e.g., with Sybase you need to add up segments when computing database sizes). This small program just scratches the surface of the sort of server monitoring we can do. It would be easy to take the results we get from SYS.DBA_TS_QUOTAS and graph them over time to get a better notion of how our server is being used. There are lots of other things we can (and probably should) monitor, including CPU usage and various data- base performance metrics (cache hit rates, etc.). There are entire books on “Tuning Database X” from which you can get a notion of the key parameters to watch from a Perl script. Let creeping featurism be your muse. Module Information for This Chapter Module CPAN ID Version DBI TIMB 1.50 DBD::mysql RUDY 2.9008 DBD::Oracle PYTHIAN 1.17 DBD::ODBC JURL 1.13 Win32::ODBC (from JDB 970208 References for More Information There are a number of good SQL tutorials on the Web, including http://www.sqlzoo .net and Search for “SQL tutorial” and you’ll find a bunch more. Reading them can give you a good jumpstart into using the language. is a swell guide to the vagaries of the more popular database engines. Though its focus is on writing portable code, as you saw in this chapter, often one needs to know database-specific commands to help administer a server. DBI is the official DBI home page. It’s quite dated in places (according to Tim Bunce), but this should be your first stop. Programming the Perl DBI (, by Alligator Descartes and Tim Bunce (O’Reilly), is a great DBI resource. has some useful DBI recipes for common tasks. References for More Information | 263 Microsoft SQL Server In addition to the Microsoft SQL Server information available at the Microsoft web- site (, there’s a tremendous amount of information at Microsoft’s training materials for Microsoft SQL Server administration from MS Press are also quite good. ODBC contains Microsoft’s ODBC information. You’ll need to dig a little because it has been subsumed (as of this writing) into their larger Data Access and Storage Center rubric. You may also want to search for ODBC on the site, looking carefully at the library material on ODBC in the MDAC SDK. is the official Win32::ODBC home page. (For legacy pur- poses. You should use DBD::ODBC whenever possible.) Win32 Perl Programming: The Standard Extensions, by Dave Roth (Macmillan), the author of Win32::ODBC, is still a good reference for Windows Perl module-based programming. Oracle The Oracle universe is very large. There are many, many Oracle-related books and websites. One site I find really useful is; this is a fabulous resource for getting answers to both basic and more sophisticated Oracle questions. The Documentation and Tutorials paths on are a great source for in-depth information about the different releases of Oracle databases. 264 | Chapter 7: SQL Database Administration CHAPTER 8 Email Unlike the other chapters in this book, this chapter does not discuss how to administer a particular service, technology, or knowledge domain. Instead, we’re going to look at how to use email from Perl as a tool for system administration. Email is a great notification mechanism: often we want a program to tell us when something goes wrong, provide the results of an automatic process (like a late-night cron or scheduler service job), or let us know when something we care about changes. In this chapter we’ll explore how to send mail from Perl for these purposes and then look at some of the pitfalls associated with the practice of sending ourselves mail. We’ll also look at how Perl can be used to fetch and post-process mail we receive to make it more useful to us. Perl can be useful for dealing with spam and managing user questions. This chapter will assume that you already have a solid and reliable mail infrastructure. We’re also going to assume that your mail system, or one that you have access to, uses protocols that follow the IETF specifications for sending and receiving mail. The ex- amples in this chapter will use protocols like SMTP (Simple Mail Transfer Protocol, RFC 2821) and expect messages to be RFC 2822-compliant. We’ll go over these terms in due course. Sending Mail Let’s talk about the mechanics of sending email before we tackle the more sophisticated issues. The traditional (Unix) Perl mail-sending code often looks something like this example from the Perl Frequently Asked Questions list ( # assumes we have sendmail installed in /usr/sbin # sendmail install directory varies based on OS/distribution open my $SENDMAIL, '|-', '|/usr/sbin/sendmail -oi -t -odq' or die "Can't fork for sendmail: $!\n"; print $SENDMAIL <<'EOF'; From: User Originating Mail To: Final Destination 265 Subject: A relevant subject line Body of the message goes here after the blank line in as many lines as you like. EOF close(SENDMAIL) or warn "sendmail didn't close nicely"; A common error (which has its roots in Perl4 when made by Perl old- timers) is to write something like this, including the @ sign directly in a double-quoted string: $address = ""; # array interpolates This line needs to be changed to one of the following to work properly: $address = "fred\"; $address = ''; $address = q{ }; $address = join('@', 'fred', ''); Code that calls sendmail (like the preceding example) works fine in many circumstan- ces, but it doesn’t work on any operating system that lacks a mail transport agent called “sendmail” (e.g., Windows-based operating systems). If you’re using such an OS you have a few choices, which are described in the next few sections. Getting sendmail (or a Similar Mail Transport Agent) Various sendmail and sendmail-like ports (most of which are commercial) are available for Windows. If you want a free version of something that can potentially pretend to be sendmail, try the Cygwin exim port ( If you’d like something more lightweight and are willing to make small modifications to your Perl code to support different command-line arguments, programs like blat (http://www.blat .net) will do the trick. The advantage of this approach is that it offloads much of the mail-sending complexity from your script. A good mail transport agent (MTA) handles the process of retrying a destination mail server if it’s unreachable, selecting the right destination server (finding and choosing between Mail eXchanger DNS records), rewriting the headers if neces- sary, dealing with bounces, and so on. If you can avoid having to take care of all of that in Perl, that’s often a good thing. Using the OS-Specific IPC Framework to Drive a Mail Client On Mac OS X or Windows, you can drive a mail client using the native interprocess communication (IPC) framework. Mac OS X ships with the Postfix MTA installed, but only minimally configured. If you don’t want to bother setting it up, you can ask Perl to use AppleScript to drive the built- in mail client (often called 266 | Chapter 8: Email use MacPerl; my $to = ''; my $subject = 'Hi there'; my $body = 'message body'; MacPerl::DoAppleScript(<new('Outlook.Application'); my $ol = Win32::OLE::Const->Load($outl); my $message = $outl->CreateItem(olMailItem); $message->Recipients->Add(''); $message->{Subject} = 'Perl to Outlook Test'; $message->{Body} = "Hi there!\n\nLove,\nPerl\n"; $message->Send; To drive Outlook, we request an Outlook Application object and use it to create a mail message for sending. To make our lives easier during that process, we use Win32::OLE::Const to suck in the OLE constants associated with Outlook. This gives us olMailItem, and from there things are straightforward. The preceding code is pretty simple, but we still had to know more than should have been necessary about how to talk to Outlook. Figuring out how to expand upon this idea (e.g., how to attach a file or move messages around in Outlook folders) would * The code shown here controls Outlook using the Application object, found in reasonably modern versions of Outlook (2000 and beyond). The first edition of this book predated those Outlook versions, so it described how to do this task using the lower-level MAPI calls. This is a much easier tack to take. Sending Mail | 267 require more probing of the MSDN website for clues. To make this easier, the developer known as “Barbie” created Mail::Outlook, which allows us to write code like this instead: use Mail::Outlook; my $outl = new Mail::Outlook(); my $message = $outl->create(); $message->To(''); $message->Subject('Perl to Outlook Test'); $message->Body("Hi there!\n\nLove,\nPerl\n"); $message->Attach(@files); $message->send() or die "failed to send message"; Ultimately, programs that rely on AppleScript or Application objects are equally as non-portable as those that call a program called “sendmail.” They offload some of the work, but they’re relatively inefficient. Such approaches should probably be your methods of last resort. Speaking the Mail Protocols Directly Our final choice is to write code that speaks to the mail server in its native language. Most of this language is documented in RFC 2821. Here’s a basic SMTP conversation. The data we send is in bold: % telnet 25 - connect to the SMTP port on Trying ... Connected to Escape character is '^]'. 220 ESMTP Sendmail 8.9.1a/8.9.1; Sun, 13 Apr 2008 15:32:16 −0400 (EDT) HELO - identify the machine we are connecting from (can also use EHLO) 250 Hello [], pleased to meet you MAIL FROM: - specify the sender 250 ... Sender ok RCPT TO: - specify the recipient 250 ... Recipient ok DATA - begin to send message, note we send several key header lines 354 Enter mail, end with "." on a line by itself From: David N. Blank-Edelman To: Subject: SMTP is a fine protocol Just wanted to drop myself a note to remind myself how much I love SMTP. Peace, dNb . - finish sending the message 268 | Chapter 8: Email 250 PAA26624 Message accepted for delivery QUIT - end the session 221 closing connection Connection closed by foreign host. It’s not difficult to script a network conversation like this; we could use the IO::Socket module, or even something like Net::Telnet, which we’ll see in the next chapter. However, there are good mail modules out there that make our job easier, such as Jenda Krynicky’s Mail::Sender, Milivoj Ivkovic’s Mail::Sendmail, the Mail::Mailer module in Graham Barr’s MailTools package, and Email::Send (main- tained by Ricardo Signes for the Perl Email Project). All four of these packages are operating system-independent and will work almost anywhere a modern Perl distri- bution is available. We’ll look at Email::Send for two reasons: because it offers a single interface to two of the mail-sending methods we’ve discussed so far, and because it offers us a good entry into the phalanx of modules connected to the Perl Email Project. A late-breaking tip: after this book went to production, Ricardo Signes, the developer who maintains Email::Send (and most of the Perl Email Project modules), announced he was going to deprecate Email::Send in favor of a new module called Email::Sender. Email::Sender isn’t fully written yet (e.g., there is an Email::Sender::Simple module on the way that will make using that module even easier) and hasn’t had the same level of field testing Email::Send has seen. Signes says he’ll still maintain Email::Send for a year or so, so I would recommend sticking with it until it is clear Email::Sender is ripe. Sending vanilla mail messages with Email::Send Email::Send will happily send a mail message stored in plain text in a scalar variable along the lines of: my $message = <<'EOM'; From: To: Subject: advice I am the mother-of-all-things and all things should wear a sweater. Love, Mom EOM You can also get some free error checking by using an object from another of the Perl Email Project’s modules: Email::Simple. Email::Simple and its plug-in module Email::Simple::Creator make it easy to programmatically construct email messages using an object-oriented approach. This is less prone to errors than writing email mes- sages like the one in our last code snippet directly into your program. Let’s see these modules in action; then we can bring Email::Send back into the picture to actually send the message we create. Sending Mail | 269 Email::Simple::Creator takes the hassle out of creating a message by providing a straightforward create() method. It takes two arguments, header (containing a list of headers and their contents) and body (a scalar with the body of the message), like so: use Email::Simple; use Email::Simple::Creator; use Email::Send; my $message = Email::Simple->create( header => [ From => '', To => '', Subject => 'Test Message from Email::Simple::Creator', ], body => "Hi there!\n\tLove,\n\tdNb", ); Easy, no? Now let’s look at how this message gets sent. If we wanted to directly send the message via SMTP, we’d write: my $sender = Email::Send->new({mailer => 'SMTP'}); $sender->mailer_args([Host => '']); $sender->send($message) or die "Unable to send message!\n"; To send it using sendmail, or whatever is pretending to be sendmail on the system (e.g., Exim or Postfix), we’d change this to: my $sender = Email::Send->new({mailer => 'Sendmail'}); $Email::Send::Sendmail::SENDMAIL = '/usr/sbin/sendmail'; $sender->send($message) or die "Unable to send message: $!\n"; You might note that the code is setting the package variable $Email::Send::Sendmail::SENDMAIL. This is required because Email::Send::Sendmail, at least as of this writing, makes no attempt to find the sendmail binary any place other than in the current path (a strange choice because the binary is very rarely in users’ paths). We have to help it out by pointing it to the correct location. There are a number of other possible values for mailer, corresponding to various Email::Send:: helper modules. One of my favorites is 'test', which uses Email::Send::Test. The Email::Send::Test module lets your application think it is sending mail, but actually “traps” all outgoing mail and stores it in an array for your inspection. This is a great way to debug mail-sending code before accidentally irritating thousands of recipients with a mistake you didn’t catch until after the mail was sent. Sending mail messages with attachments using Email::Send Once people find sending mail via Perl is so easy, they often want to do more compli- cated things in this vein. Despite email being a poor medium for file transfer, it is pretty common to find yourself needing to send mail with arbitrary attachments. That process can get complex quickly, though, because you’re now on the path down the rabbit hole known as the Multipurpose Internet Mail Extensions (MIME) standards. Described in RFCs 2045, 2046, 2047, 2077, 4288, and 4289 (yes, it takes at least six standards 270 | Chapter 8: Email documents to document this beast), MIME is a standard for including various kinds of content within an email message. We don’t have the space in this chapter to do anything but skim the surface of MIME, so I’ll just note that a MIME message is composed of parts, each of which is labeled with a content type and other metadata, such as how it is represented or encoded. There is an Email::MIME module in the Perl Email Project (maintained by Ricardo Signes) for working with MIME in this context. Luckily for us, Email::MIME has a helper plug-in module called Email::MIME::Creator (also maintained by Signes) that makes creating attachments much less painful than usual. Let’s look at some example code first, and then we’ll talk about how it works: use Email::Simple; use Email::MIME::Creator; use File::Slurp qw(slurp); use Email::Send; my @mimeparts = ( Email::MIME->create( attributes => { content_type => 'text/plain', charset => 'US-ASCII', }, body => "Hi there!\n\tLove,\n\tdNb\n", ), Email::MIME->create( attributes => { filename => 'picture.jpg', content_type => 'image/jpeg', encoding => 'base64', name => 'picture.jpg', }, body => scalar slurp('picture.jpg'), ), ); my $message = Email::MIME->create( header => [ From => '', To => '', Subject => 'Test Message from Email::MIME::Creator', ], parts => [@mimeparts], ); my $sender = Email::Send->new({mailer => 'Sendmail'}); $Email::Send::Sendmail::SENDMAIL = '/usr/sbin/sendmail'; $sender->send($message) or die "Unable to send message!\n"; The first step is to create the two parts that will make up the message: the plain-text part (the body of the message a user will see) and the attachment part. This is again pretty straightforward, with the only tricky part being the actual inclusion of the file being attached. Email::MIME->create() needs a scalar value containing the entire con- tents of the file being attached. One of the easiest and most efficient ways to suck an Sending Mail | 271 entire file into a variable is to use Dave Rolsky’s File::Slurp module. Being explicit about what type of value we expect to get back from the slurp() call ensures that we get the scalar value we need. After defining the two MIME parts for the message and loading them with data, we now need to construct the message consisting of those two parts. The second call to Email::MIME->create() creates the message consisting of our required headers and the parts objects we just created. With this message in hand, sending the actual message is just like the vanilla send shown earlier. Sending HTML mail messages using Email::Send I’m loath to show you how to do this because I personally dislike HTML mail, but if it gets around that you know how to send mail programmatically, someday someone is going to come to you and demand you send HTML mail messages for him. If that person is your boss, you can say “HTML mail is icky” as many times as you want, but it probably won’t get you out of the assignment. To help you keep your job, I’ll show you one example of how this is done—just don’t tell anyone you learned this from me. HTML mail messages are just another example of using MIME to transport non-plain- text data in a mail message. Given that, you could construct a mail message using the same Email::MIME::Creator technique demonstrated in the last section. This would be relatively straightforward for a very basic, text-only page if you already knew the MIME metadata for each part of the HTML message. However, it starts to get a little tricky if you want to have things like images rendered in the HTML page, particularly if you’d prefer to send those things within the message itself. There are a couple of reasons to embed images: URL-sourced images make for a slow message display, and, more im- portantly, many email clients block URL-based images for security reasons (so spam- mers cannot use them as web bugs to confirm that the messages were received). Luckily, there’s a similar message-creation module called Email::MIME::CreateHTML, created and maintained by programmers at the BBC, that can handle all the heavy lifting for us. Here’s a very simple example of sending HTML mail with a plain-text alternative: use Email::MIME::CreateHTML; use Email::Send; my $annoyinghtml=< Hi there!
  dNb HTML my $message = Email::MIME->create_html( header => [ From => '', 272 | Chapter 8: Email To => '', Subject => 'Test Message from Email::MIME::CreateHTML', ], body => $annoyinghtml, text_body => "Hi there!\n\tLove,\n\tdNb", ); my $sender = Email::Send->new( { mailer => 'Sendmail' } ); $Email::Send::Sendmail::SENDMAIL = '/usr/sbin/sendmail'; $sender->send($message) or die "Unable to send message!\n"; Our part in the process is super-simple—we’re just passing in scalar values that contain the HTML and plain-text versions of the message. The HTML we’re passing in doesn’t have any external references to images and such, but if it did, the method Email::MIME->create_html would have parsed them out of the message and attached the files for us accordingly. You’ll also notice that the actual sending of the message is handled in exactly the same way as in our previous examples. This is one of the benefits of using Email::Send. One last comment about this code before we move on: Email::MIME::CreateHTML re- moves the need for a lot of complex fiddling around, but there’s a price to pay for all the power under the hood. In order to work its magic, Email::MIME::CreateHTML de- pends on a relatively large list of other modules (each with its own dependencies). Installing these dependencies isn’t a problem, thanks to the and CPANPLUS modules, but if you’re looking for something lightweight you’ll want to look for another way to create your mail messages. Common Mistakes in Sending Email Now that you know how to send email, you can begin using email as a notification method. Once you start to write code that performs this function, you’ll quickly find that the issue of how to send mail is not nearly as interesting as the questions of when and what to send. This section explores those questions by taking a contrary approach. If we look at when and how not to send mail, we’ll get a deeper insight into these issues.† Let’s begin by exploring some of the most common mistakes made by system administration pro- grams that send mail. Overzealous Message Sending By far, the most common mistake is sending too much mail. It’s a great idea to have scripts send mail. If there’s a service disruption, normal email and email sent to a pager † This assumes that you’ve decided email is still the best communication method for your purposes. When making that decision, you should take into account that it can be subject to large delays, isn’t generally secure, etc. Common Mistakes in Sending Email | 273 are good ways to bring this problem to the attention of a human. But under most circumstances, it is a very bad idea to have your program send mail about the problem every five minutes or so. Overzealous mail generators tend to be quickly added to the mail filters of the very humans who should be reading the messages, with the end result being that important mail is routinely and automatically ignored. Controlling the frequency of mail The easiest way to avoid what I call “mail beaconing” is to build into the programs safeguards to gate the delay between messages. If your script runs constantly, it’s easy to stash the time when the last mail message was sent in a variable like this: my $last_sent = time; If your program is started up every N minutes or hours via Unix’s cron or the Windows Task Scheduler service mechanisms, this information can be written to a one-line file and read again the next time the program is run. Be sure in this case to pay attention to some of the security precautions outlined in Chapter 1. Depending on the situation, you may be able to get fancy about your delay times. One suggestion is to perform an exponential backoff where you have a routine that gives the OK to send mail once every minute (20), then every two minutes (21), every four minutes (22), every eight minutes (23), and so on until you reach some upper limit like “once a day.” Alternatively, sometimes it is more appropriate to have your program act like a two- year-old, complaining more often as time goes by. In that case you can do an expo- nential ramp-up where the routine initially gives the OK to send messages starting with the maximum delay value (say, “once a day”) and becomes exponentially more per- missive until it reaches a minimum value, like “every five minutes.” Controlling the amount of mail Another subclass of the “overzealous message sending” syndrome is the “everybody on the network for themselves” problem. If all the machines on your network decide to send you a piece of mail at the same time, you may miss something important in the subsequent message blizzard. A better approach is to have them all report to a central repository of some sort.‡ The information can then be collated and mailed out later in a single message. ‡ One good tool (under Unix) for the central reporting of status is syslog (or one of its descendants, such as syslog-ng). To be able to use this tool effectively in this context, however, you need to be able to control its configuration on the receiving end. That’s not always an option, for any number of technical and administrative reasons, so this chapter presents another method. For more info on dealing with syslog logs, see Chapter 10. 274 | Chapter 8: Email Let’s consider a moderately contrived example. For this scenario, assume each machine in your network drops a one-line file into a shared directory.* Named for the machine, that file will contain that machine’s summary of the results of last night’s scientific computation. The single line in the file might be of this form: hostname success-or-failure number-of-computations-completed A program that collates the information and mails the results might look like this: use Email::Simple; use Email::Simple::Creator; use Email::Send; use Text::Wrap; use File::Spec; # the list of machines reporting in my $repolist = '/project/machinelist'; # the directory where they write files my $repodir = '/project/reportddir'; # send mail "from" this address my $reportfromaddr = ''; # send mail to this address my $reporttoaddr = ''; my $statfile; # the name of the file where status reports are recorded my $report; # the report line found in each statfile my %success; # the succesful hosts my %fail; # the hosts that failed my %missing; # the list of hosts missing in action (no reports) # Now we read the list of machines reporting in into a hash. # Later, we'll depopulate this hash as each machine reports in, # leaving behind only the machines that are missing in action. open my $LIST, '<', $repolist or die "Unable to open list $repolist:$!\n"; while (<$LIST>) { chomp; $missing{$_} = 1; } close $LIST; # total number of machines that should be reporting my $machines = scalar keys %missing; # Read all of the files in the central report directory. # Note: this directory should be cleaned out automatically # by another script. opendir my $REPO, $repodir or die "Unable to open dir $repodir:$!\n"; * Another good rendezvous spot for status information like this would be in a database. A third possibility would be to have all of the mail sent to another mailbox. You could then have a separate Perl program retrieve the messages via POP3 and post-process them. Common Mistakes in Sending Email | 275 while ( defined( $statfile = readdir($REPO) ) ) { next unless -f File::Spec->catfile( $repodir, $statfile ); # open each status file and read in the one-line status report open my $STAT, File::Spec->catfile( $repodir, $statfile ) or die "Unable to open $statfile:$!\n"; chomp( $report = <$STAT> ); my ( $hostname, $result, $details ) = split( ' ', $report, 3 ); warn "$statfile said it was generated by $hostname!\n" if ( $hostname ne $statfile ); # hostname is no longer considered missing delete $missing{$hostname}; # populate these hashes based on success or failure reported if ( $result eq 'success' ) { $success{$hostname} = $details; } else { $fail{$hostname} = $details; } close $STAT; # we could remove the $statfile here to clean up for the # next night's run, but only if that works in your setup } closedir $REPO; # construct a useful subject for our mail message my $subject; if ( scalar keys %success == $machines ) { $subject = "[report] Success: $machines"; } elsif ( scalar keys %fail == $machines or scalar keys %missing >= $machines ) { $subject = "[report] Fail: $machines"; } else { $subject = '[report] Partial: ' . keys(%success) . ' ACK, ' . keys(%fail) . ' NACK' . ( (%missing) ? ', ' . keys(%missing) . ' MIA' : '' ); } # create the body of the message my $body = "Run report from $0 on " . scalar localtime(time) . "\n"; if ( keys %success ) { $body .= "\n==Succeeded==\n"; foreach my $hostname ( sort keys %success ) { 276 | Chapter 8: Email $body .= "$hostname: $success{$hostname}\n"; } } if ( keys %fail ) { $body .= "\n==Failed==\n"; foreach my $hostname ( sort keys %fail ) { $body .= "$hostname: $fail{$hostname}\n"; } } if ( keys %missing ) { $body .= "\n==Missing==\n"; $body .= wrap( '', '', join( ' ', sort keys %missing ) ), "\n"; } my $message = Email::Simple->create( header => [ From => $reportfromaddr, To => $reporttoaddr, Subject => $subject, ], body => $body, ); my $sender = Email::Send->new( { mailer => 'Sendmail' } ); $Email::Send::Sendmail::SENDMAIL = '/usr/sbin/sendmail'; $sender->send($message) or die "Unable to send message!\n"; The code first reads in a list of the machine names that will be participating in this scheme. Later, it will use a hash based on this list to check whether there are any machines that have not placed a file in the central reporting directory. We’ll open each file in this directory and extract the status information. Once we’ve collated the results, we construct a mail message and send it out. Here’s an example of the resulting message: Date: Mon, 14 Apr 2008 13:06:09 −0400 (EDT) Message-Id: <> Subject: [report] Partial: 3 ACK, 4 NACK, 1 MIA To: From: Run report from reportscript on Mon Apr 14 13:06:08 2008 ==Succeeded== barney: computed 23123 oogatrons betty: computed 6745634 oogatrons fred: computed 56344 oogatrons ==Failed== bambam: computed 0 oogatrons dino: computed 0 oogatrons pebbles: computed 0 oogatrons wilma: computed 0 oogatrons Common Mistakes in Sending Email | 277 ==Missing== mrslate Another way to collate results like this is to create a custom logging daemon and have each machine report in over a network socket. Let’s look at the code for the server first. This example reuses code from the previous example. We’ll talk about the important new bits right after you see the listing: use IO::Socket; use Text::Wrap; # used to make the output prettier # the list of machines reporting in my $repolist = '/project/machinelist'; # the port number clients should connect to my $serverport = '9967'; my %success; # the succesful hosts my %fail; # the hosts that failed my %missing; # the list of hosts missing in action (no reports) # load the machine list using a hash slice (end result is a hash # of the form %missing = { key1 => undef, key2 => undef, ...}) @missing{ loadmachines() } = (); my $machines = keys %missing; # set up our side of the socket my $reserver = IO::Socket::INET->new( LocalPort => $serverport, Proto => "tcp", Type => SOCK_STREAM, Listen => 5, Reuse => 1 ) or die "Unable to build our socket half: $!\n"; # start listening on it for connects while ( my ( $connectsock, $connectaddr ) = $reserver->accept() ) { # the name of the client that has connected to us my $connectname = gethostbyaddr( ( sockaddr_in($connectaddr) )[1], AF_INET ); chomp( my $report = $connectsock->getline ); my ( $hostname, $result, $details ) = split( ' ', $report, 3 ); # if we've been told to dump our info, print out a ready-to-go mail # message and reinitialize all of our hashes/counters if ( $hostname eq 'DUMPNOW' ) { printmail($connectsock); close $connectsock; undef %success; undef %fail; undef %missing; 278 | Chapter 8: Email @missing{ loadmachines() } = (); # reload the machine list my $machines = keys %missing; next; } warn "$connectname said it was generated by $hostname!\n" if ( $hostname ne $connectname ); delete $missing{$hostname}; if ( $result eq 'success' ) { $success{$hostname} = $details; } else { $fail{$hostname} = $details; } close $connectsock; } close $reserver; # Prints a ready-to-go mail message. The first line is the subject, # and subsequent lines are all the body of the message. sub printmail { my $socket = shift; my $subject; if ( keys %success == $machines ) { $subject = "[report] Success: $machines"; } elsif ( keys %fail == $machines or keys %missing >= $machines ) { $subject = "[report] Fail: $machines"; } else { $subject = '[report] Partial: ' . keys(%success) . ' ACK, ' . keys(%fail) . " NACK" . ( (%missing) ? ', ' . keys(%missing) . ' MIA' : '' ); } print $socket "$subject\n"; print $socket "Run report from $0 on " . scalar localtime(time) . "\n"; if ( keys %success ) { print $socket "\n==Succeeded==\n"; foreach my $hostname ( sort keys %success ) { print $socket "$hostname: $success{$hostname}\n"; } } if ( keys %fail ) { print $socket "\n==Failed==\n"; foreach my $hostname ( sort keys %fail ) { Common Mistakes in Sending Email | 279 print $socket "$hostname: $fail{$hostname}\n"; } } if ( keys %missing ) { print $socket "\n==Missing==\n"; print $socket wrap( '', '', join( ' ', sort keys %missing ) ), "\n"; } } # loads the list of machines from the given file sub loadmachines { my @missing; open my $LIST, '<', $repolist or die "Unable to open list $repolist:$!\n"; while (<$LIST>) { chomp; push( @missing, $_ ); } close $LIST; return @missing; } Besides moving some of the code sections to their own subroutines, the key change is the addition of the networking code. The IO::Socket module makes the process of opening and using sockets pretty painless. Sockets are usually described using a tele- phone metaphor. We start by setting up our side of the socket (IO::Socket->new()), essentially turning on our phone, and then wait for a call from a network client (IO::Socket->accept()). Our program will pause (or “block”) until a connection re- quest comes in. As soon as it arrives, we note the name of the connecting client. We then read a line of input from the socket. This line of input is expected to look just like those we read from the individual files in our previous example. The one difference is the magic hostname DUMPNOW. If we see this hostname, we print the subject and body of a ready-to-mail message to the connecting client and reset all of our counters and hash tables. The client is then responsible for actually sending the mail it receives from the server. Let’s look at our sample client and what it can do with this message: use IO::Socket; # the port number clients should connect to my $serverport = '9967'; # the name of the server my $servername = 'reportserver'; # name-to-IP address mapping my $serveraddr = inet_ntoa( scalar gethostbyname($servername) ); my $reportfromaddr = ''; my $reporttoaddr = ''; my $reserver = IO::Socket::INET->new( PeerAddr => $serveraddr, 280 | Chapter 8: Email PeerPort => $serverport, Proto => 'tcp', Type => SOCK_STREAM ) or die "Unable to build our socket half: $!\n"; if ( $ARGV[0] ne '-m' ) { print $reserver $ARGV[0]; } else { # These 'use' statements will load their respective modules when the # script starts even if we don't get to this code block. We could use # require/import instead (like we did in Chapter 3), but the goal here # is to just make it clear that these modules come into play when we # use the -m switch. use Email::Simple; use Email::Simple::Creator; use Email::Send; print $reserver "DUMPNOW\n"; chomp( my $subject = <$reserver> ); my $body = join( '', <$reserver> ); my $message = Email::Simple->create( header => [ From => $reportfromaddr, To => $reporttoaddr, Subject => $subject, ], body => $body, ); my $sender = Email::Send->new( { mailer => 'Sendmail' } ); $Email::Send::Sendmail::SENDMAIL = '/usr/sbin/sendmail'; $sender->send($message) or die "Unable to send message!\n"; } close $reserver; First, we open up a socket to the server. In most cases, we pass it our status information (received on the command line as $ARGV[0], i.e., "dino fail computed 0 oogatrons") and drop the connection. If we were really going to set up a logging client/ server like this, we would probably encapsulate this client code in a subroutine and call it from within a much larger program after its processing had been completed. If this script is passed an -m flag, it instead sends “DUMPNOW” to the server and reads the subject line and body returned by the server. Then this output is fed to Email::Send and sent out via mail using the same code we saw earlier. To limit the example code size and keep the discussion on track, the server and client code presented here is as bare bones as possible. There’s no error or input checking, access control or authentication (anyone on the Net who can get to our server can feed Common Mistakes in Sending Email | 281 and receive data from it), persistent storage (what if the machine goes down?), or any of a number of other routine precautions in place. On top of this, we can only handle a single request at a time. If a client should stall in the middle of a transaction, we’re sunk. For more sophisticated server examples, I recommend you check out the client/ server treatments in Lincoln Stein’s Network Programming With Perl (Addison-Wesley) and Tom Christiansen and Nathan Torkington’s Perl Cookbook ( alog/9780596003135/) (O’Reilly). Jochen Wiedmann’s Net::Daemon module will also help you write more sophisticated daemon programs. Now that we’ve dealt with regulating the volume of mail sent, let’s move on to other common mistakes made when writing system administration programs that send mail. Subject Line Waste A Subject line is a terrible thing to waste. When sending mail automatically, it is pos- sible to generate a useful Subject line on the fly for each message. This means there is very little excuse to leave someone with a mailbox that looks like this: Super-User File history database merge report Super-User File history database merge report Super-User File history database merge report Super-User File history database merge report Super-User File history database merge report Super-User File history database merge report Super-User File history database merge report when it could look like this: Super-User Backup OK, 1 tape, 1.400 GB written. Super-User Backup OK, 1 tape, 1.768 GB written. Super-User Backup OK, 1 tape, 2.294 GB written. Super-User Backup OK, 1 tape, 2.817 GB written. Super-User Backup OK, 1 tape, 3.438 GB written. Super-User Backup OK, 3 tapes, 75.40 GB written. or even like this: Super-User Backup of Hostname OK, 1 tape, 1.400 GB written. Super-User Backup of Hostname:/usr OK, 1 tape, 1.768 GB written. Your Subject line should provide a concise and explicit summary of the situation. It should be very clear from that line whether the program generating the message is reporting success, failure, or something in between. A little more programming effort will pay off handsomely in reduced time reading mail. Insufficient Information in the Message Body As with the Subject line, in the message body a little specificity goes a long way. If your script is going to complain about problems or error conditions via email, it should strive to provide certain pieces of information. They boil down to the canonical questions of journalism: 282 | Chapter 8: Email Who? Which script is complaining? Include the contents of $0 (if you haven’t set it ex- plicitly) to show the full path to the current script. Mention the version of your script if it has one. Where? Give some indication of the place in your script where trouble occurred. The Perl function caller() returns all sorts of useful information for this purpose: # Note: what caller() returns can be specific to a # particular Perl version, so be sure to see the perlfunc docs ($package, $filename, $line, $subroutine, $hasargs, $wantarray, $evaltext, $is_require) = caller($frames); $frames is the number of stack frames (if you’ve called subroutines from within subroutines) desired. Most often you’ll want $frames set to 1. Here’s a sample list returned by the caller() function when it’s called in the middle of the server code from our last full code example: ('main','repserver',32,'main::printmail',1,undef) This shows that the script was in the main package while running from the filename repserver at line 32 in the script. At that point it was executing code in the main::printmail subroutine (which has arguments and has not been called in a list context). If you want to be even kinder to the people who will read your mail, you can pair caller() up with the Carp module shipped with Perl to output diagnostic infor- mation that is (at best guess) most relevant to the issue at hand. For our purposes, we’ll want to use the longmess() routine, explicitly imported because the module does not export it by default: use Carp qw(longmess); longmess() provides the contents of the warning message that would be produced if one called a warn()-like substitute called cluck(). In addition to printing out this warning, it also produces a whole stack backtrace that can be helpful for deter- mining exactly where in a long program things failed. When? Describe the program state at the time of the error. For instance, what was the last line of input read? Why? If you can, answer the reader’s unspoken question: “Why are you bothering me with a mail message?” The answer may be as simple as “the accounting data has not been fully collated,” “DNS service is not available now,” or “the machine room is on fire.” This provides context to the reader (and perhaps some motivation to investigate). Common Mistakes in Sending Email | 283 What? Finally, don’t forget to mention what went wrong in the first place. Here’s some simple Perl code that covers all of these bases: use Text::Wrap; use Carp qw(longmess); sub problemreport { # $shortcontext should be a one-line description of the problem # $usercontext should be a detailed description of the problem # $nextstep should be the best suggestion for how to remedy the problem my ( $shortcontext, $usercontext, $nextstep ) = @_; my ( $filename, $line, $subroutine ) = ( caller(1) )[ 1, 2, 3 ]; my $report = ''; $report .= "Problem with $filename: $shortcontext\n"; $report .= "*** Problem report for $filename ***\n\n"; $report .= fill( '', ' ', "- Problem: $usercontext" ) . "\n\n"; $report .= "- Location: line $line of file $filename in " . "$subroutine\n\n"; $report .= longmess('Stack trace ') . "\n"; $report .= '- Occurred: ' . scalar localtime(time) . "\n\n"; $report .= "- Next step: $nextstep\n"; return $report; } sub fireperson { my $report = problemreport( 'the computer is on fire', < 'user', PASSWORD => 'secretsquirrel', HOST => '', USESSL => 'true', ); die 'Connection failed: ' . $pop3->Message() . "\n" if $pop3->Count() == −1; print 'Number of messages in this mailbox: ' . $pop3->Count() . "\n\n"; print "The first message looks like this: \n" . $pop3->Retrieve(1) . "\n"; $pop3->Close(); There’s not much to this code as written because there doesn’t have to be much. If we wanted to extend it, we could call Delete(message #) to mark a message for deletion or Uidl() if we wanted to get back UIDLs for all messages or a particular one. Both the Head() and HeadAndBody() methods will return either a scalar or an array based on their 286 | Chapter 8: Email calling context, so it’s easy to get a mail header or message in the form desired by packages like Mail::SpamAssassin, discussed later in this chapter. Talking IMAP4rev1 to Fetch Mail IMAP4rev1, called IMAP4 from this point on, is a significantly more powerful (read: complex) protocol documented in RFC 3501. Its basic model is different from that of POP3. With POP3 it is assumed that the POP3 client polls the POP3 server and down- loads mail periodically, while with IMAP4 a client connects to a server for the duration of the mail reading session.‡ With POP3 the client is expected to handle all of the sophisticated work, like deciding what messages to download, while with IMAP4 the discussion between the server and the client is much richer, so the protocol has to be considerably smarter. Smarter how? Here are some of the characteristics of IMAP4: • It can deal with a whole hierarchical structure of mail folders and the contents of each folder. According to RFC 3051, “IMAP4rev1 includes operations for creating, deleting, and renaming mailboxes, checking for new messages, permanently re- moving messages, setting and clearing flags, RFC 2822 and RFC 2045 parsing, searching, and selective fetching of message attributes, texts, and portions thereof.” • It has a much more sophisticated understanding of the structure of an individual mail message. POP3 lets us grab a mail message’s headers or the headers plus the first N lines of the message body. IMAP4 lets us ask for just “the text part of the message” in messages that have lots of attachments and doodads. It does this by building MIME into the official specification. • It lets a client send a whole bunch of commands to the server at once and receive the results back in whatever fashion the server chooses to send them. This is dif- ferent from the standard process of having a client send a command and then wait for the server to respond before it can send a second command. Each IMAP4 com- mand and response is prefaced with a unique “tag” that allows both the client and the server to keep track of what has been asked and answered. • It has a “disconnected mode” that allows clients to connect to a server, cache as much information as they need, and then disconnect. The user can then potentially operate on that cache as if the connection was still in place. When the connection returns, the client can play the changes made to the local mail store back to the server and the server will catch the client up on what happened while the client was out of touch. This mode allows you to sit on a plane without network access, deleting and filing mail, later to have those changes be reflected on the server once you get back on the network. With all of this power comes the price of complexity. You won’t want to do much IMAP4 programming without RFC 3501 close at hand. Even that only gets you so far, ‡ Warning: there’s a little bit of hand waving going on in this statement, because IMAP4 has something known as “disconnected mode” that doesn’t fit this description. We’ll talk about that in just a moment. Fetching Mail | 287 because different server authors have decided to implement certain edge cases differ- ently. You may have to play around a bit to get the results you want when it comes to more advanced IMAP4 programming. For the example code we’re about to see, I’ll be using my current preferred IMAP module, Mail::IMAPClient (originally by David J. Kernen, rewritten and now main- tained by Mark Overmeer). This is the same module that forms the basis of the superb imapsync program (, a great tool for migrating data from one IMAP4 server to another. In addition to imapsync’s vote of confidence, I like this module because it is mostly complete when it comes to features while still offering the ability to send raw IMAP4 commands should it become neces- sary. The other module that I would consider looking at is Mail::IMAPTalk by Rob Mueller, the primary developer behind Even though it hasn’t been upda- ted in a few years, the module’s author assures me that the current release still works well and is in active use. For our first IMAP4 example, here’s some code that connects (securely) to a user’s mailbox, finds everything that was previously labeled as spam by SpamAssassin (it adds the header X-Spam-Flag: YES), and moves those messages to a SPAM folder. We’ll start with connecting to the IMAP server: use IO::Socket::SSL; use Mail::IMAPClient; my $s = IO::Socket::SSL->new(PeerAddr =>'', PeerPort => '993', Proto => 'tcp'); die $@ unless defined $s; my $m = Mail::IMAPClient->new(User => 'user', Socket=>$s, Password => 'topsecret'); Mail::IMAPClient does not have SSL built-in in the same way that Mail::POP3Client does, so we’re forced to construct an SSL-protected socket by hand and pass it to Mail::IMAPClient. Without specifying this connection, all communication, including the password, would be sent in clear text. Chained to an Old Version If you rely on imapsync, you may find yourself in the unfortunate position of having to keep an old version of Mail::IMAPClient around because, as of this writing, imapsync doesn’t yet completely work with the 3.x rewrite of Mail::IMAPClient. If this is still the case when you read this text, you are going to find that the code in this section won’t work as written. There are two non-obvious changes of the hair-pulling kind that you’ll need to make if you are going to use your own secure socket. First, Mail::IMAPClient doesn’t properly handle the greeting that comes back from the server. You’ll need to “eat” the greeting yourself right after the socket is created using code like this: 288 | Chapter 8: Email my $greeting = <$s>; my ( $id, $answer ) = split /\s+/, $greeting; die "connect problem: $greeting" if $answer ne 'OK'; Second, Mail::IMAPClient doesn’t know that it is connected and doesn’t automatically initiate a login sequence, so the following is necessary right after the call to new(): $m->State( Mail::IMAPClient::Connected() ); $m->login() or die 'login(): ' . $m->LastError(); Both of these issues get fixed in the 3.x versions of Mail::IMAPClient, so hopefully the module will play nicely with imapsync in the future. STOP THE PRESSES: Literally as this book was being produced, a set of patches that purport to fix a number of the major incompatibilities with the latest Mail::IMAPClient version came across the imapsync mailing list. Looks like hope is in sight—perhaps by the time you have the chance to read this sidebar it will be a non- issue. The moral of the story: sometimes an application you use can lock you into a specific version of a Perl module.* Once connected, the first thing one typically does is tell the server which folder to operate on. In this case, we’ll select the user’s INBOX: $m->select('INBOX'); Now let’s get to work and look for all of the messages in the INBOX with the X-Spam-Flag header set to YES: my @spammsgs = $m->search(qw(HEADER X-Spam-Flag YES)); die $@ if $@; @spammsgs now contains the list of messages we want to move, so we move each one in turn, close the folder, and log out of the server: foreach my $msg (@spammsgs){ $m->move('SPAM', $msg) or die 'move failed: '.$m->LastError; } $m->close(); # expunges currently selected folder $m->logout; There’s a hidden detail in the last two lines of code that I feel compelled to mention. You might remember from the POP3 discussion that we talked about messages being “marked as deleted.” The same tombstoning process takes place here as well. Deletes are always a two-step process in IMAP4: we first flag messages as \Deleted, then ex- punge messages marked with that flag. When we requested that a message be moved, the server copied the message to the new folder and marked the message in the source folder as being deleted. Ordinarily you would need to expunge() the source folder to actually remove the message, but RFC 3501 says that performing a CLOSE operation on a folder implicitly expunges that folder. * The local::lib module by Matt S. Trout, now maintained by Christopher Nehren, can help a considerable amount with module version lock-in like this. Fetching Mail | 289 Let’s look at one more IMAP4 example that will offer a good segue into our next section on processing mail. Earlier in this section we mentioned IMAP4’s ability to work with a message’s component MIME parts. Here’s some code that demonstrates this at work. To save a tree or two of book paper, I’ll leave out the initial module load, object creation, secure connection to the server, and mailbox selection code, because it’s exactly the same as what we’ve already seen: my @digests = $m->search(qw(SUBJECT digest)); foreach my $msg (@digests) { my $struct = $m->get_bodystructure($msg); next unless defined $struct; # Messages in a mailbox get assigned both a sequence number and # a unique identifier. By default Mail::IMAPClient works with UIDs. print "Message with UID $msg (Content-type: ",$struct->bodytype,'/', $struct->bodysubtype, ") has this structure:\n\t", join("\n\t",$struct->parts) ,"\n\n"; } $m->logout; This code searches for all of the messages in the currently selected folder that have “digest” in the Subject line. Then the loop examines the structure of each message and prints the MIME parts of each. Here’s some sample output for two messages in my INBOX: Message with UID 2457 (Content-type: TEXT/PLAIN) has this structure: HEAD 1 Message with UID 29691 (Content-type: MULTIPART/MIXED) has this structure: 1 2 3 3.1 3.1.HEAD 3.1.1 3.1.2 3.2 3.2.HEAD 3.2.1 3.2.2 3.3 3.3.HEAD 3.3.1 3.3.2 4 Once you know the MIME part you’re looking for, you can call bodypart_string() with the message UID and the MIME part number to retrieve it. For example, the following: 290 | Chapter 8: Email print $m->bodypart_string(29691,'4'); prints out the footer of the message with UID 29691: Perl-Win32-Database mailing list To unsubscribe: Mail::IMAPClient uses the Parse::RecDescent module to take apart MIME messages. Its parser works most of the time, but I have found that some messages cause it to malfunction. If you find yourself doing a good deal of MIME-related mail processing, you may want to call on one of the dedicated MIME-processing modules, such as Email::MIME, or even use the Mail::IMAPTalk module mentioned earlier. We’ll see an example of using Email::MIME in the next section. This discussion of extracting parts of messages leads us right into our next subject. Processing Mail It is useful to be able to fetch mail, but that’s just the beginning. In this section we’ll explore what can be done with that mail once it has been transferred. Let’s start with the basics and look at the tools available for the dissection of both a single mail message and an entire mailbox. For the first topic, we will again turn to modules provided by the Perl Email Project. In the first edition of this book the examples in this section used the Mail::Internet, Mail::Header, and Mail::Folder modules. I’ve switched to the modules from the Perl Email Project for consistency’s sake, but the first edition’s modules are all still viable (especially now that the first two are being updated regularly under the stewardship of Mark Overmeer). Mark is also the author of Mail::Box, a copiously featured package for mail handling. If the modules from the Perl Email Project don’t provide what you need, you should definitely take a look at Mail::Box. Dissecting a Single Message The Email::Simple module offers a convenient way to slice and dice the headers of an RFC 2822-compliant mail message. RFC 2822 dictates the format of a mail message, including the names of the acceptable header lines and their formats. To use Email::Simple, feed it a scalar variable that contains a mail message: use Email::Simple; my $message = <<'EOM'; Processing Mail | 291 From Mon Aug 6 05:43:22 2007 Received: from localhost (localhost []) by (Postfix) with ESMTP id 6A39577490A for ; Mon, 6 Aug 2007 05:43:22 −0400 (EDT) Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id OIIgygSczEdt for ; Mon, 6 Aug 2007 05:43:22 −0400 (EDT) Received: from ( []) by (Postfix) with ESMTP id 2828A774909 for ; Mon, 6 Aug 2007 05:43:22 −0400 (EDT) Received: from ([]) by with esmtps (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.50) id 1IHzA6-0002GV-7g for; Mon, 06 Aug 2007 05:46:06 −0400 Date: Mon, 6 Aug 2007 05:46:06 −0400 (EDT) From: My User To: "David N. Blank-Edelman" Subject: About mail server Message-ID: Hi David, Boy, that's a spiffy mail server you have there! Best, Your User EOM my $esimple = Email::Simple->new($message); There are two methods on the $esimple object that you would typically call at this point: header('field') and body(). The body() method returns the body of the message, as you’d expect, but the header() method is a little more interesting. It returns either all of the headers with that field (if called in a list context), or the first one (if called in a scalar context): my @received = $esimple->header('Received'); my $first_received = $esimple->header('Received'); One difference between Email::Simple and some other mail-parsing modules is that Email::Simple returns only the data for the header, and not the entire line from the mail including that header. For example: print scalar $esimple->header('Date') prints: Mon, 6 Aug 2007 05:46:06 −0400 (EDT) not: Date: Mon, 6 Aug 2007 05:46:06 −0400 (EDT) 292 | Chapter 8: Email If for some reason you need to know which header fields are present in a message, the header_names() method will return that information. The other kind of mail message dissection one often does beyond just header/body processing is the extraction of certain contents from the body of the message. In the case of a MIME-encoded message, for example, we may want to extract an attachment from the contents of the mail and save it as a different file. Here’s an example of us- ing Email::MIME to that end: use Email::MIME; use File::Slurp qw(slurp write_file); my $message = slurp('mime.txt'); my $parsed = Email::MIME->new($message); foreach my $part ($parsed->parts) { if ($part->content_type =~ /^application\/pdf;/i){ write_file ($part->filename, $part->body); } } This code uses slurp() to bring in the contents of a message stored in mime.txt and then parses it (this is done automatically by the new() method). We then iterate over each MIME part and decide whether it is a PDF file based on the MIME content type. If it is, we write that part of the message out to a file using the filename provided in the MIME header (or one autogenerated by Email::MIME if the sender didn’t specify a name). It is important to note that this code is less than ideal in at least two ways. First, it looks only at the top-level MIME parts in the message when looking for the attach- ment. That approach won’t work if the attachment is embedded in another part (e.g., when someone forwards the entire contents of a message, attachments included, as an attachment itself).† The second and much more serious problem with this code is that it trusts the filename as specified in the header. Real code would be much more paranoid (see the cautionary note in the following sidebar). Don’t Cut Corners When Parsing Mail Here’s a quick warning that should accompany all of the mail-parsing material that we’ve just covered. Parsing mail is tricky business for at least two reasons: the com- plexity of the data and “the bad guys.” Your code needs to be robust and complete. Here’s a good example that demonstrates the first peril: many people write code that uses simple regular expressions to validate email addresses. Don’t be one of them. The RFC 2822 syntax is sufficiently complex that I can almost guarantee that some day your code will break if you cut corners like this. It is far better to use a tool such as † You might think that looking for embedded attachments using recursive parts() calls or the subparts() method would be a good exercise to leave to the reader, but I’ll make it easier than that: there is a separate module called Email::MIME::Attachment::Stripper, also maintained by Ricardo Signes, that does this work for you. Processing Mail | 293 the Email::Valid module, currently maintained by Ricardo Signes, or the Mail::Message::Field::* modules in the Mail::Box package. Look first to modules like Regexp::Common (e.g., the net module) for parsing IP addresses and so on. Packages like these can help you manage the complexity of the data. You can also use Perl’s -T switch (taint mode) to look carefully at how data is being passed around in your script. As for “the bad guys,” they mostly appear when you’re parsing mail received by spam- mers and other nogoodniks. As Bill Cole, someone who has worked in the anti-spam community for over a decade, said to me in an email he gave me permission to publish: Spammers toss all sorts of pathological garbage at filters, both as stupid accident and as conscious attack. Spam filtering is no place to assume that someone else has validated your input as meeting any sort of norm. You really need to protect yourself in any code that looks at email, because even if you are being shielded in principle by some other tool (e.g., MIMEDefang, sendmail, whatever) you have to assume that someday spam will start coming in that gets malicious content through that armor to your code. Before we end this section, I’d like to give two examples of “deep” parsing of a message. For this kind of parsing, we’re going to take apart the message body itself. To keep things simple for the example, I’m going to assume that the message body is plain text and is not encoded in any way, as in the previous MIME examples. You should be able to use the modules mentioned earlier to whittle down a message to this point if you need to deal with more complex message formats. For the first example, we’ll explore how to do keyword scanning efficiently. Let’s say it’s important to quarantine all messages that contain any words on a special “dirty words” list. The key to efficient scanning of a message text (especially when given a whole list of items) is to pass over the same text as few times as possible—ideally, only once. Sure, we could do this: my @dirty_words = qw ( sod ground soil earth filth mud shmutz ); foreach my $word (@dirty_words){ return 'dirty' if ($body =~ /$word/is); } But that would drag the regular expression engine over the whole message again and again, and it would force the regexp engine to reparse/recompile the regexp each time. There are a number of ways to get around this problem, but one of the most efficient is to combine the strings using regular expression alternation. We could jam all of the words together, with pipe characters (|) separating them: my $wordalt = join('|',@dirty_words); my $regex = qr/$wordalt/is; return 'dirty' if ($message =~ $regex); 294 | Chapter 8: Email That’s a bit better, but we can still go one step further. If you stare at your list of dirty words for a while, you’ll probably notice that they have some things in common (lexicographically). Several of them will start with the same letters, which means we could start optimizing the regexp using shorthand like so(d|il). Perl 5.10+’s regular expression engine will do some of this optimization for us. If you really have a need for speed and/or if you’re working with an earlier version of the Perl interpreter, you can use modules like Aaron Crane’s Text::Match::FastAlternatives or David Landgren’s Regexp::Assemble. The former is made exactly for the case we’re describing here and is even faster than the 5.10+ optimized regular expression engine. Regexp::Assemble isn’t as fast, but it has a number of additional features that may make it a good choice for more complicated tasks. Here’s a quick example of Text::Match::FastAlternatives in action: use Text::Match::FastAlternatives; use Email::Simple; use File::Slurp qw(slurp); my $message = slurp('message.txt'); my $esimple = Email::Simple->new($message); my @dirty_words = qw ( sod ground soil earth filth mud shmutz ); # this gets much more impressive when the size of the list is huge my $matcher = Text::Match::FastAlternatives->new( @dirty_words ); print 'dirty' if $matcher->match( $esimple->body() ); There are two important restrictions that should be mentioned when talking about Text::Match::FastAlternatives. First, it only works with data consisting of printable ASCII characters (a problem if you are scanning email messages that use a different character set); second, and more importantly, it only performs case-sensitive matches. If we wanted to catch both “filth” and “Filth,” we’d probably have to rewrite the last two lines of the code to look like this instead: my $matcher = Text::Match::FastAlternatives->new( map { lc } @dirty_words ); print 'dirty' if $matcher->match( lc $esimple->body() ); Text::Match::FastAlternatives usually gives enough of a speed boost to your program that you probably won’t mind having to downcase everything first. So that’s looking for keywords. What if you wanted to find more sophisticated content? Sometimes it is useful to extract URIs from a message. As mentioned in the sidebar “Don’t Cut Corners When Parsing Mail” on page 293, Regexp::Common is one module that can make this task easier/safer: use File::Slurp qw(slurp); use Email::Simple; use Regexp::Common qw /URI/; Processing Mail | 295 my $esimple = Email::Simple->new( scalar slurp $ARGV[0] ); my $body = $esimple->body; while ( $body =~ /$RE{URI}{HTTP}{-keep}/g ) { print "$1\n"; } This code uses a regular expression from Regexp::Common to find all URIs in a message. The use of the -keep flag means we capture those URIs into $1. We’ll discuss something more interesting to do with the URIs a little later in this chapter. Dissecting a Whole Mailbox Taking this subject to the next level, where we slice and dice entire mailboxes, is straightforward. If our mail is stored in the classical Unix mbox, maildir, or mh format, we can use Email::Folder from the Perl Email Project (also currently maintained by Ricardo Signes). Even many common non-Unix mail agents, like Eudora, store their mail in classical Unix mbox format, so this module can be useful on multiple platforms. The drill is very similar to the examples we’ve seen before: use Email::Folder; my $folder = Email::Folder->('FilenameOrDirectory'); The new() constructor takes the filename (for mbox format storage) or the directory where mail is stored (for the maildir and mh formats) to parse. It returns a folder object instance, which represents a mail folder containing a number of messages.‡ We can retrieve messages from that folder as Email::Simple objects. We can either retrieve all of the messages: my @messages = $folder->messages; or retrieve one message at a time: foreach my $message ($folder->next_message){ ... # do something with that message object } $message will contain an Email::Simple object instance. With this object instance, you can use all of the methods we just discussed. For example, if you need just the Subject header of the same message: $subject = $message->header('Subject'); These methods can be chained, so the following code will get the Subject line for the next message in the mailbox: $subject = $folder->next_message->header('Subject'); ‡ This folder representation does not depend on how the messages are actually stored, be they messages kept in a single file (for mbox format) or one file per message (for maildir/mh format). 296 | Chapter 8: Email Email::Folder by itself is very basic.* If you need to do anything beyond simply dicing a folder, you should consider using the Mail::Box package mentioned earlier. Into the Fray Again Of all the material that was updated in this second edition of the book, the sections on spam turned out to be some of the most challenging. The war between spammers and the anti-spam community escalated to such an extent in the interim that the approach and tools presented in the first edition seem ridiculously naïve to me now, even though the advice I gave was good at the time. Once upon a time, you could help fight the good fight by first identifying the origin of a message and then complaining to the ISP that sent the mail. However, in the current age, which is populated with legions of zombie machines herded into massive botnets doing spammers’ bidding, reporting one sending host just isn’t going to make an appreciable dent. The advice to locate and report the host is so outdated, it would be like someone telling you to be sure to keep the pointy part of your pike upright as you entered a modern battlefield. Truth be told, when I first started to rewrite this section I wasn’t sure what Perl tools would be helpful, not just at the time I was writing it, but for the shelf life of this edition. It’s very hard to predict how the battle between good and evil will rage in the coming years, especially given how far things have progressed since the publication of the last edition. To figure out what to present to you, dear reader, I wound up turning to a group of people who I knew had some of the greatest expertise and experience in the anti-spam realm. They were kind enough to offer a slew of suggestions, many of which I’ve incorporated into the section “Dealing with Spam” and the rest of the chapter. Hopefully this will provide you with some best-practices advice that will serve you well for some time to come. Dealing with Spam So far in this chapter we’ve looked at general tools for slicing and dicing mail messages and briefly touched on some of the applications that could benefit from a pile of finely chopped message parts. One (unfortunately increasingly large) application domain for these techniques is that surrounding the handling of unsolicited commercial email, or “spam” for short. SpamAssassin As the sidebar “Into the Fray Again” mentions, dealing with spam from any angle has become a tricky business. Thus, it behooves us to bring to bear on the problem as much firepower as possible. Ideally, we’d like to use software assembled from the collective * There are additional Email::Folder:: helper modules available that allow you to specify a folder on a POP3, IMAP(s), or Exchange server and parse it as if it were local. Still basic, sure, but cool nonetheless. Processing Mail | 297 intelligence of lots of people working on the issue. The easiest way for us to do that is to do something unusual for this book and focus on how to program using just one Perl-based anti-spam tool: Apache SpamAssassin ( The SpamAssassin Perl API is provided by the Mail::SpamAssassin set of modules. The API has stayed stable for the last five years and is likely to continue to be useful for quite a few more years to come. Another reason to look at this module is that it provides quite a few handy utility functions for mail processing. The package provides easy ways to decode HTML and MIME structures, extract readable URLs, perform blacklist look- ups, and much more. This in itself makes it worth exploring. Like most Perl modules that are indistinguishable from magic, Mail::SpamAssassin makes the hardest thing the easiest to do. Want to figure out whether a message is spam (according to SpamAssassin)? It’s this easy, though there is more going on here than meets the eye: use Mail::SpamAssassin; use File::Slurp qw(slurp); my $spama = Mail::SpamAssassin->new(); my $message = $spama->parse(scalar slurp 'message.txt'); my $status = $spama->check($message); print (($status->is_spam()) ? 'spammy!' : "hammy!" . "\n"); $status->finish(); $message->finish(); This code requires three steps to answer the spam/not spam question: create the Mail::SpamAssassin object, use it to parse a mail message into an object it can use, and then call the check() method from that object. We could actually do this in two steps if we eliminated the parse step and called check_message_text() instead of check(). The check_message_text() method will work on a plain mail message, but if we eliminate the parsing step we don’t get a cool message object that can be used later if we need to query or manipulate parts of the original message. Let’s look at some of the things we can do with that message object. The first thing we can do is extract RFC 2822-related parts of the message, such as the headers. For example, to get a list of all of the Received headers, we can write: use Mail::SpamAssassin; use File::Slurp qw(slurp); my $spama = Mail::SpamAssassin->new(); my $message = $spama->parse(scalar slurp 'message.txt'); my @received = $message->header('Received'); # or, to retrieve only the last one (as opposed to the first one, # which most packages give you when called in a scalar context): # my $received = $message->header('Received'); $message->finish(); 298 | Chapter 8: Email Mail::SpamAssassin can also help us extract MIME parts found in the message. For example, to print all the HTML parts of a message, we could write code like this: use Mail::SpamAssassin; use File::Slurp qw(slurp); my $sa = Mail::SpamAssassin->new(); my $message = $sa->parse( scalar slurp 'mime.txt' ); my @html_parts = $message->find_parts( qr(text/html), 1 ); foreach my $part (@html_parts) { print @{ $part->raw() }; } $message->finish(); Let’s talk for a second about the two highlighted lines, because we’ve already seen the others. The first line calls find_parts(). This method does a complete MIME parse of the message, walks the potentially complex structure of the message, and returns pointers (Mail::SpamAssassin::Message::Node objects, to be precise) to the parts that match the regular expression provided. The second parameter (1) tells the method to return only the individual parts found. Without that parameter, find_parts() will also return the containing parts should the desired type be found nested in some other part. To see all of the parts of the message after a full parse has been completed (see the sidebar “Parse As Little As Possible”), we can call content_summary() on the object. This will return output that looks like this: DB<1> x $message->content_summary(); 0 'multipart/mixed' 1 'multipart/alternative,text/plain,text/html' 2 'application/pdf' Parse As Little As Possible One gotcha that you may encounter is that Mail::SpamAssassin doesn’t do a complete MIME parse of a message until it has no choice but to take that extra step. This can be confusing if you expect the parse() method to return an object that reflects a complete parse. Not so! It only initially parses the headers of a message. This will throw you the first time you try to use content_summary() and don’t see all the parts you’d expect in a message. The easiest way to deal with this is to first do a find_parts(), perhaps something like this: $message->find_parts(qr/./,1); This unexpected behavior shows that this is a highly optimized spam-fighting tool, not just yet another general-purpose mail-parsing module. Ideally, you want messages to pass through Mail::SpamAssassin as quickly as possible so it can process high volumes of mail. To this end, the less work/parsing the module has to do initially per message, the better. Processing Mail | 299 Once we’ve found the parts of a message that we need, we can print them in their raw form as we do in our sample code, decode them (e.g., from base64 encoding), render them as text (e.g., for HTML parts), and so on. Right about now you may be experiencing some feelings of déjà vu, because it seems like we’ve already seen how to handle most of these parsing tasks using other modules from the Perl Email Project. You’re not imagining things; there is definitely some overlap between these two sets of mail-handling modules. So how do you choose which mail-handling package to use? My incli- nation is to avoid mixing and matching modules from different pack- ages. If you’re doing only generic mail parsing and don’t need anything that Mail::SpamAssassin provides, stick with the Email:: modules. If you do need something anti-spam-related, you want to be paranoid about the input messages, and/or you want to use some of the convenience functions (like those we’re about to see), go with Mail::SpamAssassin. We’ve just looked at the functionality provided to us if we parse() a message to receive a Mail::SpamAssassin::Message object, but it turns out that there are a number of ben- efits to check()ing a message as well. These benefits go beyond just the ability to decide whether or not a message is spam. Running check() on a message object or check_message_text() on a plain-text message returns a Mail::SpamAssassin::PerMsgStatus object. We called $status->is_spam() earlier in this chapter to check whether the object was classified as spam, but there are other methods that we could call as well. Here are some of the ones I find most handy: get_content_preview() Returns a small, text-only excerpt from the first few lines of a message. get_decoded_body_text_array(), get_decoded_stripped_body_text_array() Returns a message body with all encoded data (base64, etc.) decoded and all of the non-text parts (e.g., attachments) removed. The ..._stripped_... method will also try to render HTML message parts into their text equivalents. get_uri_list(), get_uri_detail_list() Retrieves a list of the URIs mentioned in the message. The ..._detail_... method provides a data structure with more details about the URIs. get() Very similar to the header() functionality we’ve seen, but with a couple of helpful twists. Let’s look at the URI-related methods in more detail. There may be multiple reasons to extract all of the URIs found in a message. In the context of SpamAssassin, doing so offers another tool for spam detection. When 300 | Chapter 8: Email sending spam related to products or services for sale over the Internet, spammers usu- ally include URLs in the messages so people can go to the sites that sell their wares. If you extracted the URLs, you could then conceivably filter based on the URLs them- selves. You could even look for the domains mentioned in those URLs and then, once you’ve identified a set of spammer’s domains, look for messages that contain them and deal with them accordingly. The SURBL blacklists ( are predica- ted on this idea (they list the domains extracted from known spam messages) and tend to be very effective. This is the use for URIs foreshadowed at the end of “Dissecting a Single Message” on page 291. The get_uri_detail_list() method returns a data structure like this (from the documentation): raw_uri => { types => { a => 1, img => 1, parsed => 1 }, cleaned => [ canonified_uri ], anchor_text => [ "click here", "no click here" ], domains => { domain1 => 1, domain2 => 1 }, } The hash of a hash data structure requires us to do a little work to get the list of unique domains mentioned in the URIs, but it’s not that hard: use Mail::SpamAssassin; use File::Slurp qw(slurp); use List::MoreUtils qw(uniq); my $sa = Mail::SpamAssassin->new(); my $status = $sa->check_message_text( scalar slurp 'spam.txt' ); my $uris = $status->get_uri_detail_list(); my @domains; foreach my $uri ( keys %{$uris} ) { next if $uri =~ /^mailto:/; push( @domains, keys %{ $uris->{$uri}->{domains} } ); } print join( "\n", uniq @domains ); Once you have a set of unique domains, you can look them up in a blacklist like those available at ( Each blacklisted domain is recorded as a DNS “A” resource record (hostname) entry under the domain. This provides an easy way to check whether a domain has been blacklisted: simply prepend the do- main name in question to (e.g., and performing a DNS lookup for the resulting hostname. This can be done using Net::DNS, as demonstrated in Chapter 5. If the hostname resolves, you can take action accordingly, because that domain is in the blacklist. To finish this section, let’s briefly look at the get() method. To use get(), we first run check() on the message object or check_message_text() on the message. These methods Processing Mail | 301 each return a status object from which a get() method call can be made. The get() method extracts headers just as Email::Simple’s header() method does, but the addi- tional parsing done by a check() gives it a few extra superpowers. For example, header('From') will return the From header from a message, but get('From:addr') takes an extra step and returns just the address part of that header. Likewise, get('From:name') will return just the “name” part of the header. For instance, if the header contains: David Blank-Edelman we can use the :addr form to return “” and the :name form to return “David Blank-Edelman”. The get() method also provides a set of pseudo-headers that can be queried. These are typically aggregates of other headers. For example, if you want to retrieve all of the stated recipients of a message (Bccs aside), using ToCc as the header name will get that list for you. SpamAssassin allows you to configure “trusted” and “untrusted” hosts at install time. This helps it distinguish between hosts that are locally controlled and those in the big, bad, scary Internet for the purpose of determining how likely they are to contribute to the “spaminess” of a message. There are several get() pseudo-headers that return information based on this distinction, but I think the most interesting ones are X-Spam-Relays-Untrusted and X-Spam-Relays-Trusted. Here’s an example set of received headers from a real piece of spam: Received: from ( []) by (8.8.6/8.8.6) with ESMTP id FAA29389 for ; Tue, 2 Dec 1997 05:51:56 −0500 (EST) Received: from ( []) by (8.8.7/8.8.7) with SMTP id CAA01384; Tue, 2 Dec 1997 02:53:33 −0800 (PST) Received: from ( (208.137.887.15)) by (8.8.5/8.6.5) with SMTP id GAA00064 for <>; Tue, 02 Dec 1997 01:49:32 −0600 (EST) Asking for X-Spam-Relays-Untrusted gives us (slightly reformatted): [ ip= helo=smtp. ident= envfrom= intl=0 id=FAA29389 auth= msa=0 ] [ ip= ident= envfrom= intl=0 id=CAA01384 auth= msa=0 ] SpamAssassin has parsed the headers into a form that makes it easy to see the incon- sistencies (to put it charitably) between the information the sender presented to us and the actual origins of the message. This is most clear in the last line of the output, cor- responding to the second Received header in the input, where the sender claimed to be from but instead was actually coming from (presumably) a dial-up line at 302 | Chapter 8: Email Feedback loops There’s another side to the spam discussion that often gets forgotten. We’ve just looked at how you deal with messages you’ve received to determine if they are spam. But what if you’re on the other side of the fence and want to avoid being labeled a spammer due to the mail you send? Legitimate bulk email senders and people who run email systems, especially the large ones, actually have some goals in common. Both want people to receive the mail they have opted to receive, and neither wants to be part of a process that leaves the users feeling like they’ve received spam. The bulk emailer doesn’t want to send something the user does not want to receive, and the email system administrator doesn’t want to anger the users by continuing to deliver mail they don’t desire. This common ground gives the two parties a reason to collaborate. One thing they can do is share information about which messages are considered to be spam, either because the service provider uses software that has tagged it as such, or because the recipient has actually pressed a mark this as spam button in her mail client. In either case, the sender will generally want to know that this has taken place so that the recipient’s address can be removed from its database, and so on. That’s where feedback loops come into play. Some of the large email providers let bulk-email-sending companies subscribe to a feedback loop. This loop sends information back to the sender about the messages the email provider has received (from the bulk-email company) that were labeled as spam. The best collection of available feedback loops I know about at the time of this writing can be found in the Spamhaus FAQ mentioned in the references section at the end of this chapter. You may be wondering what role Perl can play in all of this. A number of the big players in the arena got together and hashed out a standardized format for the reports that are sent as part of a feedback loop. This format is called the Abuse Reporting Format (ARF) and is documented in the draft specification also pointed to in the references section. It’s a MIME-based format that automated systems can use to send and receive this sort of spam report. The availability of this common format makes it easier to write Perl scripts that can parse incoming reports and act on them. You may be saying “MIME? Great!† We went over MIME parsing before, I know how to do that!” And if you said this, you wouldn’t be wrong. However, there are two specialized Perl packages that make handling ARF reports a little easier than rolling your own code based on the standard Perl MIME parsers. I’ll show you an example of parsing an ARF message using Email::ARF::Report, from the Perl Email Project. If you plan to send ARF messages, you may also want to look at MIME::ARF, by Steve Atkins (found at † Or, if you’re as ambivalent about MIME as I am, you may choose a word that begins with a letter a little earlier in the alphabet. Processing Mail | 303 Just so we know what we’re dealing with, Table 8-1 shows the three mandatory parts of an ARF message. The example text comes from the ARF spec. Table 8-1. ARF message structure Part Contents Example 1 Human-readable information This is an email abuse report for an email message received from IP on Thu, 8 Mar 2005 14:00:00 EDT. For more information about this format please see 2 Machine-readable metadata Feedback-Type: abuse User-Agent: SomeGenerator/1.0 Version: 0.1 3 Full copy of message or message headers From: Received: from ( []) by with ESMTP id M63d4137594e46; Thu, 08 Mar 2005 14:00:00 −0400 To: Subject: Earn money MIME-Version: 1.0 Content-type: text/plain Message-ID: Date: Thu, 02 Sep 2004 12:31:03 −0500 Spam Spam Spam Spam Spam Spam Spam Spam Spam Please refer to the ARF spec to see these parts in the context of a full email message. Taking apart a message like this is pretty easy with Email::ARF::Report. Here’s example code that prints some information from the original message copied into an ARF report: use Email::ARF::Report; use File::Slurp qw(slurp); my $message = slurp('arfsample1.txt'); my $report = Email::ARF::Report->new($message); foreach my $header (qw(to date subject message-id)) { print ucfirst $header . ': ' . $report->original_email->header($header) . "\n"; } If this looks remarkably like the Email::Simple code we looked at earlier, that’s no coincidence. Email::ARF::Report parses an ARF message and provides methods to re- trieve parts of that report. In this case we’re using the method original_email() to access the original message. original_email() is kind enough to return an Email::Simple object, so we can put to use all of our previous knowledge and call the 304 | Chapter 8: Email header() method on that object as desired. Once we’ve extracted the information we need from the report, we can do whatever we like with that info (unsubscribe the user, etc.). But spam is such an unpleasant subject. Let’s move on to a cheerier topic, such as interacting with users via email. Support Mail Augmentation Even if you don’t have a “help desk” at your site, you probably have some sort of support email address for user questions and problems. Email as a medium for support com- munications has certain advantages: • It can be stored and tracked, unlike hallway conversations. • It is asynchronous; the system administrator can read and answer mail during the more rational nighttime hours. • It can be a unicast, multicast, or broadcast medium. If 14 people write in, it’s possible to respond to all of them simultaneously when the problem is resolved. • It can easily be forwarded to someone else who might know the answer or have authority over that service domain. These are all strong reasons to make email an integral part of any support relationship. However, email does have certain disadvantages: • If there is a problem with your email system itself, or if the user is having email- related problems, another medium must be used. • Users can and will type anything they want into an email message. There’s no guarantee that this message will contain the information you need to fix the prob- lem or assist the user. You may not gain even a superficial understanding of the purpose of the email. This leads us to the conundrum we’ll attempt to address in this section. My favorite support email of all time is reproduced in its entirety here, with only the name of the sender changed to protect the guilty: Date: Sat, 28 Sep 1996 12:27:35 −0400 (EDT) From: Special User To: Subject: [Req. #9531] printer help something is wrong and I have know idea what If the user hadn’t mentioned “printer” in the subject of the mail, we would have had no clue where to begin and would probably have chalked the situation up to existential angst. Granted, this was perhaps an extreme example. More often, you’ll receive mail like this: Processing Mail | 305 From: Another user Subject: [Req #14563] broken macine To: Date: Wed, 11 Mar 1998 10:59:42 −0500 (EST) There is something wrong with the following machine: A user does not send mail devoid of contextual content like this out of malice. I believe the root cause of these problems is an impedance mismatch between the user’s and the system administrator’s mental model of the computing environment. For most users, the visible structure of the computing environment is limited to the client machine they are logged into, the nearby printer, and their storage (i.e., home directory). For a system administrator, the structure of the computing environment is considerably different. It consists of a set of servers providing services to clients, all of which may have a multitude of different peripheral devices. Each machine may have a different set of software installed and a different state (system load, configuration, etc.). To users, the question “Which machine is having a problem?” may seem strange. They’re talking about the computer, the one they’re using now. Isn’t that obvious? To a system administrator, a request for “help with the printer” is equally odd; after all, there are likely many printers in his charge. So too it goes with the specifics of a problem. System administrators around the world grit their teeth every day when they receive mail that says, “My machine isn’t working, can you help me?” They know “not working” could indicate a whole panoply of symp- toms, each with its own array of causes. To a user that has experienced three screen freezes in the last week, however, “not working” is unambiguous. One way to address this disconnect is to constrain the material sent in email. Some sites force the users to send in trouble reports using a custom support application or web form. The problem with this approach is that very few users enjoy engaging in a click- and-scroll fest just to report a problem or ask a question. The more pain is involved in the process, the less likely it is that someone will go to the trouble of using these mech- anisms. It doesn’t matter how carefully constructed or beautifully designed your web form is if no one is willing to use it. Hallway requests will become the norm again. Back to square one? Well, with the help of Perl, maybe not. Perl can help us augment normal mail receiving to assist us in the support process. One of the first steps in this process for a system administrator is the identification of locus: “Where is the problem? Which printer? Which machine?” And so on. Let’s take a look at a program I call suss, which demonstrates the augmentation I have in mind in a simple fashion. It looks at an email message and attempts to guess the name of the machine associated with that message. The upshot of this is that we can often determine the hostname for the “My machine has a problem” category of email 306 | Chapter 8: Email without having to engage in a second round of email with the vague user. That host- name is typically a good starting point in the troubleshooting process. suss uses an extremely simple algorithm to guess the name of the machine in question (basically just a hash lookup for every word in the message). First it examines the mes- sage subject, then the body of the message, and finally it looks at the initial Received header on the message. Here’s a simplified version of the code that expects to be able to read an /etc/hosts file to determine the names of our hosts:‡ use Email::Simple; use List::MoreUtils qw(uniq); use File::Slurp qw(slurp); my $localdomain = ""; # read in our host file open my $HOSTS, '<', '/ccs/etc/hosts' or die "Can't open hosts file\n"; my $machine; my %machines; while ( defined( $_ = <$HOSTS> ) ) { next if /^#/; # skip comments next if /^$/; # skip blank lines next if /monitor/i; # an example of a misleading host $machine = lc( (split)[1] ); # extract the first host name & downcase $machine =~ s/\Q$localdomain\E//oi; # remove our domain name $machines{$machine}++ unless $machines{$machine}; } close $HOSTS; # parse the message my $message = new Email::Simple( scalar slurp( $ARGV[0] ) ); my @found; # check in the subject line if ( @found = check_part( $message->header('Subject'), \%machines ) ) { print 'subject: ' . join( ' ', @found ) . "\n"; exit; } # check in the body of the message if ( @found = check_part( $message->body, \%machines ) ) { print 'body: ' . join( ' ', @found ) . "\n"; exit; } # last resort: check the last Received line my $received = ( reverse $message->header('Received') )[0]; $received =~ s/\Q$localdomain\E//g; if ( @found = check_part( $received, \%machines ) ) { print 'received: ' . join( ' ', @found ) . "\n"; ‡ In real life you would probably want to use something considerably more sophisticated to get a host list, like a cached copy of a DNS zone transfer or perhaps a walk of an LDAP tree. Processing Mail | 307 } # find all unique matches from host lookup table in given part of message sub check_part { my $part = shift; # the text from that message part my $machines = shift; # a reference to the machine lookup table $part =~ s/[^\w\s]//g; $part =~ s/\n/ /g; return uniq grep { exists $machines->{$_} } split( ' ', lc $part ); } One comment on this code: the simplicity of our word check becomes painfully appa- rent when we encounter perfectly reasonable hostnames like monitor in sentences like “My monitor is broken.” If you have hostnames that are likely to appear in support messages, you’ll either have to special-case them, as we do with next if /monitor/i;, or preferably create a more complicated parsing scheme. Let’s take this code out for a spin. Here are two real support messages: Received: from ( []) by (8.8.4/8.7.3) with ESMTP id RAA27043 for ; Thu, 29 Mar 2007 17:07:44 −0500 (EST) From: User Person Received: (user@localhost) by (8.8.4/8.6.4) id RAA10500 for systems; Thu, 29 Mar 2007 17:07:41 −0500 (EST) Message-Id: <> Subject: [Req #11509] Monitor To: Date: Thu, 29 Mar 2007 17:07:40 −0500 (EST) Hi, My monitor is flickering a little bit and it is tiresome whe working with it to much. Is it possible to fix it or changing the monitor? Thanks. User. ------------------------------------- Received: from ( []) by (8.8.4/8.7.3) with SMTP id SAA00732 for ; Thu, 29 Mar 2007 18:34:54 −0500 (EST) Date: Thu, 29 Mar 2007 18:34:54 −0500 (EST) From: Another User To: Subject: [Req #11510] problems with two computers Message-Id: In Jenolen (in room 292), there is a piece of a disk stuck in it. In intrepid, there is a disk with no cover (or whatever you call that silver thing) stuck in it. We tried to turn off intrepid, but it wouldn't work. We (the proctor on duty and I) tried to get the disk piece out, but it didn't work. The proctor in charge 308 | Chapter 8: Email decided to put signs on them saying 'out of order' AnotherUser Aiming our code at these two messages yields: received: strontium and: body: jenolen intrepid Both hostname guesses were right on the money, and that’s with just a little bit of simple code. To take things one step further, let’s assume we got this email from a user who doesn’t realize we’re responsible for a herd of 30 printers: Received: from [] ( []) by (8.8.6/8.8.6) with SMTP id JAA16638 for ; Tue, 7 Aug 2007 09:07:15 −0400 (EDT) Message-Id: Date: Tue, 7 Aug 2007 09:07:16 −0400 To: From: (Nice User) Subject: [Req #15746] printer Could someone please persuade my printer to behave and print like a nice printer should? Thanks much :) -Nice User. Fortunately, we can use Perl and a basic observation to help us make an educated guess about which printer is causing the problem. Users tend to print to printers that are geographically close to the machines they are using at the time. If we can determine which machine the user sent the mail from, we can probably figure out which printer she’s using. There are many ways to retrieve a machine-to-printer mapping (e.g., from a separate file, from a field in the host database we mentioned in Chapter 5, or even a directory service from LDAP). Here’s some code that uses a simple hostname-to- associated printer database: use Email::Simple; use File::Slurp qw(slurp); use DB_File; my $localdomain = ''; my $printdb = 'printdb'; # parse the message my $message = new Email::Simple( scalar slurp $ARGV[0] ); # check in the subject line my $subject = $message->header('Subject'); if ( $subject =~ /print(er)?/i ) { Processing Mail | 309 # find sending machine my $received = ( reverse $message->header('Received') )[0]; my ($host) = $received =~ /\((\S+)\Q$localdomain\E \[/; tie my %printdb, 'DB_File', $printdb or die "Can't tie $printdb database:$!\n"; print "Problem on $host may be with printer " . $printdb{$host} . ".\n"; untie %printdb; } If the message mentions “print,” “printer,” or “printing” in its subject line, we pull out the hostname from the Received header. We know the format our mail hub uses for Received headers, so we can construct a regular expression to extract this information. (If this does not match your MTA’s format, you may have to fiddle with the regexp a little bit.) With the hostname in hand, we can look up the associated printer in a Berkeley DB database. The end result: Problem on buggypeak may be with the printer called prints-charming. If you take a moment to examine the fabric of your environment, you will see other ways to augment the receiving of your support email. The examples in this section were small and designed to get you thinking about the possibilities. The suss program, with rules specific to your environment, could become a frontend to almost any kind of ticketing system. Combining it with the earlier concepts of using Perl to retrieve and parse mail would allow you to build a system that lets users send mail to a “catachall” address, such as “,” where it’s automat- ically parsed. If there’s good confidence about the determination of the subject of the question (the Mail::SpamAssassin rules provide an example of that kind of scoring), the mail could automatically be forwarded to the person designated for handling that type of problem. Perl gives you many ways to analyze your email, place it in a larger context, and then act upon that information. I’ll leave it to you to consider other kinds of help programs that read mail (perhaps mail sent by other programs) could provide you. Module Information for This Chapter Module CPAN ID Version Mac::Carbon CNANDOR 0.77 Win32::OLE (ships with ActiveState Perl) JDB 0.1709 Mail::Outlook (found in MailTools) BARBIE 0.13 Email::Send RJBS 2.192 310 | Chapter 8: Email Module CPAN ID Version Email::Simple RJBS 2.003 Email::Simple::Creator RJBS 1.424 Email::MIME RJBS 1.861 Email::MIME::Creator RJBS 1.454 File::Slurp DROLSKY 9999.13 Email::MIME::CreateHTML BBC 1.026 Text::Wrap (found in Text-Tabs+Wrap and also ships with Perl) MUIR 2006.1117 File::Spec (found in PathTools and also ships with Perl) KWILLIAMS 3.2701 IO::Socket (found in IO and also ships with Perl) GBARR 1.30 Carp (ships with Perl) 1.08 Mail::POP3Client SDOWD 2.18 Mail::IMAPClient MARKOV 3.08 Mail::IMAPTalk ROBM 1.03 IO::Socket::SSL SULLR 1.13 Mail::Box MARKOV 2.082 Text::Match::FastAlternatives ARC 1.00 Regexp::Common ABIGAIL 2.122 Email::Folder RJBS 0.854 Mail::SpamAssassin JMASON 3.24 List::MoreUtils VPARSEVAL 0.22 Email::ARF::Report RJBS 3.01 DB_File (ships with Perl) PMQS 1.817 References for More Information The POP3 and IMAPv4 sections of this chapter are revised and modified from a column I originally wrote for the February 2008 issue of the USENIX Association’s ;login magazine ( Network Programming with Perl, by Lincoln Stein (Addison-Wesley), is one of the best books on programming network servers in Perl. Perl Cookbook (, by Tom Christiansen and Nathan Torkington (O’Reilly), also addresses the programming of network servers. is the website of the Coalition Against Unsolicited Commercial Email. There are many sites devoted to fighting spam; this site is a good place to start. It has pointers to many other sites, including those that go into greater detail about the analysis of mail headers for this process. References for More Information | 311 is the home page for the Perl Email Project. has a number of good anti-spam-related FAQ lists, in- cluding one on ISP spam issues that addresses feedback loops and other ways ISPs can address spam issues for and with their customers. and are two good resources for information on the ARF standard. The latest draft of the stand- ard itself (as of this writing) can be found at -shafranovich-feedback-report-07.txt. (See for the latest draft.) If you’d like to experiment with high-volume mail handling from a server perspective (especially in the anti-spam context), there are two very interesting pieces of software you may want to investigate: qpsmtpd ( and Traffic Con- trol ( The first is an open source package, and the second is a commercial package free for use under some conditions. Both are SMTP handlers/ daemons written in Perl that are meant to sit in front of a standard MTA and proxy only good mail to it. What makes these two interesting for this chapter in particular is their plug-in functionality. A user can write plug-ins in Perl to change or direct how messages that pass through these packages are processed. Often these plug-ins attempt to do some sort of spam/ham determination, but really, the sky is the limit. You may also want to take a look at these RFCs: • RFC 1939: Post Office Protocol - Version 3, by J. Myers and M. Rose (1996) • RFC 2045: Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies, by N. Freed and N. Borenstein (1996) • RFC 2046: Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types, by N. Freed and N. Borenstein (1996) • RFC 2047: MIME (Multipurpose Internet Mail Extensions) Part Three: Message Header Extensions for Non-ASCII Text, by K. Moore (1996) • RFC 2077: The Model Primary Content Type for Multipurpose Internet Mail Ex- tensions, S. Nelson and C. Parks (1997) • RFC 2821: Simple Mail Transfer Protocol, by J. Klensin (2001) • RFC 2822: Internet Message Format, by P. Resnick (2001) • RFC 3501: INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1, by M. Crispin (2003) • RFC 3834: Recommendations for Automatic Responses to Electronic Mail, by K. Harrenstien and K. Moore (2004) • RFC 4288: Media Type Specifications and Registration Procedures, by N. Freed and J. Klensin (2005) • RFC 4289: Multipurpose Internet Mail Extensions (MIME) Part Four: Registration Procedures, N. Freed and J. Klensin (2005) 312 | Chapter 8: Email CHAPTER 9 Directory Services The larger an information system gets, the harder it becomes to find anything in that system, or even to know what’s available. As networks grow and become more com- plex, they are well served by some sort of directory. Network users might make use of a directory service to find other users for email and messaging services. A directory service might advertise resources on a network, such as printers and network-available disk areas. Public-key and certificate infrastructures could use a directory service to distribute information. In this chapter we’ll look at how to use Perl to interact with some of the more popular directory services, including Finger, WHOIS, LDAP, and Active Directory (via Active Directory Service Interfaces). What’s a Directory? In Chapter 7, I suggested that all the system administration world is a database. Di- rectories are a good example of this characterization. For the purpose of our discussion, we’ll distinguish between “databases” and “directories” by observing a few salient characteristics of directories: Networked Directories are almost always networked. Unlike a database, which may live on the same machine as its clients (e.g., the venerable /etc/passwd file), directory services are usually provided over a network. Simple communication/data manipulation Databases often have complex query languages for data queries and manipulation. We looked at the most common of these, SQL, in Chapter 7 (and in Appen- dix D). Communicating with a directory is a much simpler affair. A directory client typically performs only rudimentary operations and does not use a full-fledged language as part of its communication with the server. As a result, any queries made are often much simpler. Hierarchical Modern directory services encourage the building of tree-like information struc- tures, whereas databases on the whole do not. 313 Read-many, write-few Modern directory servers are optimized for a very specific data traffic pattern. Un- der normal use, the number of reads/queries to a directory service far outweighs the number of writes/updates. If you encounter something that looks like a database (and is probably backended by a database) but has the preceding characteristics, you’re probably dealing with a directory. In the four directory services we’re about to discuss, these characteristics will be easy to spot. Finger: A Simple Directory Service Finger and WHOIS are good examples of simple directory services. Finger exists pri- marily to provide read-only information about a machine’s users (although we’ll see some more creative uses shortly). Later versions of Finger, like the GNU Finger server and its derivatives, expanded upon this basic functionality by allowing you to query one machine and receive back information from all the machines on your network. Churn Your Own Butter, Too? I think you’d be hard-pressed to find a site running Finger these days (the World Wide Web and privacy concerns drove it to near extinction)—so why is it still in the book? I’m including a very short discussion of Finger here for one reason: it is an excellent training-wheels protocol. The protocol itself is simple, which makes it ideal for learning to deal with text-based network services that don’t have their own custom client mod- ules. You’ll be happy you paid attention to this little slice of history the first time you need to interact with a service like that. Finger was one of the first widely deployed directory services. Once upon a time, if you wanted to locate a user’s email address at another site, or even within your own, the finger command was the best option. finger would tell you whether Harry’s email address was harry, hpotter, or something more obscure (along with listing all of the other Harrys at that school). Finger’s popularity has waned over time as web home pages have become prevalent and the practice of freely giving out user information has become problematic. Using the Finger protocol from Perl provides another good example of TMTOWTDI. In 2000, when I first looked on CPAN for a module to perform Finger operations, there were none available. If you look now, you’ll find Dennis Taylor’s Net::Finger module, which he published six months or so after my initial search. We’ll see how to use it in a moment, but in the meantime, let’s pretend it doesn’t exist and take advantage of this opportunity to learn how to use a more generic module to talk to a specific protocol when the “perfect” module doesn’t exist. 314 | Chapter 9: Directory Services The Finger protocol itself is a very simple TCP/IP-based text protocol. Defined in RFC 1288, it calls for a standard TCP connection to port 79. The client passes a simple CRLF-terminated* string over the connection. This string either requests specific user information or, if empty, asks for information about all the machine’s users. The server responds with the requested data and closes the connection at the end of the data stream. You can see this in action by telneting to the Finger port directly on a remote machine:† $ telnet 79 Trying Connected to Escape character is '^]'. /W quake RAPID EARTHQUAKE LOCATION SERVICE U.S. Geological Survey, Menlo Park, California. U.C. Berkeley Seismological Laboratory, Berkeley, California. (members of the Council of the National Seismic System) ... DATE-(UTC)-TIME LAT LON DEP MAG Q COMMENTS yy/mm/dd hh:mm:ss deg. deg. km ------------------------------------------------------------------------------ 09/01/12 16:29:37 36.03N 120.59W 4.5 2.3Md A* 20 km NW of Parkfield, CA 09/01/13 08:17:38 38.81N 122.82W 2.4 2.1Md A* 2 km NNW of The Geysers, CA 09/01/13 11:51:09 40.66N 124.04W 23.6 2.5Md B* 12 km NE of Fortuna, CA 09/01/13 18:27:01 36.80N 121.51W 5.5 2.4Md A* 5 km SSE of San Juan Bautista, CA 09/01/14 00:29:11 39.37N 123.27W 3.1 2.2Md B* 8 km ESE of Willits, CA 09/01/14 01:48:23 38.24N 118.69W 12.0 2.3Md C* 17 km WSW of Qualeys Camp, NV 09/01/14 02:06:57 38.24N 118.69W 6.0 2.2Md C* 17 km WSW of Qualeys Camp, NV 09/01/14 03:44:02 38.82N 122.83W 2.1 2.1Md A* 3 km NW of The Geysers, CA 09/01/14 05:08:21 36.74N 121.34W 9.1 3.4Ml A* 6 km SSW of Tres Pinos, CA 09/01/14 07:46:02 39.04N 123.34W 0.1 2.2Md C* 17 km SW of Ukiah, CA 09/01/14 10:24:53 40.42N 125.07W 1.2 3.1Ml C* 67 km W of Petrolia, CA 09/01/14 17:32:54 38.84N 122.83W 1.9 2.2Md A* 5 km NNW of The Geysers, CA 09/01/14 17:57:34 36.56N 121.16W 6.7 2.4Md A* 4 km NNW of Pinnacles, CA ... $ In this example we’ve connected directly to’s Finger port. We typed the username “quake” (with a /W to ask for verbose information), and the server returned information about that user. I chose this particular host and user just to show you some of the variety of information that used to be available via Finger servers back in the early days of the Internet. Finger servers got pressed into service for all sorts of tasks. You used to be able to send Finger requests to soda machines, hot tubs, and sensor machines of all sorts. The Finger * Carriage return + linefeed; i.e., ASCII 13 + ASCII 10. † There used to be a whole battalion of earthquake information Finger servers. This is the only one of the 16 I tried that was still working as of this writing. If you don’t get a response from this server, you’ll have to trust me that it did return this data once upon a time. Finger: A Simple Directory Service | 315 example just shown, for instance, allows anyone anywhere on the planet to see infor- mation on earthquakes recorded by seismic sensors. Unfortunately, making interesting information like this available via Finger seems to be a dying art. At the time of this writing, none of the Finger hosts listed in Bennet Yee’s “Internet Accessible Coke Machines” and “Internet Accessible Machines” pages (http: // were operational. HTTP has almost entirely supplanted the Finger protocol for these sorts of tasks. There may still be Finger servers available on the Internet, but they are mostly set up for internal use. Even though Finger servers are less prevalent today than they used to be, the simplicity of the protocol itself makes it a good place to start if you are looking to learn how to roll your own simple network service clients. Let’s take the network communication we just performed using a telnet binary back to the world of Perl. With Perl, we can also open up a network socket and communicate over it. Instead of using lower-level socket commands, we’ll use Jay Rogers’s Net::Telnet module.‡ Net::Telnet will handle all of the connection setup work for us and provide a clean interface for sending and receiving data over this connection. Though we won’t use them in this example, Net::Telnet also provides some handy pattern-scanning mech- anisms that allow programs to watch for specific responses from the other server. Here’s a Net::Telnet version of a simple Finger client. This code takes an argument of the form user@finger_server. If the username is omitted, the server will return a list of all users it considers active. If the hostname is omitted, we query the local host: use Net::Telnet; my($username,$host) = split(/\@/,$ARGV[0]); $host = $host ? $host : 'localhost'; # create a new connection my $cn = new Net::Telnet(Host => $host, Port => 'finger'); # send the username down this connection # /W for verbose information as per RFC 1288 unless ($cn->print("/W $username")){ $cn->close; die 'Unable to send finger string: '.$cn->errmg."\n"; } # grab all of the data we receive, stopping when the # connection is dropped my ($ret,$data); while (defined ($ret = $cn->get)) { $data .= $ret; } ‡ If Net::Telnet didn’t fit the bill so nicely, another alternative would be (written by Austin Schutz and now maintained by Roland Giersig), driving telnet or another network client. 316 | Chapter 9: Directory Services # close the connection $cn->close; # display the data we collected print $data; You may have noticed the /W in the string we passed to print(): RFC 1288 specifies that a /W switch can be prepended to the username sent to the server to request it to provide “a higher level of verbosity in the user information output.” If you needed to connect to another TCP-based text protocol besides Finger, you’d use very similar code. For example, the following code connects to a daytime server (which shows the local time on a machine): use Net::Telnet; my $host = $ARGV[0] ? $ARGV[0] : 'localhost'; my $cn = new Net::Telnet(Host => $host, Port => 'daytime'); port 13 my ($ret,$data); while (defined ($ret = $cn->get)) { $data .= $ret; } $cn->close; print $data; Now you have a sense of how easy it is to create generic TCP-based network clients. If someone has taken the time to write a module specifically designed to handle a proto- col, it can be even easier. In the case of Finger, you can use Taylor’s Net::Finger to turn the whole task into a single function call: use Net::Finger; # finger() takes a user@host string and returns the data received print finger($ARGV[0]); Just to present all of the options, there’s also the fallback position of calling another executable (if it exists on the machine), like so: my($username,$host) = split('@',$ARGV[0]); $host = $host ? $host : 'localhost'; # location of finger executable my $fingerex = ($^O eq 'MSWin32') ? $ENV{'SYSTEMROOT'}.'\\System32\\finger' : '/usr/bin/finger'; # (could also be /usr/ucb/finger) print `$fingerex ${username}\@${host}` Now you’ve seen three different methods for performing Finger requests. The third method is probably the least ideal because it requires spawning another process. Finger: A Simple Directory Service | 317 Net::Finger will handle simple Finger requests; for everything else, Net::Telnet or any of its kin should work well for you. The WHOIS Directory Service WHOIS is another useful read-only directory service. WHOIS provides a service like a telephone directory for machines, networks, and the people who run them. Some larger organizations (such as IBM, UC Berkeley, and MIT) provide WHOIS services, but the most important WHOIS servers by far are those run by the InterNIC and other Internet registries such as RIPE (European IP address allocations) and APNIC (Asia/Pacific ad- dress allocations). If you have to contact a system administrator at another site to report suspicious net- work activity, you can use WHOIS to get the contact info.* GUI and command-line tools are available, making WHOIS queries possible on most operating systems. All of the registrars also have web-based WHOIS query pages. Under Unix, a typical query using a command-line interface looks like this: % whois -h Registrant: Brandeis University Library and Technology Services MS017 415 South Street Waltham, MA 02453-2728 UNITED STATES Administrative Contact: Director for Networks & Systems Brandeis University Library and Technology Services MS017 PO Box 9110 Waltham, MA 02454-9110 UNITED STATES (781) 736-4569 Technical Contact: NetSys Brandeis University Library and Technology Services MS017 PO Box 9110 Waltham, MA 02454-9110 UNITED STATES (781) 736-4571 * If you feel a breeze after reading this sentence, that’s because there’s a lot of hand waving behind this overly simplistic statement. In a page or two the reality of the situation will make its entrance. 318 | Chapter 9: Directory Services Name Servers: LILITH.UNET.BRANDEIS.EDU FRASIER.UNET.BRANDEIS.EDU NS1.UMASS.EDU NS2.UMASS.EDU NS3.UMASS.EDU Domain record activated: 27-May-1987 Domain record last updated: 11-Jun-2008 Domain expires: 31-Jul-2009 If you need to track down the owner of a particular IP address range, WHOIS is also the right tool: % whois -h 129.64.2 OrgName: Brandeis University OrgID: BRANDE Address: 415 South Street City: Waltham StateProv: MA PostalCode: 02454 Country: US NetRange: - CIDR: NetName: BRANDEIS NetHandle: NET-129-64-0-0-1 Parent: NET-129-0-0-0-0 NetType: Direct Assignment NameServer: LILITH.UNET.BRANDEIS.EDU NameServer: FRASIER.UNET.BRANDEIS.EDU Comment: RegDate: 1987-09-04 Updated: 2002-10-24 TechHandle: ZB114-ARIN TechName: Brandeis University Information Technology TechPhone: +1-781-736-4800 TechEmail: # ARIN WHOIS database, last updated 2009-01-13 19:10 # Enter ? for additional hints on searching ARIN's WHOIS database. The previous sessions used a command-line WHOIS client like that found in Unix and Mac OS X distributions. Windows-based operating systems do not ship with such a client, but that shouldn’t stop users of Windows systems from accessing this informa- tion. There are many fine free and shareware clients available; the cygwin distribution contains one, and the Net::Whois::Raw module introduced in a few paragraphs also provides a client. A recent wise footnote warned you that there was some hand waving going on. Let’s dispense with that now and get to the reality of the situation: as of this writing, the WHOIS situation on the Internet continues to be in considerable flux. Several of the The WHOIS Directory Service | 319 previous Perl solutions for doing WHOIS queries are now, quite frankly, in shambles as a result of this situation. Let me try to explain without going too deep into the morass. Once upon a time there was one registry for all Internet-related WHOIS information. This made it easy to write Perl code that created a query and properly parsed the response. For political and per- haps technical reasons, the Pangaea of registries was split into different subregistries. This meant that WHOIS query code had to become smarter about where to send a query and how to parse the response (thanks to variations in the different output formats of new severs as they were introduced). Even with this added complexity, the Perl module authors were able to keep up. The changes in the WHOIS landscape happened infrequently enough that authors were able to release new versions to handle them. Some created frameworks for plugging in new server formats and locations. Vipul Ved Prakash wrote one good example of this, called Net::XWhois. As the registrar churn continued and even accelerated, the amount of bitrot in this area became more and more apparent. Net::Whois, now maintained by Dana Hudes but last updated in 1999, doesn’t work much of the time: a change to the registry provider for the top-level domain (TLD) .org broke Net::XWhois’s lookups for those sites, and so on. For a while, none of the existing modules really could be trusted to work. Before we break out the guitar and start to compose a blues number about this sad state of affairs, it turns out there is a ray of hope that can help us get out of this situation. The fine folks at CenterGate Research Group LLC set up the domain In this domain, they’ve registered CNAMEs for all the TLDs on the Internet. These CNAMEs point to the name of the registrar for each TLD. For example, to find the registrar for the .com TLD, we could type: $ host is an alias for has address It would be easy enough to use a module like Net::DNS to retrieve this information, but luckily, at least one module author has beaten us to it. The Net::Whois::Raw module, maintained by Walery Studennikov, uses and is still being actively developed. Using it is as trivial as using Net::Finger was in the last section: use Net::Whois::Raw; my $whois = whois(''); Puny as this code sample is, there are a couple of small details behind it that you’ll want to know. First, using the default options, as we’ve done here, only queries the name servers for top-level domains not already in the module’s hard- coded registrar table. To always rely on for registrar info, you need to import and set an option like this: 320 | Chapter 9: Directory Services use Net::Whois::Raw; $Net::Whois::Raw::USE_CNAMES = 1; my $whois = whois(''); The second detail worth knowing about is the $OMIT_MSG option, set in the same way $USE_CNAMES was in this the last example. $OMIT_MSG will do its best to remove the lengthy copyright disclaimers most WHOIS servers return these days. It uses a set of hardcoded regular expressions, though, so rely on it with caution. $OMIT_MSG aside, Net::Whois::Raw just returns the results of a WHOIS query in raw form: it makes no attempt to parse the information returned, like Net::Whois and Net::Xwhois used to do. That’s probably a wise decision on the author’s part, because the format of the response seems to change from registrar to registrar. All of the suc- cessful queries will have fields of some sort. You’ll likely find at least Name, Address, and Domain fields in the response, but who knows how they’ll be formatted, what order they’ll appear in, etc. This can make WHOIS data really annoying to parse and render the resulting programs brittle. To get away from this problem, we have to look at more complex directory protocols, like LDAP. One final idea before we leave this section. There’s one other approach we haven’t discussed because it also can be a bit dicey. There are a few public services that provide WHOIS proxy servers that attempt to do the work for you. You can query them like any other server, and you’ll get results based on someone’s code that works hard to query the right places on your behalf and format the output in a reasonable way. Two such services are found at (sponsored by Center- Gate Research Group; see for general WHOIS queries and (see for data based on the global routing tables. In both cases you can just point a standard whois client at them (e.g., whois -h and whois -h and they will do the right thing. The key issues with using public servers like these for your mission-critical ap- plication are: a) they typically have usage limits (to prevent abuse), and b) someone else is running them, so if they go down, sorry! But for the occasional query, they can be very handy. LDAP: A Sophisticated Directory Service The Lightweight Directory Access Protocol, or LDAP (including its Active Directory implementation), is a much richer and more sophisticated directory service than the ones we’ve considered thus far. There are two widely deployed versions of the LDAP protocol out there: version 2 and version 3. Anything that is version-specific will be clearly noted as such. This protocol is the industry standard for directory access. System administrators have embraced LDAP because it offers them a way to centralize and make available all sorts LDAP: A Sophisticated Directory Service | 321 of infrastructure information. Besides the standard “company directory” examples, applications include: • NIS-to-LDAP gateways • Authentication databases of all sorts (e.g., for use on the Web) • Resource advertisement (i.e., which machines and peripherals are available) LDAP is also the basis of other sophisticated directory services, such as Microsoft’s Active Directory (explored later, in the section “Active Directory Service Interfa- ces” on page 354). Even if your environment doesn’t use LDAP to provide anything but a fancy phone book, there are still good reasons to learn how to use the protocol. LDAP servers them- selves can be administered using the same protocol they serve, similar to SQL database servers being administered via SQL. To this end, Perl offers an excellent glue environ- ment for automating LDAP administrative tasks. Before we get there, though, it’s important that you understand LDAP itself. Appendix C contains a quick introduction to LDAP for the uninitiated. The biggest barrier new system administrators encounter when they begin to learn about LDAP is the unwieldy nomenclature it inherited from its parent protocol, the X.500 Directory Service. LDAP is a simplified version of part of X.500, but unfortunately the distillation process did not make the terminology any easier to swallow. Taking a few moments with Appendix C to get the language under your belt will make understanding how to use LDAP from Perl easier. LDAP Programming with Perl Like so many other system administration tasks in Perl, a good first step toward LDAP programming is the selection of the required Perl module. LDAP is not the most com- plex protocol out there, but it is not a plain-text protocol. As a result, cobbling together something that speaks LDAP is not a trivial exercise. Luckily, there are two modules available for this purpose: Net::LDAPapi (a.k.a. PerLDAP and Mozilla::LDAP) by Leif Hedstrom and Clayton Donley, and Graham Barr’s Net::LDAP. In the first edition of this book, the code examples used both modules. Since then, Net::LDAP has continued to evolve† while PerLDAP has suffered bitrot for about 10 years. Though you’ll occa- sionally see a piece of PerlLDAP code go by, at this point I can only recommend using Net::LDAP and will use it exclusively in the code we’re about to explore.‡ For demonstration servers, we’ll be using the commercial (formerly Sun One, formerly iPlanet, formerly Netscape) JES Directory Server and the free OpenLDAP server (found † Quanah Gibson-Mount recently (around January 2008) took over Net::LDAPapi and published the first update of the module to CPAN since 1998. ‡ As an aside, Donley, one of the original authors, himself uses Net::LDAP in his book LDAP Programming, Management and Integration (Manning). 322 | Chapter 9: Directory Services at and almost interchangeably. Both come with nearly identical command-line utilities that you can use to prototype and crosscheck your Perl code. The Initial LDAP Connection Connecting with authentication is the usual first step in any LDAP client/server trans- action. In LDAP-speak this is known as “binding to the server.” Binding to a server before sending commands to it was required in LDAPv2, but this requirement was relaxed for LDAPv3. When you bind to an LDAP server, you are said to be doing so in the context of a specific distinguished name (DN), described as the bind DN for that session. This is similar to logging in as a particular user on a multiuser system. On such a system, your current login (for the most part) determines your level of access to data on the system; with LDAP, it is the bind DN context that determines how much data on the LDAP server you can see and modify. There is also a special DN known as the root distinguished name (which is not given an acronym to avoid confusing it with the term “relative distinguished name”). The root distinguished name is the DN context that has total control over the whole tree; it’s similar to being logged in as root under Unix/ Mac OS X or Administrator on Windows. Some servers also refer to this as the manager DN. If a client provides no authentication information (e.g., DN and password) as part of a bind, or does not bother to bind before sending commands, this is known as anonymous binding. Anonymously bound clients typically receive very restricted access to a server’s data. There are two flavors of binding in the LDAPv3 specification: simple and SASL. Simple binding uses plain-text passwords for authentication. SASL (Simple Authentication and Security Layer) is an extensible authentication framework defined in RFC 2222 that allows client/server authors to plug in a number of different authentication schemes, such as Kerberos and one-time passwords. When a client connects to a server, it re- quests a particular authentication mechanism. If the server supports this mechanism, it will begin the challenge/response dialogue specific to that mechanism to authenticate the client. During this dialogue, the client and server may also negotiate a security layer (e.g., “all traffic between us will be encrypted using TLS”) for use after the initial au- thentication has been completed. Some LDAP servers and clients add one more authentication method to the standard simple and SASL choices. This method comes as a by-product of running LDAP over an encrypted channel via the Secure Sockets Layer (SSL) or its successor, Transport Layer Security (TLS). To set up this channel, LDAP servers and clients exchange public- key cryptography certificates just like a web server and browser do for HTTPS. Once the channel is in place, some LDAP servers can be told to use a trusted client’s certificate for authentication without having to bother with other authentication info. LDAP: A Sophisticated Directory Service | 323 There are two ways to handle SSL/TLS connections. In LDAPv2 days, some servers started to provide SSL-encrypted connections on a separate port designated for this purpose (port 636). A client could connect to this special port and immediately nego- tiate an SSL connection before performing any LDAP operations. This is often referred to as LDAPS, just like the HTTP/HTTPS analogue. However, HTTPS differs from LDAPS in one very important respect: LDAPS isn’t part of the LDAP specifications and hence isn’t a “real” protocol, even though quite a few servers still implement it. RFC 2830 defines the real extension to the LDAPv3 protocol for this purpose. In LDAPv3, clients can connect to the standard LDAP port (port 389) and request an encrypted connection by making a Start TLS request. Servers that implement this ex- tension to the protocol (most do at this point) will then begin the process of negotiating a TLS-encrypted connection through which the normal authentication and other LDAP requests will take place. To keep our examples from getting too complicated, we’ll stick to simple authentica- tion and unencrypted transport sessions in everything but the upcoming sidebar on this topic. Here’s how you do a simple bind and unbind in Perl: use Net::LDAP; # create a Net::LDAP object and connect to server my $c = Net::LDAP->new($server, port => $port) or die "Unable to connect to $server: $@\n"; # use no parameters to bind() for anonymous bind # $binddn is presumably set to something like: # "uid=bucky,ou=people,dc=example,dc=edu" my $mesg = $c->bind($binddn, password => $passwd); if ($mesg->code){ die 'Unable to bind: ' . $mesg->error . "\n"; } ... $c->unbind(); # not strictly necessary, but polite All Net::LDAP methods—e.g., bind()—return a message response object. When we call that object’s code() method it will return the result code of the last operation. The result code for a successful operation (LDAP_SUCCESS) is 0, hence the test in the preceding code. Using Encryption for LDAP Communications Given the wild and woolly nature of today’s network life, it would be irresponsible of me not to show you how to encrypt your LDAP communications (either the initial authentication or subsequent operations). Luckily, the simple methods are pretty easy. 324 | Chapter 9: Directory Services First, you have to determine what encryption methods the server you are using imple- ments. The choices are (in order of decreasing preference): 1. Start TLS 2. LDAPS 3. SASL You may be surprised that I listed SASL last, so let’s get that question out of the way first. Yes, SASL is the most flexible of the methods available, but it also requires the most work on your part. The most common reason to use SASL is for times when Kerberos (via the GSSAPI mechanism in SASL*) is used as the authentication source. Another scenario would be for server configurations that don’t require encryption for simple queries (e.g., a company directory), but require them for operations where the information will be updated (e.g., updating your own record). In that case they might use SASL since simple binds are performed in clear text. Other uses exist but are rela- tively rare. It is much more common to use the first two choices in my list: Start TLS and LDAPS. These are both easy from Net::LDAP: • For Start TLS, call the start_tls() method after you use new() but before making a bind() call. • For LDAPS, either use the Net::LDAPS module and add additional certificate- related parameters to new(), or use the normal Net::LDAP module but feed an ldaps:// URI to new() along with additional certificate-related parameters. Performing LDAP Searches The D in LDAP stands for Directory, and the one operation you’ll perform most often on a directory is a search. Let’s start our exploration of LDAP functionality by looking at how to find information. An LDAP search is specified in terms of: Where to begin the search This is called the base DN or search base. A base DN is simply the DN of the entry in the directory tree where the search should begin. Where to look This is known as the search scope. The scope can be either base (search just the base DN), one (search everything one level below the base DN, not including the base DN itself), or sub (search the base DN and all of the parts of the tree below it). * For generic Kerberos authentication, the Authen::SASL package (plus its dependent modules) by Graham Barr works fine. If you need to do anything funky like connect to an Active Directory server explicitly authenticated by Kerberos, you’ll probably need to use Mark Adamson’s hooks into the Cyrus-SASL libraries (Authen::SASL::Cyrus). This module has some issues, so be sure to look at the Net::LDAP mailing list archives before you head down that twisted path. LDAP: A Sophisticated Directory Service | 325 What to look for This is called the search filter. We’ll discuss filters and how they are specified in just a moment. What to return To speed up the search operation, you can select which attributes the search filter returns for each entry it finds. It is also possible to request that the search filter only return attribute names and not their values. This is useful for those times when you want to know which entries have a certain attribute, but you don’t care what value that attribute contains. Be Prepared to Carefully Quote Attribute Values A quick tip before we do any more Perl programming: if you have an attribute in your relative distinguished name with a value that contains one of the characters “+”, “(space),” “,”, “‘”, “>”, “<”, or “;”, you must specify the value surrounded by quotation marks or with the offending character escaped by a backslash (\). If the value contains quotation marks, those marks must be escaped using backslashes. Backslashes in values are also escaped with more backslashes. Later versions of Net::LDAP::Util (0.32+) have an escape_dn_value() function to help you with this. Insufficient quoting will bite you if you are not careful (of course, avoiding these char- acters all together in your directory’s RDNs wouldn’t hurt either). In Perl, a search looks like this:† ... my $searchobj = $c->search(base => $basedn, scope => $scope, filter => $filter); die 'Bad search: ' . $searchobj->error() if $searchobj->code(); Let’s talk about the mysterious $filter parameter before we get into a fully fleshed- out code example. Simple search filters are of the form:‡ where is specified in RFC 2254 as one of the operators listed in Table 9-1. † Because we do it exactly the same way each time, and to save space, the module load, creation of the connection object, and bind steps have been replaced with an ellipsis in this and later code examples. ‡ Filters also have restrictions on the characters that can be used without special handling. escape_filter_value() in version 0.32+ of Net::LDAP::Util can help with this. 326 | Chapter 9: Directory Services Table 9-1. LDAP comparison operators Operator Means = Exact value match. Can also be a partial value match if * is used in the specification (e.g., cn=Tim O*). =* Match all entries that have values for , independent of what the values are. By specifying * instead of , we test for the presence of that particular attribute in an entry (e.g., cn=* would select entries that have cn attributes). ~= Approximate value match. >= Greater than or equal to value. <= Less than or equal to value. Before you get excited because these look like Perl operators, I have bad news: they have nothing to do with the Perl operators. Two misleading constructs to a Perl person are ~= and =*. The first has nothing to do with regular expression matches; instead, it finds matches that approximate the stated value. The definition of “approximate” in this case is server-dependent. Most servers use an algorithm called soundex, originally invented for census taking, to determine the matching values. It attempts to find words that “sound like” the given value (in English) but are spelled differently.* The other construct that may clash with your Perl knowledge is the = operator. In addition to testing for exact value matches (both string and numeric), = can also be used with prefix and suffix asterisks as wildcard characters, similar to shell globbing. For example, cn=fi* will yield all of the entries that have a common name that begins with the letters “fi”. cn=*ink* likewise performs just as you would suspect, finding each entry whose common name attribute has the letters “ink” in it. We can take two or more of these simple search forms and string them together with Boolean operators to make a more complex filter. This takes the form: ( () () () ... ) People with LISP experience will have no problem with this sort of syntax; everyone else will just have to remember that the operator that combines the simple search forms is written first. To filter entries that match both criteria A and B, you would use (&(A)(B)). For entries that match criteria A or B or C, you would use (|(A)(B)(C)). The exclamation mark negates a specific criterion: A and not B is written (&(A)(!(B))). Compound filters can be compounded themselves to make arbitrarily complex search filters. Here is an example of a compound search filter that finds all of the Finkelsteins who work in Boston: (&(sn=Finkelstein)(l=Boston)) * If you want to play with the soundex algorithm, Mark Mielke’s Text::Soundex module provides a Perl implementation. LDAP: A Sophisticated Directory Service | 327 To find anyone with the last name Finkelstein or Hodgkin: (|(sn=Finkelstein)(sn=Hodgkin)) To find all of the Finkelsteins who do not work in Boston: (&(sn=Finkelstein)(!(l=Boston))) To find all the Finkelsteins or Hodgkins who do not work in Boston: (&(|(sn=Finkelstein)(sn=Hodgkin))(!(l=Boston))) Here are two code examples that take an LDAP server name and an LDAP filter and return the results of the query: use Net::LDAP; use Net::LDAP::LDIF; my $server = $ARGV[0]; my $port = getservbyname('ldap','tcp') || '389'; my $basedn = 'c=US'; my $scope = 'sub'; # anonymous bind my $c = Net::LDAP->new($server, port=>$port) or die "Unable to connect to $server: $@\n"; my $mesg = $c->bind(); if ($mesg->code){ die 'Unable to bind: ' . $mesg->error . "\n"; } my $searchobj = $c->search(base => $basedn, scope => $scope, filter => $ARGV[1]); die "Bad search: " . $searchobj->error() if $searchobj->code(); # print the return values from search() found in our $searchobj if ($searchobj){ my $ldif = Net::LDAP::LDIF->new('-', 'w'); $ldif->write_entry($searchobj->entries()); $ldif->done(); } Here’s an excerpt from some sample output: $ ldapsrch '(sn=Pooh)' ... dn: cn="bear pooh",,c=US, mail: cn: bear pooh o: givenname: bear surname: pooh ... Before we develop this example any further, let’s explore the code that processes the results returned by search(). You may be wondering what all of that Net::LDAP::LDIF 328 | Chapter 9: Directory Services stuff was. This is a sneak peek at a format called LDAP Data Interchange Format, or LDIF. Hang on for just a couple more sections and we’ll talk about LDIF in detail. More interesting at the moment is that innocuous call to $searchobj->entries(). Net::LDAP’s programming model resembles the protocol definition of RFC 2251. LDAP search results are returned in LDAP Message objects. The code we just saw calls the entries() method to return a list of all of the entries returned in these packets. We then use a method from the adjunct module Net::LDAP::LDIF to dump out these entries en masse. Let’s tweak our previous example a little bit. Earlier in this chapter I mentioned that we could construct speedier searches by limiting the attributes that are returned by a search. With the Net::LDAP module, this is as simple as adding an extra parameter to our search() method call: ... # could also add "typesonly => 1" to return just attribute types #(i.e., no values at all) my @attr = qw( sn cn ); my $searchobj = $c->search(base => $basedn, scope => $scope, filter => $ARGV[1], attrs => \@attr); Note that Net::LDAP takes a reference to an array for that additional argument, not values in the array. Entry Representation in Perl These code samples may provoke some questions about entry representation and ma- nipulation—for example, how are entries themselves stored and manipulated in a Perl program? I’ll answer a few of those questions as a follow-up to our LDAP searching discussion here and then provide a more in-depth exploration in the upcoming sections on addition and modification of entries. After you conduct a search with Net::LDAP, all of the results are available encapsulated by a single Net::LDAP::Search object. To get at the individual attributes for the entries in this object, you can take one of two approaches. First, you can ask the module to convert all of the returned entries (represented as Net::LDAP::Entry objects) into one large user-accessible data structure. $searchobj->as_struct() returns a hash-of-hash-of-lists data structure. That is, it re- turns a reference to a hash whose keys are the DNs of the returned entries. The values for these keys are references to anonymous hashes keyed on the attribute names. These keys yield references to anonymous arrays that hold the actual values for those attrib- utes. Figure 9-1 makes this clearer. LDAP: A Sophisticated Directory Service | 329 $entry uid l phones uid=rsmith,ou=system,ou=people,c=ccs,dc=hogwarts,dc=edu ref ref= rsmith[] ref ref= Boston[] ref ref= 617-555-1212,617-555-2121[] = Figure 9-1. Data structure returned by as_struct() To print the first value of the cn attribute for each entry in this data structure, you could use code like this: my $searchstruct = $searchobj->as_struct; foreach my $dn (keys %$searchstruct){ print $searchstruct->{$dn}{cn}[0],"\n"; } Alternatively, you can first use any one of these methods to unload an individual entry object from the object a search returns: # return a specific entry number my $entry = $searchobj->entry($entrynum); # acts like Perl shift() on entry list my $entry = $searchobj->shift_entry; # acts like Perl pop() on entry list my $entry = $searchobj->pop_entry; # return all of the entries as a list my @entries = $searchobj->entries; Once you have an entry object, you can use one of the method calls in Table 9-2. Table 9-2. Key Net::LDAP entry methods (see Net::LDAP::Entry for more) Method call Returns $entry->get_value($attrname) The value(s) of that attribute in the given entry. In a list context, returns all of the values. In a scalar context, returns just the first one. $entry->attributes() The list of attribute names for that entry. It is possible to chain these method calls together in a fairly legible fashion. For instance, this line of code will retrieve the first value of the cn attribute in the first returned entry: my $value = $searchobj->entry(1)->get_value('cn') Now that you know how to access individual attributes and values returned by a search, let’s look at how to get this sort of data into a directory server in the first place. 330 | Chapter 9: Directory Services Adding Entries with LDIF Before we get into the generic methods for adding entries to an LDAP directory, let’s look at a technique useful mostly to system and directory administrators. This techni- que uses a data format that helps you to bulk-load data into a directory server. We’re going to explore ways of writing and reading LDIF. LDIF, defined by Gordon Good in RFC 2849, offers a simple text representation of a directory entry. Here’s a simple LDIF example taken from that RFC: version: 1 dn: cn=Barbara Jensen, ou=Product Development, dc=airius, dc=com objectclass: top objectclass: person objectclass: organizationalPerson cn: Barbara Jensen cn: Barbara J Jensen cn: Babs Jensen sn: Jensen uid: bjensen telephonenumber: +1 408 555 1212 description: A big sailing fan. dn: cn=Bjorn Jensen, ou=Accounting, dc=airius, dc=com objectclass: top objectclass: person objectclass: organizationalPerson cn: Bjorn Jensen sn: Jensen telephonenumber: +1 408 555 1212 The format should be almost self-explanatory to you by now. After the LDIF version number, each entry’s DN, objectClass definitions, and attributes are listed. A line sep- arator alone on a line (i.e., a blank line) separates individual entries. Our first task is to learn how to write LDIF files from extant directory entries. In addition to giving us practice data for the next section (where we’ll read LDIF files), this func- tionality is useful because once we have an LDIF file, we can massage it any way we like using Perl’s usual text-manipulation idioms. LDIF has a few twists (e.g., how it handles special characters and long lines), so it is a good idea to use Net::LDAP::LDIF to handle the production and parsing of your LDIF data whenever possible. You already saw how to print out entries in LDIF format, during our discussion of LDAP searches. Let’s change the code we used in that example so it writes to a file. Instead of using this line: my $ldif = Net::LDAP::LDIF->new('-', 'w'); we use: my $ldif = Net::LDAP::LDIF->new($filename, 'w'); to print the output to the specified filename instead of the standard output channel. LDAP: A Sophisticated Directory Service | 331 Let’s work in the opposite direction now, reading LDIF files instead of writing them. The module object methods we’re about to explore will allow us to easily add entries to a directory.† When you read in LDIF data via Perl, the process is exactly the reverse of what we used in the previous LDIF-writing examples. Each entry listing in the data gets read in and converted to an entry object instance that is later fed to the appropriate directory mod- ification method. Net::LDAP handles the data reading and parsing for you, so this is a relatively painless process. In the following examples, we’re using the root or manager DN user context for demonstration purposes. In general, if you can avoid using this context for everyday work, you should. Good practice for setting up an LDAP server includes creating a powerful account or account group (which is not the root DN) for directory management. Keep this security tip in mind as you code your own applications. With Net::LDAP, the LDIF entry addition code is easier to write: use Net::LDAP; use Net::LDAP::LDIF; my $server = $ARGV[0]; my $LDIFfile = $ARGV[1]; my $port = getservbyname('ldap','tcp') || '389'; my $rootdn = 'cn=Manager, ou=Systems, dc=ccis, dc=hogwarts, dc=edu'; my $pw = 'secret'; # read in the LDIF file specified as the second argument on the command line; # last parameter is "r" for open for read, "w" would be used for write my $ldif = Net::LDAP::LDIF->new($LDIFfile,'r'); # copied from the deprecated read() command in Net::LDAP::LDIF my ($entry,@entries); push(@entries,$entry) while $entry = $ldif->read_entry; my $c = Net::LDAP-> new($server, port => $port) or die "Unable to connect to $server: $@\n"; my $mesg = $c->bind(dn => $rootdn, password => $pw); if ($mesg->code){ die 'Unable to bind: ' . $mesg->error . "\n"; } for (@entries){ my $res = $c->add($_); warn 'Error in add for '. $_->dn().': ' . $res->error()."\n" if $res->code(); † LDIF files can also contain a special changetype: directive that instructs the LDIF reader to delete or modify entry information rather than just adding it. Net::LDAP has direct support for changetype: via its Net::LDAP::LDIF::read_entry() method. 332 | Chapter 9: Directory Services } $c->unbind(); Adding Entries with Standard LDAP Operations It’s time to look under the hood of the entry addition process, so we can see how to create and populate entries manually, instead of just reading them from a file like we did in the last subsection. Net::LDAP supports two ways to go about creating entries in a directory. Feel free to choose the one that feels the most comfortable to you. If you are used to working with Perl data structures and like your programming to be terse and to the point, you can feed the add() method a naked data structure for single- step entry addition: my $res = $c->add( dn => 'uid=jay, ou=systems, ou=people, dc=ccis, dc=hogwarts, dc=edu', attr => ['cn' => 'Jay Sekora', 'sn' => 'Sekora', 'mail' => '', 'title' => ['Sysadmin','Part-time Lecturer'], 'uid' => 'jayguy', 'objectClass' => [qw(top person organizationalPerson inetOrgPerson)]] ); die 'Error in add: ' . $res->error()."\n" if $res->code(); Here, we’re passing two arguments to add(): the first is a DN for the entry, and the second is a reference to an anonymous array of attribute/value pairs. You’ll notice that multivalued attributes like title are specified using a nested anonymous array. If you’d prefer to take things one step at a time, you can construct a new Net::LDAP::Entry object and feed that object to add() instead: use Net::LDAP; use Net::LDAP::Entry; ... my $entry = Net::LDAP::Entry->new; $entry->dn( 'uid=jayguy, ou=systems, ou=people, dc=ccs, dc=hogwarts, dc=edu'); # these add statements could be collapsed into a single add() $entry->add('cn' => 'Jay Sekora'); $entry->add('sn' => 'Sekora'); $entry->add('mail' => ''); $entry->add('title' => ['Sysadmin','Part-time Lecturer']); $entry->add('uid' => 'jayguy'); $entry->add('objectClass' => [qw(top person organizationalPerson inetOrgPerson)]); # we could also call $entry->update($c) instead # of add() if we felt like it my $res = $c->add($entry); die 'Error in add: ' . $res->error()."\n" if $res->code(); LDAP: A Sophisticated Directory Service | 333 One thing that may be a bit confusing in this last example is the double use of the method name add(). There are two very different method calls being made here that unfortunately have the same name. The first is from a Net::LDAP::Entry object ($entry->add())—this adds new attributes and their values to an existing entry. The second, $c->add($entry), is a method call to our Net::LDAP connection object asking it to add our newly constructed Net::LDAP::Entry object to the directory. If you pay at- tention to the piece of the call before the arrow, you’ll be fine. If this double use of the same name bothers you too much, you could replace the second add() call with a Net::LDAP::Entry update() call, as mentioned in the final code comment. Deleting Entries Deleting entries from a directory is easy (and irrevocable, so be careful). Here is a code snippet, again with the bind code left out for brevity’s sake: ... my $res = $c->delete($dn); die 'Error in delete: ' . $res->error() . "\n" if $res->code(); It is important to note that delete() operates on a single entry at a time. With most servers, if you want to delete an entire subtree, you will need to first search for all of the child entries of that subtree using a scope of sub or one and then iterate through the return values, deleting as you go; once you’ve deleted all the children, you can remove the top of that subtree. However, the following sidebar details a few shortcuts that may work for you. Deleting an Entire Directory Subtree As of this writing, the somewhat laborious process described in the text for deleting a whole subtree from a directory is the correct canonical method for performing that task. There are a couple of easier approaches you can take in some cases, though: • Use someone else’s code—OpenLDAP ships with a command-line tool called ldapdelete that has a –r option for recursive deletions. No, you won’t get closer to your Net::LDAP merit badge by calling another executable from your program, but it does make the code considerably easier to write. • Use an unofficial LDAP control—we haven’t talked about LDAP controls yet in this chapter (we’ll get to them in another few sections), so for the moment feel free to treat the following code snippet as a magic incantation for deleting whole subtrees: my $res = $ldap->delete($dn, control => {type => LDAP_CONTROL_TREE_DELETE}); There are two complications with using this code. First, it uses a control that was proposed as a standard but never made it to the RFC stage (the last version was draft-armijo-ldap-treedelete-02.txt), and hence is “unofficial.” Second, most LDAP 334 | Chapter 9: Directory Services servers don’t implement it. I wouldn’t mention it except that the one notable exception that does implement it is Active Directory. Caveat implementor. Modifying Entry Names For our final look at LDAP operations, we will focus on two kinds of modifications to LDAP entries. The first kind of modification we’ll consider is a change of DN or RDN. Here’s an example of the Net::LDAP code used to change the relative distinguished name for an entry: # $oldDN could be something like # "uid=johnny,ou=people,dc=example,dc=edu" # $newRDN could be something like # "uid=pedro" my $res = $c->moddn($oldDN, newrdn => $newRDN, deleteoldrdn => 1); die 'Error in rename: ' . $res->error()."\n" if $res->code(); Here’s a quick review, in case you’re fuzzy on this RDN concept. LDAP servers store their entries in a tree-like form. But unlike some tree-based protocols (e.g., SNMP, which we’ll see later), LDAP doesn’t let you pick a specific entry out of the tree by its numeric position. For instance, you can’t just say “give me the third entry found in the fourth branch to the left.” Instead, you need to identify a unique path to that entry. LDAP makes it easy to find a unique path by dictating that at each level of the tree an individual entry must have something that sets it apart from every other entry at that level in the tree. Since every step of your path is guaranteed to be unique at that level, the whole path is guaranteed to be a unique way to locate a specific entry.‡ It is the RDN that keeps each entry unique at a particular level of the tree. This is why we’re fussing so much about this single line of code. When you change the RDN, you are changing the entry’s name at that level. The actual operation is pretty simple; it’s just important to understand what’s happening. Before we move on to the second kind of rename, there’s one small detail worth men- tioning. You probably noticed the deleteoldrdn parameter being set in the code and may have wondered about it. When we think about renaming objects in almost all contexts (e.g., filename changes), we don’t worry about what happens to the old name after the operation has been completed. If you rename a file, the file gets a new name, and the information about what the file was called is lost to the mists of time. With LDAP, you have a choice: ‡ There’s some moderate hand waving going on here because LDAP directory “trees” (snarky quotes provided for the benefit of readers with a computer science background) can have symlink-like aliases and other complications that make it possible to find an entry using two very different paths. This isn’t a problem for the discussion at hand, but it’s worth noting to keep ourselves honest. LDAP: A Sophisticated Directory Service | 335 • You can change the RDN and toss the old RDN information (deleteoldrdn => 1). This is almost always the right choice. • You can change the RDN and keep the old RDN information as an additional value in the entry (deleteoldrdn => 0). Don’t do this unless you have a good reason. Since this is so weird, here’s a quick example that will make it clear. Let’s assume we start off with an entry that looks in part like this: dn: uid=gmarx, ou=People, dc=freedonia, dc=com cn: Julius Henry Marx sn: Marx uid: gmarx If we execute code that includes these lines: my $oldDN = "uid=gmarx, ou=People, dc=freedonia, dc=com"; my $newRDN = "uid=cspaulding"; my $res = $c->moddn($oldDN, newrdn => $newRDN, deleteoldrdn => 1); the entry will look like this: dn: uid=cspaulding, ou=People, dc=freedonia, dc=com cn: Julius Henry Marx sn: Marx uid: cspaulding Nothing special here; it looks just like we’d expect. If we had run the same code with the last line changed to this: my $res = $c->moddn($oldDN, newrdn => $newRDN, deleteoldrdn => 0); the entry would look like this: dn: uid=cspaulding, ou=People, dc=freedonia, dc=com cn: Julius Henry Marx sn: Marx uid: gmarx uid: cspaulding That’s clearly not what we want. As mentioned earlier, you’ll almost always* want to set deleteoldrdn to 1. Time to move on. The second kind of entry name modification is the more drastic one. To move an entry to a different spot in the directory tree, you need to change its distinguished name. Version 3 of LDAP introduces a more powerful renaming operation that allows arbi- trary entry relocations within the directory tree hierarchy. Net::LDAP ’s moddn() function gives us access to that when called with the additional parameter newsuperior. If we add it like so: * To keep you from spending all day racking your brain looking for a case where you would want to keep the old RDN, here’s one idea: imagine you were changing all of your usernames due to a company merger and you wanted the ability to look up users by their old names after the renaming. There are better ways to implement this, but you asked.... 336 | Chapter 9: Directory Services # $oldDN could be something like # "uid=johnny,ou=people,dc=example,dc=edu" # $newRDN could be something like # "uid=pedro" # $parenDN could be something like # ou=boxdweller, dc=example,dc=edu $result = $c->moddn($oldDN, newrdn => $newRDN, deleteoldrdn => 1, newsuperior => $parentDN); die 'Error in rename: ' . $res->error()."\n" if $res->code(); the entry located at $oldDN will be moved to become the child of the DN specified in $parentDN. Using this method to move entries in a directory tree is more efficient than the add() or delete() sequence previously required by the protocol, but it is not sup- ported by all LDAP servers. Other server-dependent caveats may be applicable here as well: for example, the server you are using may not allow you to modify the DN of an entry that has children. In any case, if you’ve carefully designed your directory tree structure, you’ll hopefully have to relocate entries less often. Modifying Entry Attributes Let’s move on to the more common operation of modifying the attributes and attribute values in an entry. We’ll start with an example of this process as part of a global search- and-replace. Here’s the scenario: one of the facilities at your company is moving from Pittsburgh to Los Angeles. This code will change all of the entries with a Pittsburgh location: use Net::LDAP; my $server = $ARGV[0]; my $port = getservbyname('ldap','tcp') || '389'; my $basedn = 'dc=ccis,dc=hogwarts,dc=edu'; my $scope = 'sub'; my $rootdn = 'cn=Manager, ou=Systems, dc=ccis, dc=hogwarts, dc=edu'; my $pw = 'secret'; my $c = Net::LDAP->new($server, port => $port) or die "Unable to init for $server: $@\n"; my $mesg = $c->bind(dn => $rootdn, password => $pw); if ($mesg->code){ die 'Unable to bind: ' . $mesg->error . "\n"; } my $searchobj = $c->search(base => $basedn, filter => '(l=Pittsburgh)', scope => $scope, attrs => [''], typesonly => 1); die 'Error in search: '.$searchobj->error()."\n" if ($searchobj->code()); if ($searchobj){ @entries = $searchobj->entries; for (@entries){ LDAP: A Sophisticated Directory Service | 337 # we could also use replace {'l' => 'Los Angeles'} here $res=$c->modify($_->dn(), # dn() yields the DN of that entry delete => {'l' => 'Pittsburgh'}, add => {'l' => 'Los Angeles'}); die 'unable to modify, errorcode #'.$res->error() if $res->code(); } } $c->unbind( ); The crucial part of this code is the use of the mega-method called modify(), toward the end of the example. modify() takes the DN of the entry to be changed and a set of parameters that tells it just how to modify that entry. Table 9-3 lists the possible choices. Table 9-3. Net::LDAP entry modification methods Parameter Effect add => {$attrname => $attrvalue} Adds a named attribute with the given value. add => {$attrname => [$attrvalue1, $attrvalue2...]} Adds a named attribute with the specified set of values. delete => {$attrname => $attrvalue} Deletes a named attribute with the specified value. delete => {$attrname => []} delete => [$attrname1, $attrname2...] Deletes an attribute or set of attributes independent of their value or values. replace => {$attrname => $attrvalue} Like add, but replaces the current named attribute value. If $attrvalue is a reference to an empty anonymous list ([]), this becomes a synonym for the delete operation. Be sure to pay attention to the punctuation in Table 9-3. Some parameters call for a reference to an anonymous hash, while others call for a reference to an anonymous array. Mixing the two will cause problems. If you find yourself needing to make several changes to an entry, as we did in our code example, you can combine several of these parameters in the same call to modify(). However, there’s a potential problem lurking here. When you call modify() with a set of parameters, like so: $c->modify($dn,replace => {'l' => 'Medford'}, add => {'l' => 'Boston'}, add => {'l' => 'Cambridge'}); there’s no guarantee that the additions you specify will take place after the replacement. This code could have an unpredictable, if not downright unpleasant, result. 338 | Chapter 9: Directory Services If you need your operations to take place in a specific order, you’ll need to use a slight twist on the normal syntax. Instead of using a set of discrete parameters, pass in a single array containing a queue of commands. In this version, modify() takes a changes pa- rameter whose value is a list. This list is treated as a set of pairs: the first half of the pair is the operation to be performed, and the second half is a reference to an anonymous array of data for that operation. For instance, if we wanted to ensure that the operations in the previous code snippet happened in order, we could write: $c->modify($dn, changes => [ replace => ['l' => 'Medford'], add => ['l' => 'Boston'], add => ['l' => 'Cambridge'] ]); Take careful note of the punctuation: it is different from that in the earlier examples. Deeper LDAP Topics If you’ve read up to this point and things are starting to make sense to you, you’ve got all the basic skills for using LDAP from Perl ready to roll. If you’re chomping at the bit to see how this is all put together in some more complete examples, you can skip to the next section, “Putting It All Together” on page 348, and come back here when you’re done. If you can hold on for a little while longer, in this section we’ll touch on a few advanced topics to give you a really thorough grounding in this stuff. Referrals and references The hardest part of understanding LDAP referrals and references is simply keeping the two of them distinct in your memory. In LDAPv2, referrals were pretty simple (so simple, in fact, that they really didn’t exist in the spec). If you asked an LDAPv2 server for data it didn’t have, the server could return a default referral that said, “I don’t know anything about that. Why don’t you go look over here at this LDAP URL instead?” An LDAP client could then use that URL (whose format is defined in RFC 2255) to deter- mine the name of the server to query and the base DN. For example, if your LDAP client asked the server responsible for ou=sharks,dc=jeromerobbins,dc=org about ou=jets,dc=jeromerobbins,dc=org, it could return a response that said “Sorry, ask ldap:://,dc=robertwise,dc=org instead.” Your client could then attempt to connect to the LDAP server running on and retry its query. LDAPv3 made this concept a little more complex, by adding the LDAPv2 behavior to the spec and expanding upon it. Now, when a server is queried for data it knows it doesn’t have, it can return a response like “Sorry, never heard of that. Why don’t you check over yonder at this URL or set of URLs?” The client is then free to choose for itself which URL to follow to get its information. LDAP: A Sophisticated Directory Service | 339 The second enhancement to the referral concept in LDAPv3 came in the form of continuation references. Continuation references are a type of referral (see, I told you it was hard to keep the two things straight!†) that only comes into play during an LDAP search. If the server realizes during a search that more information for that search could be found at another server, it is free to return a continuation reference along with the other entries being returned. This continuation reference says, “Hey, I couldn’t answer the entire question, but I think I know who can. Try asking over at this URL (or set of URLs) for the rest of your data.” It is then up to the client to query all of those additional servers to complete its query. Continuation references usually come into play when dealing with a very large directory tree where parts of the tree have been split onto multiple servers for load management. Let’s see how all this manifests itself in Perl code. Though they are related, we’ll examine referrals and continuation references separately. To deal with referrals, here are the steps: 1. When the operation has completed, check to see if we’ve received any referrals. If not, just proceed. 2. If we did receive a referral, extract an LDAP URL‡ from the response and dissect it into its component parts. 3. Bind to the appropriate server based on this information and query it. Go back to step 1 (since we might have received another referral). The code for these steps is pretty easy: 1. Check for a referral: use Net::LDAP qw(LDAP_REFERRAL); # be sure to import this constant use URI::LDAP; # going to use this to dissect our LDAP URL # bind as usual ... # perform a search as usual my $searchobj = $c->search(...); # check if we've received a referral if ($searchobj->code() == LDAP_REFERRAL) { 2. Extract an LDAP URL: # the return code indicates we have referrals, so retrieve all of them my @referrals = $searchobj->referrals(); † To make it easier for you to remember the difference between referrals and references, I’ll always refer to references as “continuation references.” ‡ RFC 2251, the LDAPv3 spec, says that while multiple URLs can be returned as part of the referral process, “All the URLs MUST be equally capable of being used to progress the operation.” This means you get to choose which one to follow. The level of difficulty of your strategy for making that choice can be low (pick the first one, pick a random one), medium (pick the one with the shortest ping time), or high (pick the closest one in your network topology). It’s your call. 340 | Chapter 9: Directory Services # RFC 2251 says we can choose any of them - let's pick the first one my $uri = URI->new($referrals[0]); 3. Bind and query again using the new info (dissecting the URL we received as nec- essary with URI::LDAP method calls): $c->unbind(); my $c = Net::LDAP-> new ($uri->host(), port => $uri->port()) or die 'Unable to init for ' . $uri->$host . ": $@\n"; my $mesg = $c->bind(dn => $rootdn, password => $pw); if ($mesg->code){ die 'Unable to bind: ' . $mesg->error . "\n"; } # RFC 2251 says we must use the filter in the referral URL if one # is returned; otherwise, we should use the original filter # # Note: we're using $uri->_filter() instead of just $uri->filter() # because the latter returns a default string when no filter is # present in the URL. We want to use our original filter in that case # instead of the default of (objectClass=*). $searchobj = $c->search(base => $uri->dn(), scope => $scope, filter => $uri->_filter() ? $uri->_filter() : $filter, ...); } You may find it easier to think about referral processing as just sophisticated error handling (because that is essentially what it is). You query a server and it replies, “Sorry, can’t handle your request. Please try again, but this time, try again at this server on this port with this baseDN and filter.” It is important to note that the preceding code isn’t as sophisticated or as rigorous as it could be. The first flaw is that, while RFC 2251 states that almost all LDAP operations can return a referral, the code only checks for this condition after the search operation (not after the initial bind). I would recommend that you sit down and have a good long think before you decide to follow referrals from bind operations, even if the spec says you should. If you are going to present your authentication credentials to some other server besides the one you originally intended, be sure you completely trust both servers (perhaps by checking the server certificates) first. Similar dire warnings apply to fol- lowing referrals during the other LDAP operations. The second flaw is that there’s nothing (besides good directory architecture practices) stopping the second server you query from handing you back another reference for you to chase. It is highly inefficient to keep a client hopping from server to server, so you shouldn’t see this in the real world, but it is possible. And finally, in the same category of “you shouldn’t see this,” the code doesn’t check for referral loops where server A says to go talk to server B, which sends you back to server A. It is easy to keep a list of the servers you’ve contacted to avoid this issue if you think it may happen for some reason. Caveat implementor. LDAP: A Sophisticated Directory Service | 341 Now that you have referrals under your belt, let’s move on to continuation references. Continuation references are marginally easier to deal with; they occur only during a search operation and they come into play only if a search can successfully begin (i.e., if the place you’ve asked to start searching from really exists in the tree). Unlike the referrals we just talked about, receiving a continuation reference is not an error condi- tion that requires restarting the whole operation. Continuation references are more like outstanding IOUs to a dull-witted debt collector. If your program were the debt col- lector, it would ask a server for information it felt entitled to have, and the server might say, “I’m sorry, I can’t make the entire payment (of LDAP entries you are looking for), but you can get the rest by asking at these three places....” Instead of trying to collect the whole amount from a single other server (as with a referral), your program will dutifully trudge off and try to get the rest of the information from all the additional sources. Those sources are, unfortunately, allowed to send you on a further chase to other places as well. From a coding perspective, the difference between continuation references and referrals is twofold: 1. The methods for determining whether a referral or a continuation reference is in play are very different. For a referral, we check the result code of an operation and then call the referrals() method. For a continuation reference, we examine the data we receive back from the server and then call the references() method if we find a continuation reference: ... # bind and search have taken place if ($searchobj){ my @returndata = $searchobj->entries; foreach my $entry (@returndata){ if ($entry->isa('Net::LDAP::Reference'){ # @references is a list of LDAP URLs push(@references,$entry->references()); } } } 2. Unlike with referrals, where we have a choice for which URLs to follow, we’re supposed to follow all continuation references. Most people code this using a recursive subroutine* along the lines of: ... # assume a search has taken place that has yielded continuation # references foreach my $reference (@references){ ChaseReference($reference) } sub ChaseReference ($reference){ my $reference = shift; * For a refresher on recursion, see Chapter 2. 342 | Chapter 9: Directory Services # this code should look very familiar because we stole it almost # verbatim from the previous example on referrals # dissect the LDAP URL, bind to the indicated server, and search it my $uri = URI->new($reference); my $c = Net::LDAP-> new ($uri->host(), port => $uri->port()) or die 'Unable to init for ' . $uri->$host . ": $@\n"; my $mesg = $c->bind(dn => $rootdn, password => $pw); if ($mesg->code){ die 'Unable to bind: ' . $mesg->error . "\n"; } my $searchobj = $c->search(base => $uri->dn(), scope => $scope, filter => $uri->_filter() ? $uri->_filter() : $filter, ...); # assuming we got a result, collect the entries and the references into # different lists if ($searchobj){ my @returndata = $searchobj->entries; my @references = (); foreach my $entry (@returndata){ if ($entry->isa('Net::LDAP::Reference'){ # @references will contain a list of LDAP URLs push(@references,$entry->references()); } else { push @entries, $entry ); } } # now, chase any more references we received from that last search # (here's the recursion part) foreach my $reference (@references){ ChaseReference($reference) } } Now, if you wanted to be a troublemaker, you might ask whether any of the operations in this code could return referrals, and whether the code should be handling these cases. “Yes” and “Yes.” Next question? Seriously though, the code presented so far on this topic has been intentionally kept as simple as possible to help explain the concepts and keep referrals and continuation references distinct in your mind. If you wanted to write the most robust code possible to handle these cases, you’d probably need to write wrapper subroutines around each LDAP operation that are prepared to handle referrals and deal with continuation ref- erences during searches. Controls and extensions The best explanation I’ve ever heard for LDAP controls comes from Gerald Carter’s book LDAP System Administration (O’Reilly). Carter described them as “adverbs” for LDAP: A Sophisticated Directory Service | 343 LDAP operations: they modify, change, or enhance an ordinary LDAP operation. For example, if you wanted a server to pre-sort the results of a search, you would use the Server Side Sorting control, as documented by RFC 2891. Let’s look at some code that presumes the server supports this control (not all do—for example, the Sun JES Directory Server does, but the OpenLDAP server does not). In most cases, the first step is to locate the Net::LDAP::Control subclass module for that particular control. All of the common controls have one.† In this case we’ll be using Net::LDAP::Control::Sort. Using this module, we create a control object: use Net::LDAP; use Net::LDAP::Control::Sort; ... # create a control object that will ask to sort by surname $control = Net::LDAP::Control::Sort->new(order => 'sn'); Once we have the control object, it is trivial to use it to modify a search: # this should return back the entries in a sorted order $searchobj= $c->search (base => $base, scope => $scope, filter => $filter, control => [$control]); Some controls require more effort than others to use, but now you have the basic idea. Extensions (also called “extended operations” in some contexts) are like controls, only more powerful. Instead of modifying a basic LDAP operation, they actually allow for extending the basic LDAP protocol to include entirely new operations. Examples of new operations added to the LDAP world through this mechanism include Start TLS (RFC 2830) for secure transmission of LDAP data and LDAP Password Modify (RFC 3062) for changing passwords stored on an LDAP server. Using extensions from Perl is usually a very simple affair, because all of the common extensions exist in their own module as part of Net::LDAP. For example, using Password Modify is this easy: use Net::LDAP; use Net::LDAP::Extension::SetPassword; ... # usual connection and bind here $res = $c->set_password( user => $username, oldpassword => $oldpw, newpassword => $newpw, ); die 'Error in password change : ' . $res->error()."\n" if $res->code(); † If you get unlucky and can’t find one for the control you want to use, it’s not hard to roll your own. The controls included with Net::LDAP should provide enough examples to get you all or most of the way there. 344 | Chapter 9: Directory Services If you need to use an extension that isn’t already implemented in the package, then your best bet is to cheat by copying a module file such as Net::LDAP::Extension::Set Password and modifying it accordingly. One question you may have had while reading this section is, “How do I know which controls and extensions are supported by the server I’m using?” Besides looking at the server’s documentation or source code (if available), you could also query the root DSE. That’s the subject of the next section. The root DSE The hardest thing about dealing with this topic is hacking through the overgrown ter- minology inherited from X.500 just to get to the actual meaning. Machete in hand, here’s how it goes: A DSE is a DSA-specific entry. What’s a DSA, you ask? A DSA is a directory system agent. What’s a directory system agent? A directory system agent is a server (an LDAP server, in this case). Besides the ability to impress all your friends at party with your command of X.500 terminology, why do you care about any of this? The root DSE is a special entry in a directory server that contains information about that server. If you interrogate the root DSE, RFC 2251 says you should be able to find the following attributes: namingContexts Which suffixes/directory trees (e.g., dc=ccis, dc=hogwarts, dc=edu) the server is ready to serve. subschemaSubentry The location in the directory where you can query the server’s schema (see Ap- pendix C for an explanation of LDAP schemas). altServer According to RFC 2251, a list of “alternative servers in case this one is later un- available” (this is stated without noting the irony). This information might come in handy if you were storing the data for future queries after your first contact with the server, but it still seems like the time you’ll be most interested in this informa- tion (i.e., during an outage) is the time when it is least accessible. supportedExtension The list of extensions this server can handle. supportedControl The list of controls that can be used with this server. supportedSASLMechanisms The list of available SASL mechanisms (e.g., Kerberos). supportedLDAPVersion Which LDAP versions the server is willing to speak (as of this writing, probably 2 and 3). LDAP: A Sophisticated Directory Service | 345 Getting this info from Perl is really easy. Net::LDAP has a Net::LDAP::RootDSE module that gets called like this: use Net::LDAP; use Net::LDAP::RootDSE; my $server = ''; my $c = Net::LDAP->new($server) or die "Unable to init for $server: $@\n"; my $dse = $c->root_dse(); # let's find out which suffixes can be found on this server print join("\n",$dse->get_value('namingContexts')),"\n"; This code returns something like this (i.e., a list of suffixes served from that server): dc=hogwarts,dc=edu o=NetscapeRoot You may have noticed we’re missing the usual “bind happens here” ellipsis we’ve seen in most of the code examples up until this point. That’s because Net::LDAP::RootDSE is actually arranging for an anonymous bind() followed by a search() to happen on our behalf behind the scenes. If you looked at the LDAP server log after this operation, you’d see what was really going on: [16/May/2004:21:25:46 −0400] conn=144 op=0 msgId=1 - SRCH base="" scope=0 filter="(objectClass=*)" attrs="subsch emaSubentry namingContexts altServer supportedExtension supportedControl supportedSASLMechanisms supportedLDAPVersion" This says we’re performing a search with a baseDN of an empty string (meaning the root DSE), a scope of 0 (which is “base”), a filter for anything in that entry, and a list of specific attributes to return. If you ever want to query this information for attributes in the root DSE not normally returned by Net::LDAP::RootDSE, now you know how to do it. DSML Our last advanced topic before we look at a small sample application is the Directory Services Markup Language (DSML). DSML comes in two flavors: version 1 and version 2. For our purposes, you can think of DSMLv1 as a slightly improved version of LDIF in XML. Acronym parking lot aside, this means that DSML represents entry data in XML instead of the LDIF format we learned about in “Adding Entries with LDIF” on page 331. It slightly improves on LDIF in this regard because it has an explicit standard for representing not just entries but also directory schemas (mentioned in Appendix C). That’s the good news. The bad news is that DMSLv1 can’t actually rep- resent directory operations like LDIF can (via changetype: delete). This deficiency was remedied in the more complex DSMLv2. As of this writing, the Perl world hasn’t caught up yet, so the only modules available specific to DSML are for version 1 only. 346 | Chapter 9: Directory Services However, if DSMLv1 is your bag, Net::LDAP::DSML offers a handy way to write DSMLv1- formatted files (though as of this writing, it can’t read them‡). The process is very similar to the one we used for writing LDIF: use Net::LDAP; use Net::LDAP::DSML; open my $OUTPUTFILE, '>', 'output.xml' or die "Can't open file to write:$!\n"; my $dsml = Net::LDAP::DSML->new(output => $OUTPUTFILE, pretty_print => 1 ) or die "OUTPUTFILE problem: $!\n"; ... # bind and search here to @entries $dsml->start_dsml(); foreach my $entry (@entries){ $dsml->write_entry($entry); } $dsml->end_dsml(); close $OUTPUTFILE; When we run this code (with the ellipsis replaced with real code), we get output like this (hand-indented for clarity): top organizationalunit Colin Johnson colinguy top person organizationalPerson inetorgperson ‡ If you want to read DSML, you can use any of the XML reading modules (e.g., XML::Simple), to read the data and then hand it to the Net::LDAP calls we saw in the section “Adding Entries with Standard LDAP Operations” on page 333. LDAP: A Sophisticated Directory Service | 347 This all begs the question, “Why use DSML instead of LDIF for entry representation?” It’s a reasonable question. DSML is meant to be an abstract representation of directory data (and directory operations, in version 2) in XML form. If you are doing lots of inter- organizational directory sharing, or you find a use for this abstraction, DSML might be right for you. But if you plan to stick to the LDAP arena and you don’t need the inter- operability XML provides, stick to LDIF. LDIF is (on the whole) simpler, well tested, and well supported by directory vendors. Putting It All Together Now that we’ve toured all of the major LDAP areas (and even some of the minor ones), let’s write some small system administration-related scripts. We’ll import our machine database from Chapter 5 into an LDAP server and then generate some useful output based on LDAP queries. Here are a couple of listings from that flat file, just to remind you of the format: name: shimmer address: aliases: shim shimmy shimmydoodles owner: David Davis department: software building: main room: 909 manufacturer: Sun model: M4000 -=- name: bendir address: aliases: ben bendoodles owner: Cindy Coltrane department: IT building: west room: 143 manufacturer: Apple model: Mac Pro -=- The first thing we need to do is prepare the directory server to receive this data. We’re going to use nonstandard attributes, so we’ll need to update the server’s schema. Dif- ferent servers handle this process in different ways. For instance, the Sun JES Directory Server has a pleasant Directory Server Console GUI for changing details like this. Other servers require modifications to a text configuration file. With OpenLDAP, we could use something like this in a file that the master configuration file includes to define our own object class for a machine: 348 | Chapter 9: Directory Services objectclass machine requires objectClass, cn allows address, aliases, owner, department, building, room, manufacturer, model Once we’ve configured the server properly, we can think about importing the data. One approach would be to bulk load it using LDIF. If the sample from our flat-file database reminded you of the LDIF format, you were right on target. This similarity makes the translation easy. Still, we’ll have to watch out for a few snares: Continuation lines Our flat-file database does not have any entries with values spanning several lines, but if it did we’d need to make sure that the output conformed to the LDIF stand- ard. The LDIF standard dictates that all continuation lines must begin with exactly one space. Entry separators Our database uses the adorable character sequence -=- between each entry. Two line separators (i.e., a blank line) must separate LDIF entries, so we’ll need to axe this character sequence when we see it in the input. Attribute separators Right now our data has only one multivalued attribute: aliases. LDIF deals with multivalued attributes by listing each value on a separate line. If we encounter multiple aliases, we’ll need special code to print out a separate line for each. If it weren’t for this misfeature in our data format, the code to go from our format to LDIF would be a single line of Perl. Even with these snares, the conversion program is still pretty simple: my $datafile = 'database'; my $recordsep = "-=-\n"; my $suffix = 'ou=data, ou=systems, dc=ccis, dc=hogwarts, dc=edu'; my $objectclass = <<"EOC"; objectclass: top objectclass: machine EOC open my $DATAFILE, '<', $datafile or die "unable to open $datafile:$!\n"; print "version: 1\n"; # LDAP: A Sophisticated Directory Service | 349 while (<$DATAFILE>) { # print the header for each entry if (/name:\s*(.*)/){ print "dn: cn=$1, $suffix\n"; print $objectclass; print "cn: $1\n"; next; } # handle the multivalued aliases attribute if (s/^aliases:\s*//){ my @aliases = split; foreach my $name (@aliases){ print "aliases: $name\n"; } next; } # handle the end of record separator if ($_ eq $recordsep){ print "\n"; next; } # otherwise, just print the attribute as we found it print; } close $DATAFILE; If we run this code, it prints an LDIF file that looks (in part) like this: version: 1 dn: cn=shimmer, ou=data, ou=systems, dc=ccis, dc=hogwarts, dc=edu objectclass: top objectclass: machine cn: shimmer address: aliases: shim aliases: shimmy aliases: shimmydoodles owner: David Davis department: software building: main room: 909 manufacturer: Sun model: M4000 dn: cn=bendir, ou=data, ou=systems, dc=ccis, dc=hogwarts, dc=edu objectclass: top objectclass: machine cn: bendir address: aliases: ben aliases: bendoodles owner: Cindy Coltrane department: IT building: west 350 | Chapter 9: Directory Services room: 143 manufacturer: Apple model: Mac Pro ... With this LDIF file, we can use one of the bulk-load programs that come with our servers to load our data into the server. For instance, ldif2ldbm, packaged with both the OpenLDAP and Sun JES Directory Servers, reads an LDIF file and imports it directly into the directory server’s native backend format without having to go through LDAP. Though you can only use this program while the server is not running, it can provide the quickest way to get lots of data into a server. If you can’t take the server offline, you can use the LDIF-reading Perl code we developed earlier to feed a file like this to an LDAP server. To throw one more option into the mix, here’s some code that skips the intermediate step of creating an LDIF file and imports our data directly into an LDAP server: use Net::LDAP; use Net::LDAP::Entry; my $datafile = 'database'; my $recordsep = '-=-'; my $server = $ARGV[0]; my $port = getservbyname('ldap','tcp') || '389'; my $suffix = 'ou=data, ou=systems, dc=ccis, dc=hogwarts, dc=edu'; my $rootdn = 'cn=Manager, ou=Systems, dc=ccis, dc=hogwarts, dc=edu'; my $pw = 'secret'; my $c = Net::LDAP-> new ($server,port => $port) or die "Unable to init for $server: $@\n"; my $mesg = $c->bind(dn => $rootdn,password => $pw); if ($mesg->code){ die 'Unable to bind: ' . $mesg->error . "\n"; } open my $DATAFILE, '<', $datafile or die "unable to open $datafile:$!\n"; while (<$DATAFILE>) { chomp; my $entry; my $dn; # at the start of a new record, create a new entry object instance if (/^name:\s*(.*)/){ $dn="cn=$1, $suffix"; $entry = Net::LDAP::Entry->new; $entry->add('cn',$1); next; } # special case for multivalued attribute if (s/^aliases:\s*//){ $entry->add('aliases',[split()]); next; } LDAP: A Sophisticated Directory Service | 351 # if we've hit the end of the record, add it to the server if ($_ eq $recordsep){ $entry->add('objectclass',['top','machine']); $entry->dn($dn); my $res = $c->add($entry); warn 'Error in add for ' . $entry->dn() . ':' . $res->error()."\n" if $res->code(); undef $entry; next; } # add all of the other attributes $entry->add(split(':\s*')); # assume single valued attributes } close $DATAFILE; $c->unbind(); Now that we’ve imported the data into a server, we can start to do some interesting things. To save space, in the following examples the header at the top that sets our configuration variables and the code that binds us to a server will not be repeated. So what can we do with this data when it resides in an LDAP server? We can generate a host file on the fly: use Net::LDAP; ... my $searchobj = $c->search (base => $basedn, scope => 'one', filter => '(objectclass=machine)' attrs => ['cn','address','aliases']); die 'Bad search: ' . $searchobj->error() if $searchobj->code(); if ($searchobj){ print "#\n\# host file - GENERATED BY $0\n # DO NOT EDIT BY HAND!\n#\n"; foreach my $entry ($searchobj->entries()){ print $entry->get_value(address),"\t", $entry->get_value(cn)," ", join(' ', $entry->get_value(aliases),"\n"; } } $c->close(); Here’s the output: # # host file - GENERATED BY ldap2hosts # DO NOT EDIT BY HAND! # shimmer shim shimmy shimmydoodles bendir ben bendoodles sulawesi sula su-lee sander sandy mickey mickeydoo 352 | Chapter 9: Directory Services We can also find the names of all of our machines made by Apple: use Net::LDAP; ... my $searchobj = $c->search(base => $basedn, filter => '(manufacturer=Apple)', scope => 'one', attrs => ['cn']); die 'Bad search: ' . $searchobj->error() if $searchobj->code(); if ($searchobj){ foreach my $entry ($searchobj->entries){ print $entry->get_value('cn'),"\n"; } } $c->unbind(); Here’s the output: bendir sulawesi We can generate a list of machine owners: use Net::LDAP; ... my $searchobj = $c->search(base => $basedn, filter => '(manufacturer=Apple)', scope => 'one', attrs => ['cn','owner']); die 'Bad search: ' . $searchobj->error() if $searchobj->code(); my $entries = $searchobj->as_struct; foreach my $dn (sort byOwner keys %{entries}){ print $entries->{$dn}->{owner}->[0]. ":\t" . $entries->{$dn}->{cn}->[0]."\n"; } # to sort our data structure by owner instead of its DN key sub byOwner { $entries->{$a}->{owner}->[0] <=> $entries->{$b}->{owner}->[0] } Here’s the output: Alex Rollins: sander Cindy Coltrane: bendir David Davis: shimmer Ellen Monk: sulawesi And we can check to see if the current user ID is the owner of the current Unix machine (maybe some kind of pseudo-authentication): use Net::LDAP; use Sys::Hostname; $user = (getpwuid($<))[6]; LDAP: A Sophisticated Directory Service | 353 my $hostname = hostname; my $hostname =~ s/\..*//; # strip domain name off of host ... my $searchobj = $c->search (base => "cn=$hostname,$suffix", scope => 'base', filter => "(owner=$user)" typesonly => 1); if ($searchobj){ print "Owner ($user) can log on to machine $hostname.\n"; } else { print "$user is not the owner of this machine ($hostname).\n"; } These snippets should give you an idea of some of the system administration uses for LDAP access through Perl, and provide inspiration to write your own code. In the next section we’ll take these ideas to the next level and look at a whole administration framework based on the conceptual groundwork laid by LDAP. Not (Really) a Database Before we move on to ADSI, I just want to offer a quick note about one way not to use LDAP. It might be tempting to use an LDAP server as your central repository for all information (as discussed in Chapter 7). Heck, to a certain extent Microsoft uses Active Directory in this fashion. This is up for debate, but I believe this isn’t the best of ideas for a homegrown system. LDAP makes things look very database-like, but it doesn’t have the power of a good relational database. It is very forgiving about what is stored (vis-à-vis data validation), doesn’t really use a relational model, has a limited query language, etc. My preference is to keep most information in a relational database and feed an LDAP server from it. This gives you the power of both models without having to work as hard to make LDAP into something it is not. Microsoft has a considerable amount of code in its management tools and APIs to allow it to use LDAP as a central data store. You probably don’t want to have to write code like that. If you do decide to go this route, be sure to think carefully about it first. Active Directory Service Interfaces For the final section of this chapter, we’ll discuss a platform-dependent directory service framework that is heavily based on the material we’ve just covered. Microsoft created a sophisticated LDAP-based directory service called Active Directory for use at the heart of its Windows administration framework. Active Directory serves 354 | Chapter 9: Directory Services as the repository for all of the important configuration information (users, groups, system policies, software installation support, etc.) used in a network of Windows machines. During the development of Active Directory, the folks at Microsoft realized that a higher-level applications interface to this service was needed. They invented Active Directory Service Interfaces (ADSI) to provide this interface. To their credit, the devel- opers at Microsoft also realized that their new ADSI framework could be extended to cover other system administration realms, such as printers and Windows services. This coverage makes ADSI immensely useful to people who script and automate system administration tasks. Before we show this power in action, we need to cover a few basic concepts and terms. ADSI Basics You can think of ADSI as a wrapper around any directory service that wishes to par- ticipate in the ADSI framework. There are providers, as these ADSI glue implementa- tions are called, for LDAP, Security Accounts Manager (i.e., local/WinNT-domain style) databases, and Novell Directory Services, among others. In ADSI-speak, each of these directory services and data domains are called namespaces. ADSI gives you a uniform way to query and change the data found in these namespaces. To understand ADSI, you have to know a little about the Microsoft Component Object Model (COM) upon which ADSI is built. There are many books about COM, but we can distill the basics down to these key points: • Everything we want to work with via COM is an object.* • Objects have interfaces that provide a set of methods for us to use to interact with these objects. From Perl, we can use the methods provided by or inherited from the interface called IDispatch. Luckily, most of the ADSI methods provided by the ADSI interfaces and their children (e.g., IADsUser, IADsComputer, IADsPrintQueue) are inherited from IDispatch. • The values encapsulated by an object, which is queried and changed through these methods, are called properties. We’ll refer to two kinds of properties in this chapter: interface-defined properties (those that are defined as part of an interface) and schema-defined properties (those that are defined in a schema object—more on this in just a moment). Unless I refer explicitly to “schema properties” in the following discussion, you can assume we’re using interface properties. This is standard object-oriented programming fare, but it starts to get tricky when the nomenclature for ADSI/COM and other object-oriented worlds, like LDAP, collide. * COM is in fact the protocol used to communicate with these objects as part of the larger framework called Object Linking and Embedding (OLE). In this section, I’ve tried to keep us out of the Microsoft morass of acronyms, but if you want to dig deeper, some good resources are available at Active Directory Service Interfaces | 355 For instance, in ADSI we speak of two different kinds of objects: leaf and container. Leaf objects encapsulate real data; container objects hold, or parent, other objects. In LDAP-speak, a close translation for these terms might be “entry” and “branching point.” On the one hand we talk about objects with properties, and on the other we talk about entries with attributes. So how do you deal with this discrepancy, since both names refer to the exact same data? Here’s one way to think about it: an LDAP server does indeed provide access to a tree full of entries and their associated attributes. When you use ADSI instead of native LDAP to get at an entry in that tree, ADSI sucks the entry out of the LDAP server, wraps it up in a few layers of shiny wrapping paper, and hands it to you as a COM object. You use the necessary methods to get the contents of that parcel, which are now called “properties.” If you make any changes to the properties of this object, you can hand the object back to ADSI, which will take care of unwrapping the information and put- ting it back in the LDAP tree for you. A reasonable question at this point is, “Why not go directly to the LDAP server?” There are three good answers: • Once we know how to use ADSI to communicate with one kind of directory service, we know how to communicate with them all (or at least the ones that have ADSI providers). • ADSI’s encapsulation can make directory service programming a little easier. • Microsoft tells us to use ADSI. Using Microsoft’s supported API is almost always the right decision. To head in the direction of ADSI programming from Perl, we need to introduce ADsPaths. ADsPaths give us a unique way to refer to objects in any of our namespaces. They look like this: : where is the programmatic identifier for a provider and is a provider-specific way of finding the object in its namespace. The two most common progIDs are LDAP and WinNT (WinNT uses the SAM databases mentioned in Chapter 3). Here are some ADsPath examples taken from the ADSI SDK documentation: WinNT://MyDomain/MyServer/User WinNT://MyDomain/JohnSmith,user LDAP://ldapsvr/CN=TopHat,DC=DEV,DC=MSFT,DC=COM,O=Internet LDAP://,DC=DEV,DC=MSFT,DC=COM,O=Internet It’s no coincidence that these look like URLs, since both URLs and ADsPaths serve roughly the same purpose: they both try to provide an unambiguous way to reference a piece of data made available by different data services. In the case of LDAP ADsPaths, we are using the LDAP URL syntax from the RFC mentioned in Appendix C (RFC 2255). 356 | Chapter 9: Directory Services The portion is case-sensitive. Using winnt, ldap, or WINNT in- stead of WinNT and LDAP will cause your programs to fail. Also be sure to note that there are some characters that can’t be used in an ADsPath without being escaped with a backslash or represented in hexadecimal format.† At the time of this writing, they were the line feed and carriage return, ,, ;, ", #, +, <, =, >, and \. We’ll look more closely at ADsPaths when we discuss the two namespaces, WinNT and LDAP, referenced earlier. Before we get there, let’s see how ADSI in general is used from Perl. The Tools of the ADSI Trade Any machine running Windows 2000 or later has ADSI built into the OS. I recommend downloading the ADSI SDK found at, because it pro- vides this documentation and a handy ADSI object browser called ADsVW. The SDK comes with ADSI programming examples in a number of languages, including Perl. Unfortunately, the examples in the current ADSI distribution rely on the deprecated module, so while you might be able to pick up a few tips, you should not use these examples as your starting point. At this URL you will also find crucial ADSI documentation including adsi25.chm, a compressed HTML help file that contains some of the best ADSI documentation available. Before you begin to code, you will also want to pick up Toby Everett’s ADSI object browser (written in Perl) from It will help you navigate around the ADSI namespaces. Be sure to visit this site early in your ADSI programming career. It hasn’t been updated in a while, but it remains a good starting place for using ADSI from Perl. One last tip: even if it makes you queasy, it is in your best interest to gain just enough familiarity with VBScript to be able to read scripts written in that language. The deeper you get into ADSI, the more VBScript code you’ll find yourself reading and adapting. Appendix F and some of the references listed at the end of this chapter should help a bit with this learning process. Using ADSI from Perl The Win32::OLE family of modules, maintained by Jan Dubois, gives us a Perl bridge to ADSI (which is built on COM as part of OLE). After loading the main module, we use it to request an ADSI object: † There’s an old vaudeville skit where a man goes to the doctor and complains, “Doc, my arm hurts when I move it like this,” only to receive the advice, “So, don’t move it like that!” I have to offer the same advice. Don’t set up a situation where you might need to use these characters in an ADsPath. You’ll only be asking for trouble. Active Directory Service Interfaces | 357 use Win32::OLE; $adsobj = Win32::OLE->GetObject($ADsPath) or die "Unable to retrieve the object for $ADsPath\n"; Here are two tips that may save you some consternation. First, if you run these two lines of code in the Perl debugger and examine the con- tents of the object reference that is returned, you might see something like this: DB<3> x $adsobj 0 Win32::OLE=HASH(0x10fe0d4) empty hash Don’t panic. Win32::OLE uses the power of tied variables. The seemingly empty data structure you see here will magically yield information from our object when we access it properly. Second, if your GetObject call returns something like this (especially from within the debugger): Win32::OLE(0.1403) error 0x8007202b: "A referral was returned from the server" it often means you’ve requested an LDAP provider ADsPath for an LDAP tree that doesn’t exist on your server. This is usually the result of a simple typo: e.g., you typed LDAP://dc=exampel,dc=com when you really meant LDAP://dc=example,dc=com. Win32::OLE->GetObject() takes an OLE moniker (a unique identifier to an object, which in this case is an ADsPath) and returns an ADSI object for us. This call also handles the process of binding to the object, which is a process you should be familiar with from our LDAP discussion. By default we bind to the object using the credentials of the user running the script. Perl’s hash reference syntax is used to access the interface property values of an ADSI object: $value = $adsobj->{key} For instance, if that object had a Name property defined as part of its interface (and they all do), you could retrieve it like this: print $adsobj->{Name}."\n"; Interface property values can be assigned using the same notation: $adsobj->{FullName}= "Oog"; # set the property in the cache An ADSI object’s properties are stored in an in-memory cache called the property cache. The first request for an object’s properties populates this cache. Subsequent queries for the same property will retrieve the information from this cache, not from the directory service. If you want to populate the cache by hand, you can call that object 358 | Chapter 9: Directory Services instance’s GetInfo() or GetInfoEx() method (an extended version of GetInfo()) using the syntax we’ll see in a moment. Because the initial fetch is automatic, GetInfo() and GetInfoEx() are often overlooked. Though we won’t see any in this book, there are cases where you will need them. Here are two example cases: 1. Some object properties are only fetched by an explicit GetInfoEx() call. For exam- ple, many of the properties of Microsoft Exchange 5.5’s LDAP provider were not available without calling GetInfoEx() first. See thors/tobyeverett/ for more details on this inconsistency. 2. If you have a directory that multiple people can change, an object you may have just retrieved could be changed while you are still working with it. If this happens, the data in your property cache for that object will be stale. GetInfo() and GetInfoEx() will refresh this cache for you. To actually update the backend directory service and data source provided through ADSI, you must call the special method SetInfo() after changing an object. SetInfo() flushes the changes from the property cache to the actual directory service or data source. Calling methods from an ADSI object instance is easy: $adsobj->Method($arguments...) So, if we changed an object’s properties, we might use this line right after the code that made the change: $adsobj->SetInfo(); This would flush the data from the property cache back into the underlying directory service or data source. One Win32::OLE call you’ll want to use often is Win32::OLE->LastError(). This will re- turn the error, if any, that the last OLE operation generated. Using the -w switch with Perl (e.g., perl -w script) also causes any OLE failures to complain in a verbose manner. Often these error messages are all the debugging help you have, so be sure to make good use of them. The ADSI code we’ve seen so far should look like fairly standard Perl to you, because on the surface, it is. Now let’s introduce a few of the plot complications. Dealing with Container/Collection Objects Earlier, I mentioned that there are two kinds of ADSI objects: leaf and container objects. Leaf objects represent pure data, whereas container objects (also called “collection objects” in OLE/COM terms) contain other objects. Another way to distinguish be- tween the two in the ADSI context is by noting that leaf objects have no children, but container objects do. Active Directory Service Interfaces | 359 Container objects require special handling, since most of the time we’re interested in the data encapsulated by their child objects. There are two ways to access these objects from Perl. Win32::OLE offers a special function called in() for this purpose, though it is not available by default when the module is loaded in the standard fashion. We have to use the following line at the beginning of our code to make use of it: use Win32::OLE qw(in); in() will return a list of references to the child objects held by the specified container. This allows us to write easy-to-read Perl code like: foreach $child (in $adsobj){ print $child->{Name} } Alternatively, we can load one of Win32::OLE’s helpful progeny, called Win32::OLE::Enum. So Win32::OLE::Enum->new() will create an enumerator object from one of our container objects: use Win32::OLE::Enum; $enobj = Win32::OLE::Enum->new($adsobj); We can then call a few methods on this enumerator object to get at $adsobj’s children. $enobj->Next() will return a reference to the next child object instance (or the next N objects if given an optional parameter). $enobj->All() returns a list of object instance references. Win32::OLE::Enum offers a few more methods (see the documentation for details), but these are the ones you’ll use most often. Identifying a Container Object You can’t know if an object is a container object a priori. There is no way to ask an object itself about its “containerness” from Perl. The closest you can come is to try to create an enumerator object and fail gracefully if this does not succeed. Here’s some code that does just that: use Win32::OLE; use Win32::OLE::Enum; eval {$enobj = Win32::OLE::Enum->new($adsobj)}; print 'object is ' . ($@ ? 'not ' : '') . "a container\n"; Alternatively, you can look to other sources that describe the object. This segues nicely into our third plot complication. So How Do You Know Anything About an Object? We’ve avoided the biggest and perhaps the most important question until now. In a moment we’ll be dealing with objects in two of our namespaces. You already know how to retrieve and set object properties and how to call object methods for these 360 | Chapter 9: Directory Services objects, but only if you already know the names of these properties and methods. Where do these names come from? How do you find them in the first place? There’s no single place to find an answer to these questions, but there are a few sources we can draw upon to get most of the picture. The first place is the ADSI documentation—especially the help file mentioned in the earlier sidebar, “The Tools of the ADSI Trade” on page 357. This file contains a huge amount of helpful material. For the answer to our question about property and method names, the place to start in the file is Active Directory Service Interfaces 2.5→ADSI Reference→ADSI System Providers. The documentation is sometimes the only place to find method names, but there’s a second, more interesting approach we can take when looking for property names: we can use metadata that ADSI itself provides. This is where the schema properties concept I mentioned earlier comes into the picture (see the first part of the section “ADSI Ba- sics” on page 355 if you don’t recall the schema/interface property distinction). Every ADSI object has a property called Schema that yields an ADsPath to its schema object. For instance, the following code: use Win32::OLE; $ADsPath = 'WinNT://BEESKNEES,computer'; $adsobj = Win32::OLE->GetObject($ADsPath) or die "Unable to retrieve the object for $ADsPath\n"; print 'This is a '.$adsobj->{Class}."object, schema is at:\n". $adsobj->{Schema},"\n"; will print: This is a Computer object, schema is at: WinNT://DomainName/Schema/Computer The value of $adsobj->{Schema} is an ADsPath to an object that describes the schema for the objects of class Computer in that domain. Here we’re using the term “schema” in the same way we used it when talking about LDAP schemas. In LDAP, schemas define which attributes can and must be present in entries of specific object classes. In ADSI, a schema object holds the same information about objects of a certain class and their schema properties. If we want to see the possible attribute names for an object, we can look at the values of two properties in its schema object: MandatoryProperties and OptionalProperties. Let’s change the print statement from our last example to the following: $schmobj = Win32::OLE->GetObject($adsobj->{Schema}) or die "Unable to retrieve the object for $ADsPath\n"; print join("\n",@{$schmobj->{MandatoryProperties}}, @{$schmobj->{OptionalProperties}}),"\n"; Active Directory Service Interfaces | 361 This prints: Owner Division OperatingSystem OperatingSystemVersion Processor ProcessorCount Now we know the possible schema interface property names in the WinNT namespace for our Computer objects. Pretty nifty, eh? Schema properties are retrieved and set in a slightly different manner than interface properties. Recall that interface properties are retrieved and set like this: # retrieving and setting INTERFACE properties $value = $obj->{property}; $obj->{property} = $value; Schema properties are retrieved and set using special methods: # retrieving and setting SCHEMA properties $value = $obj->Get('property'); $obj->Put('property','value'); Everything we’ve talked about so far regarding interface properties holds true for schema properties as well (i.e., the property cache, SetInfo(), etc.). Besides the need to use special methods to retrieve and set values, the only other place where you’ll need to distinguish between the two is in their names. Sometimes the same object may have two different names for essentially the same property: one for the interface property and one for the schema property. For example, these two lines retrieve the same basic setting for a user: $len = $userobj->{PasswordMinimumLength}; # the interface property $len = $userobj->Get('MinPasswordLength'); # the same schema property There are two kinds of properties because interface properties exist as part of the un- derlying COM model. When developers define an interface as part of developing a program, they also define the interface properties. Later, if they want to extend the property set, they have to modify both the COM interface and any code that uses that interface. In ADSI, developers can change the schema properties in a provider without having to modify the underlying COM interface for that provider. It is important to become comfortable with dealing with both kinds of properties, because sometimes a certain piece of data in an object is made available only from within one kind. On a practical note, if you are just looking for interface or schema property names and don’t want to bother writing a program to find them, I recommend using Toby Everett’s ADSI browser, mentioned earlier. Figure 9-2 is a sample screen shot of this browser in action. 362 | Chapter 9: Directory Services Figure 9-2. Everett’s ADSI browser displaying an Administrators group object Alternatively, the General folder of the SDK samples contains a program called ADSIDump that can dump the contents of an entire ADSI tree for you. Searching This is the last complication we’ll discuss before moving on. In the section “LDAP: A Sophisticated Directory Service” on page 321, we spent considerable time talking about LDAP searches. But here in ADSI-land, we’ve breathed hardly a word about the subject. This is because from Perl (and any other language that uses the same OLE automation interface), searching with ADSI is a pain—that is, subtree searches, or searches that entail anything but the simplest of search filters, are excruciatingly painful (other types of search are not so bad). Complex searches are troublesome because they require you to step out of the ADSI framework and use a whole different methodology to get at your data (not to mention learn more Microsoft acronyms). Active Directory Service Interfaces | 363 But people who do system administration are trained to laugh at pain, so let’s take a look. We’ll start with simple searches before tackling the hard stuff. Simple searches that encompass one object (scope of base) or an object’s immediate children (scope of one) can be handled manually with Perl. Here’s how: • For a single object, retrieve the properties of interest and use the normal Perl com- parison operators to determine if this object is a match: if ($adsobj->{cn} eq 'Mark Sausville' and $adsobj->{State} eq 'CA'){...} • To search the children of an object, use the container object access techniques we discussed previously and then examine each child object in turn. We’ll see some examples of this type of search in a moment. If you want to do more complex searches, like those that entail searching a whole directory tree or subtree, you need to switch to using a different “middleware” tech- nology called ActiveX Data Objects (ADO). ADO offers scripting languages an interface to Microsoft’s OLE DB layer. OLE DB provides a common database-oriented interface to data sources such as relational databases and directory services. In our case we’ll be using ADO to talk to ADSI (which then talks to the actual directory service). Because ADO is a database-oriented methodology, the code you are about to see relates to the ODBC material we covered in Chapter 7. ADO only works when talking to the LDAP ADSI provider. It will not work for the WinNT namespace. ADO is a whole subject in itself that is only peripherally related to the subject of directory services, so we’ll do no more than look at one example and a little bit of explanation before moving on to some more relevant ADSI examples. For more information on ADO itself, search the Microsoft website for the term “ADO” and check out this Wikipedia page: Here’s some code that displays the names of all of the groups to be found in a given domain: use Win32::OLE qw(in); # get the ADO object, set the provider, open the connection $c = Win32::OLE->new('ADODB.Connection'); $c->{Provider} = 'ADsDSOObject'; $c->Open('ADSI Provider'); die Win32::OLE->LastError() if Win32::OLE->LastError(); # prepare and then execute the query $ADsPath = 'LDAP://ldapserver/dc=example,dc=com'; $rs = $c->Execute("<$ADsPath>;(objectClass=Group);Name;SubTree"); die Win32::OLE->LastError() if Win32::OLE->LastError(); 364 | Chapter 9: Directory Services until ($rs->EOF){ print $rs->Fields(0)->{Value},"\n"; $rs->MoveNext; } $rs->Close; $c->Close; After loading the modules, this block of code gets an ADO Connection object instance, sets that object instance’s provider name, and then instructs it to open the connection. This connection is opened on behalf of the user running the script, though we could have set some other object properties to change this. We then perform the actual search using Execute(). This search can be specified using one of two “dialects,” SQL or ADSI.‡ The ADSI dialect, as shown, uses a command string consisting of four arguments, separated by semicolons. Be careful of this ADSI ADO provider quirk: there cannot be any white- space around the semicolons, or the query will fail. The arguments are: • An ADsPath (in angle brackets) that sets the server and base DN for the search • A search filter (using the same LDAP filter syntax we saw before) • The name or names (separated by commas) of the properties to return • A search scope of either Base, OneLevel, or SubTree (as per the LDAP standard) Execute() returns a reference to the first of the ADO RecordSet objects returned by our query. We ask for each RecordSet object in turn, unpacking the objects it holds and printing the Value property returned by the Fields() method for each of these objects. The Value property contains the value we requested in our command string (the name of the Group object). Here’s an excerpt from sample output from a Windows Server 2003 machine: Domain Computers Domain Users RAS and IAS Servers Users Domain Guests Group Policy Creator Owners Enterprise Admins Server Operators Account Operators ‡ The mention of SQL in this context leads into an interesting aside: Microsoft SQL Server can be configured to know about ADSI providers in addition to normal databases. This means that you can execute SQL queries against SQL Server and have it actually query ActiveDirectory objects via ADSI instead of normal databases. Pretty cool. Active Directory Service Interfaces | 365 Print Operators Replicator Domain Controllers Schema Admins Remote Desktop Users Network Configuration Operators Incoming Forest Trust Builders Performance Monitor Users Terminal Server License Servers Pre-Windows 2000 Compatible Access Performance Log Users Windows Authorization Access Group Backup Operators Domain Admins Administrators Cert Publishers Guests DnsAdmins DnsUpdateProxy Debugger Users Performing Common Tasks Using the WinNT and LDAP Namespaces Now that we’ve safely emerged from our list of complications, we can turn to per- forming some common administrative tasks using ADSI from Perl. The goal is to give you a taste of the things you can do with the ADSI information we’ve presented. Then you can use the code we’re going to see as starter recipes for your own programming. For these tasks, we’ll use one of two namespaces. The first namespace is WinNT, which gives us access to the local Windows SAM database that includes objects like local users, groups, printers, services, etc. The second is our friend LDAP. LDAP becomes the provider of choice when we move on to the LDAP-based Active Directory of Windows 2000 and beyond. Most of the WinNT objects can be accessed via LDAP as well. But even with Windows Server 2003, there are still tasks that can only be performed using the WinNT namespace (like the creation of local machine accounts). The code that works with these different namespaces looks similar (after all, that’s part of the point of using ADSI), but you should note two important differences. First, the ADsPath format is slightly different. The WinNT ADsPath takes one of these forms, according to the ADSI SDK: WinNT:[//DomainName[/ComputerName[/ObjectName[,className]]]] WinNT:[//DomainName[/ObjectName[,className]]] WinNT:[//ComputerName,computer] WinNT: 366 | Chapter 9: Directory Services The LDAP ADsPath looks like this: LDAP://HostName[:PortNumber][/DistinguishedName] Note that the properties of the objects in the LDAP and WinNT namespaces are similar, but they are not the same. For instance, you can access the same user objects from both namespaces, but you can only get to some Active Directory properties for a particular user object through the LDAP namespace. It’s especially important to pay attention to the differences between the schemas found in the two namespaces. For example, the User class for WinNT has no mandatory prop- erties, while the LDAP User class has several. With the LDAP namespace, you need to populate at least the cn and samAccountName properties to successfully create a User object. With these differences in mind, let’s look at some actual code. To save space, we’re going to omit most of the error checking, but you’ll want to run your scripts with the -w switch and liberally sprinkle lines like this throughout your code: die 'OLE error :'.Win32::OLE->LastError() if Win32::OLE->LastError(); In the examples that follow, you’ll find that I flip-flop between using the WinNT and LDAP namespaces. This is to give you a sense of how to use both of them. Deciding which one to use depends largely on the task at hand and the size of the Active Directory implementation in play. Sometimes the decision is made for you. For example, you need to use WinNT when dealing with local machine users/services and for printer queue control; conversely, you need to use LDAP to access some user properties in AD, AD control objects, and so on. For other tasks, you have a choice. In those cases LDAP is usually prefer- red (despite being a bit more complex) because it is more efficient. With the LDAP namespace, you can operate directly on an object deep in the AD tree without having to enumerate through a list of objects as you would when using WinNT. If your AD implementation is relatively small, this efficiency gain may not matter to you and the ease of using WinNT may be more compelling. It is largely your choice. Working with Users via ADSI To dump the list of users using the WinNT namespace: use Win32::OLE qw(in); # 'WinNT://CurrentComputername,computer' - accounts local to this computer # 'WinNT://DCname, computer' - accounts for the client's current domain # 'WinNT://DomainName/DCName,computer' - to specify the domain my $ADsPath= 'WinNT://DomainName/DCName,computer'; my $c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n"; Active Directory Service Interfaces | 367 foreach my $adsobj (in $c){ print $adsobj->{Name},"\n" if ($adsobj->{Class} eq 'User'); } If you wanted to use the LDAP namespace instead of the WinNT namespace to do an exhaustive (i.e., entire-tree) search for users, you would need to use the ADO-based method demonstrated in the section on searching. To create a user (local to the machine) and set that user’s full name: use Win32::OLE; my $ADsPath='WinNT://LocalMachineName,computer'; my $c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n"; # create and return a User object my $u = $c->Create('user',$username); $u->SetInfo(); # we have to create the user before we modify it # no space between "Full" and "Name" allowed with WinNT namespace $u->{FullName} = $fullname; $u->SetInfo(); The equivalent code to create a global user (you can’t create local users using the LDAP namespace) in Active Directory looks like this: use Win32::OLE; # This creates the user under the cn=Users branch of your directory tree. # If you keep your users in a sub-OU of Users, just change the next line. my $ADsPath= 'LDAP://ldapserver,CN=Users,dc=example,dc=com'; my $c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n"; # create and return a User object my $u=$c->Create('user','cn='.$commonname); $u->{samAccountName} = $username; # IMPORTANT: we have to create the user in the dir before we modify it $u->SetInfo(); # space between "Full" and "Name" required with LDAP namespace (sigh) $u->{'Full Name'} = $fullname; $u->SetInfo(); Deleting a local user requires just a small change: use Win32::OLE; my $ADsPath= 'WinNT://DomainName/ComputerName,computer'; my $c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n"; # delete the User object; note that we are bound to the container object $c->Delete('user',$username); $c->SetInfo(); 368 | Chapter 9: Directory Services Changing a user’s password is a single method’s work: use Win32::OLE; # or 'LDAP://cn=$username,ou=staff,ou=users,dc=example,dc=com' (for example) my $ADsPath= 'WinNT://DomainName/ComputerName/'.$username; my $u = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n"; $u->ChangePasssword($oldpassword,$newpassword); $u->SetInfo(); Working with Groups via ADSI You can enumerate the available groups using the WinNT namespace with just a minor tweak of our user enumeration code. The one changed line is: print $adsobj->{Name},"\n" if ($adsobj->{Class} eq 'Group'); If you want to enumerate groups using the LDAP namespace, it is best to use ADO (see the section “Searching” on page 363). Creation and deletion of groups involves the same Create() and Delete() methods we just saw for user account creation and deletion; the only difference is the first argument needs to be 'group'. For example: my $g = $c->Create('group',$groupname); To add a user to a group (specified as a GroupName) once you’ve created it: use Win32::OLE; my $ADsPath= 'WinNT://DomainName/GroupName,group'; my $g = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n"; # this uses the ADsPath to a specific user object $g->Add($userADsPath); With the WinNT namespace, the same rules we saw earlier about local versus domain (global) users apply here as well. If we want to add a domain user to our group, our $userADsPath should reference the user at a DC for that domain. If we want to use the LDAP namespace for this task, we explicitly point at the group in the directory tree: my $ADsPath= 'LDAP://cn=GroupName,ou=Groups,dc=example,dc=com'; To remove a user from a group, use: $c->Remove($userADsPath); Working with File Shares via ADSI Now we start to get into some of the more interesting ADSI work. It is possible to use ADSI to instruct a machine to start sharing a part of its local storage with other computers: Active Directory Service Interfaces | 369 use Win32::OLE; my $ADsPath= 'WinNT://ComputerName/lanmanserver'; my $c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n"; my $s = $c->Create('fileshare',$sharename); $s->{path} = 'C:\directory'; $s->{description} = 'This is a Perl created share'; $s->SetInfo(); File shares are deleted using the Delete() method. Before we move on to other tasks, let me take this opportunity to remind you to closely consult the SDK documentation before using any of these ADSI objects. Sometimes, you’ll find useful surprises. If you look in the ADSI 2.5 help file at Active Directory Service Interfaces 2.5→ADSI Ref- erence→ADSI Interfaces→Persistent Object Interfaces→IADsFileShare, you’ll see that a fileshare object has a CurrentUserCount property that shows how many users are currently connected to this file share. This could be a very handy detail. Working with Print Queues and Print Jobs via ADSI Here’s how to determine the names of the queues on a particular server and the models of the printers being used to serve those queues: use Win32::OLE qw(in); my $ADsPath='WinNT://DomainName/PrintServerName,computer'; my $c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n"; foreach my $adsobj (in $c){ print $adsobj->{Name}.':'.$adsobj->{Model}."\n" if ($adsobj->{Class} eq 'PrintQueue'); } Once you have the name of a print queue, you can bind to it directly to query and control it: use Win32::OLE qw(in); # this table comes from this section in the ADSI 2.5 SDK: # 'Active Directory Service Interfaces 2.5->ADSI Reference-> # ADSI Interfaces->Dynamic Object Interfaces->IADsPrintQueueOperations-> # IADsPrintQueueOperations Property Methods' (phew) my %status = (0x00000001 => 'PAUSED', 0x00000002 => 'PENDING_DELETION', 0x00000003 => 'ERROR' , 0x00000004 => 'PAPER_JAM', 0x00000005 => 'PAPER_OUT', 0x00000006 => 'MANUAL_FEED', 0x00000007 => 'PAPER_PROBLEM', 0x00000008 => 'OFFLINE', 370 | Chapter 9: Directory Services 0x00000100 => 'IO_ACTIVE', 0x00000200 => 'BUSY', 0x00000400 => 'PRINTING', 0x00000800 => 'OUTPUT_BIN_FULL', 0x00001000 => 'NOT_AVAILABLE', 0x00002000 => 'WAITING', 0x00004000 => 'PROCESSING', 0x00008000 => 'INITIALIZING', 0x00010000 => 'WARMING_UP', 0x00020000 => 'TONER_LOW', 0x00040000 => 'NO_TONER', 0x00080000 => 'PAGE_PUNT', 0x00100000 => 'USER_INTERVENTION', 0x00200000 => 'OUT_OF_MEMORY', 0x00400000 => 'DOOR_OPEN', 0x00800000 => 'SERVER_UNKNOWN', 0x01000000 => 'POWER_SAVE'); my $ADsPath = 'WinNT://PrintServerName/PrintQueueName'; my $p = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n"; print 'The printer status for ' . $c->{Name} . ' is ' . ((exists $p->{status}) ? $status{$c->{status}} : 'NOT ACTIVE') . "\n"; The PrintQueue object offers the set of print queue control methods you’d hope for: Pause(), Resume(), and Purge(). These allow us to control the actions of the queue itself. But what if we want to examine or manipulate the actual jobs in this queue? To get at the actual jobs, you call a PrintQueue object method called PrintJobs(). PrintJobs() returns a collection of PrintJob objects, each of which has a set of prop- erties and methods. For instance, here’s how to show the jobs in a particular queue: use Win32::OLE qw(in); # this table comes from this section in the ADSI 2.5 SDK: # 'Active Directory Service Interfaces 2.5->ADSI Reference-> # ADSI Interfaces->Dynamic Object Interfaces->IADsPrintJobOperations-> # IADsPrintJobOperations Property Methods' (double phew) my %status = (0x00000001 => 'PAUSED', 0x00000002 => 'ERROR', 0x00000004 => 'DELETING',0x00000010 => 'PRINTING', 0x00000020 => 'OFFLINE', 0x00000040 => 'PAPEROUT', 0x00000080 => 'PRINTED', 0x00000100 => 'DELETED'); my $ADsPath = 'WinNT://PrintServerName/PrintQueueName'; my $p = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n"; $jobs = $p->PrintJobs(); foreach my $job (in $jobs){ print $job->{User} . "\t" . $job->{Description} . "\t" . $status{$job->{status}} . "\n"; } Each job can be Pause()d and Resume()d as well. Working with Windows-Based Operating System Services via ADSI For our last set of examples, we’re going to look at how to locate, start, and stop the services on a Windows machine. Like the other examples in this chapter, these code Active Directory Service Interfaces | 371 snippets must be run from an account with sufficient privileges on the target computer to effect changes. To list the services on a computer and their statuses, we could use this code: use Win32::OLE qw(in); # this table comes from this section in the ADSI 2.5 SDK: # 'Active Directory Service Interfaces 2.5->ADSI Reference-> # ADSI Interfaces->Dynamic Object Interfaces->IADsServiceOperations-> # IADsServiceOperations Property Methods' my %status = (0x00000001 => 'STOPPED', 0x00000002 => 'START_PENDING', 0x00000003 => 'STOP_PENDING', 0x00000004 => 'RUNNING', 0x00000005 => 'CONTINUE_PENDING',0x00000006 => 'PAUSE_PENDING', 0x00000007 => 'PAUSED', 0x00000008 => 'ERROR'); my $ADsPath = 'WinNT://DomainName/ComputerName,computer'; my $c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n"; foreach my $adsobj (in $c){ print $adsobj->{DisplayName} . ':' . $status{$adsobj->{status}} . "\n" if ($adsobj->{Class} eq 'Service'); } To start, stop, pause, or continue a service, we call the obvious methods (Start(), Stop(), etc.). Here’s how we might start the Network Time service on a Windows machine if it were stopped: use Win32::OLE; my $ADsPath = 'WinNT://DomainName/ComputerName/W32Time,service'; my $s = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n"; $s->Start(); # may wish to check status at this point, looping until it is started To avoid potential user and computer name conflicts, the previous code can also be written as: use Win32::OLE; my $d = Win32::OLE->GetObject('WinNT://Domain'); my $c = $d->GetObject('Computer', $computername); my $s = $c->GetObject('Service', 'W32Time'); $s->Start(); Stopping it is just a matter of changing the last line to: $s->Stop(); # may wish to check status at this point, sleep for a second or two # and then loop until it is stopped 372 | Chapter 9: Directory Services These examples should give you some idea of the amount of control using ADSI from Perl can give you over your system administration work. Directory services and their interfaces can be a very powerful part of your computing infrastructure. Module Information for This Chapter Name CPAN ID Version Net::Telnet JROGERS 3.03 Net::Finger FIMM 1.06 Net::Whois::Raw DESPAIR 0.34 Net::LDAP GBARR 0.32 Sys::Hostname (ships with Perl) 1.11 Win32::OLE (ships with ActiveState Perl) JDB 0.17 References for More Information The following sections list some resources you might want to consult for further information on the topics discussed in this chapter. RFC 1288: The Finger User Information Protocol, by D. Zimmerman (1991), defines Finger. is a list of most major WHOIS servers. RFC 954: NICNAME/WHOIS, by K. Harrenstien, M. Stahl, and E. Feinler (1985), defines WHOIS. LDAP is the home page for Net::LDAP. is the home page for OpenLDAP, a free LDAP server under active development. JXplorer ( and Apache Directory Studio (http://directory are both good, free GUI LDAP browsers that work with all of the LDAP servers I’ve ever used. led ( and ldapdiff ( are two handy command-line utilities to help with the editing of LDAP entries/trees. The first pops you into an editor of your choice to edit an LDIF representation of an entry, and the second helps with showing the difference between a live LDAP tree and an LDIF file (and patching it accordingly if you’d like). You might also want to consult the following sources on LDAP: References for More Information | 373 • Implementing LDAP, by Mark Wilcox (Wrox Press) • LDAP-HOWTO, by Mark Grennan (1999), available at ldap-HOWTO.html • Understanding and Deploying LDAP Directory Services, Second Edition, by Tim Howes et al. (Addison-Wesley) • RFC 1823: The LDAP Application Program Interface, by T. Howes and M. Smith (1995) • RFC 2222: Simple Authentication and Security Layer (SASL), by J. Myers (1997) • RFC 2251: Lightweight Directory Access Protocol (v3), by M. Wahl, T. Howes, and S. Kille (1997) • RFC 2252: Lightweight Directory Access Protocol (v3): Attribute Syntax Defini- tions, by M. Wahl et al. (1997) • RFC 2254: The String Representation of LDAP Search Filters, by T. Howes (1997) • RFC 2255: The LDAP URL Format, by T. Howes and M. Smith (1997) • RFC 2256: A Summary of the X.500(96) User Schema for Use with LDAPv3, by M. Wahl (1997) • RFC 2849: The LDAP Data Interchange Format (LDIF)—Technical Specification, by Gordon Good (2000) • Understanding LDAP, by Heinz Jonner et al. (1998), available at (a superb “Redbook” introduction to LDAP) • LDAP System Administration (, by Ger- ald Carter (O’Reilly) • LDAP Programming, Management, and Integration, by Clayton Donley (Manning) ADSI is a good (non-Perl-specific) site on scripting ADSI and other Microsoft technologies. is the canonical source for ADSI information. contains Toby Everett’s collection of documentation on using ADSI from Perl. is another good (non-Perl-specific) site on scripting ADSI and other Microsoft technologies. presents a whole system-management infrastructure for Windows written almost entirely in Perl. 374 | Chapter 9: Directory Services Robbie Allen, author/coauthor of a slew of superb books on Windows and AD, has a website at where you can find the code samples from all of his books. It truly is the mother lode of examples—one of the single most helpful websites for ADSI programming that you’ll ever find. For more on Allen’s contributions, see the references at the end of Chapter 3. You might also want to check out these sources: • Active Directory (, Second Edition, by Alistair G. Lowe-Norris (O’Reilly) • Managing Enterprise Active Directory Services, by Robbie Allen and Richard Puckett (Addison-Wesley) • Microsoft Windows 2000 Scripting Guide: Automating System Administration (Microsoft Press) References for More Information | 375 CHAPTER 10 Log Files If this weren’t a book on system administration, an entire chapter on log files would seem peculiar. But system administrators have a very special relationship with log files. System administrators are expected to be like Doctor Doolittle, who could talk to the animals: able to communicate with a large menagerie of software and hardware. Much of this communication takes place through log files, so we become log file linguists. Perl can be a big help in this process. It would be impossible to touch on all the different kinds of processing and analysis you can do with logs in a single chapter. Entire books have been devoted to just stat- istical analysis of this sort of data, and companies have been founded to sell products to help analyze it. However, this chapter does present some general approaches to the topic and some relevant Perl tools, to whet your appetite for more. Reading Text Logs Logs come in different flavors, so we need several approaches for dealing with them. The most common type of log file is one composed entirely of lines of text: popular server packages like Apache (Web), BIND (DNS), and sendmail (email) spew log text in voluminous quantities (especially in debug mode). Most logs on Unix machines look similar because they are created by a centralized logging facility known as syslog. For our purposes, we can treat files created by syslog like any other text files. Here’s a simple Perl program to scan for the word “error” in a text-based log file: open my $LOG, '<', "$logfile" or die "Unable to open $logfile:$!\n"; while(my $line = <$LOG>){ print if $line =~ /\berror\b/i; } close $LOG; Perl-savvy readers are probably itching to turn it into a one-liner. For those folks: perl -ne 'print if /\berror\b/i' logfile 377 Reading Binary Log Files Sometimes it’s not that easy writing programs to deal with log files. Instead of nice, easily parsable text lines, some logging mechanisms produce nasty, gnarly binary files with proprietary formats that can’t be parsed with a single line of Perl. Luckily, Perl isn’t afraid of these miscreants. Let’s look at a few approaches we can take when dealing with these files. We’re going to look at two different examples of binary logs: Unix’s wtmp file and Windows-based operating system event logs. Back in Chapter 4, we touched briefly on the notion of logging in and logging out of a Unix host. Login and logout activity is tracked in a file called wtmpx (or wtmp) on most Unix variants. It is common to check this file whenever there is a question about a user’s connection habits (e.g., what hosts does this person usually log in from?). It tends to live in different places depending on the operating system (e.g., Solaris has it in /var/adm, Linux in /var/log*). On Windows, the event logs play a more generalized role. They are used as a central clearinghouse for logging practically all activity that takes place on these machines, including login and logout activity, OS messages, security events, etc. Their role is analogous to the Unix syslog service we mentioned earlier. Using unpack() Perl has a function called unpack() especially designed to parse binary and structured data. Let’s take a look at how we might use it to deal with the wtmpx files. The for- mat of wtmp and wtmpx differs from Unix variant to Unix variant. For this specific example, we’ll look at the wtmpx file found on Solaris 10 and the Linux 2.6 wtmp. Here’s a plain-text translation of the first two records in a Solaris 10 wtmpx file: 0000000 d n b \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000040 t s / 1 p t s / 1 \0 \0 \0 \0 \0 \0 \0 0000060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000100 \0 \0 \0 \0 \0 \0 # 346 \0 007 \0 \0 \0 \0 \0 \0 0000120 D 9 . 253 \0 \t 313 234 \0 \0 \0 \0 \0 \0 \0 \0 0000140 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000160 \0 ' p o o l - 1 4 1 - 1 5 4 - 1 0000200 2 1 - 5 . b o s . e a s t . v e 0000220 r i z o n . n e t \0 \0 \0 \0 \0 \0 \0 0000240 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0000560 \0 \0 \0 T d n b \0 \0 \0 \0 \0 \0 \0 \0 \0 0000600 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000620 \0 \0 \0 \0 t s / 2 p t s / 2 \0 \0 \0 0000640 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000660 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 $ R \0 007 \0 \0 * And Mac OS X has, sigh, its own logging framework called the Apple System Log facility. It does keep /var/run/utmpx up-to-date at the same time, though. 378 | Chapter 10: Log Files 0000700 \0 \0 \0 \0 D 9 / 212 \0 016 L 315 \0 \0 \0 \0 0000720 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000740 \0 \0 \0 \0 \0 ' p o o l - 1 4 1 - 1 0000760 5 4 - 1 2 1 - 5 . b o s . e a s 0001000 t . v e r i z o n . n e t \0 \0 \0 0001020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001340 \0 \0 \0 \0 \0 \0 \0 T Unless you are already familiar with the structure of this file, that “ASCII dump” (as it is called) of the data probably looks like line noise or some other kind of semirandom garbage. So how do we become acquainted with this file’s structure? The easiest way to understand the format of this file is to look at the source code for programs that read and write to it. If you are not literate in the C language, this may seem like a daunting task. Luckily, we don’t actually have to understand or even look at most of the source code; we can just examine the portion that defines the file format. All of the operating system programs that read and write to the wtmp file get their file definitions from a single, short C include file, which is very likely to be found at /usr/include/utmp.h or utmpx.h. The part of the file we need to look at begins with a definition of the C data structure that will be used to hold the information. If you search for struct utmp {, you’ll find the portion we need. The next lines after struct utmp { define each of the fields in this structure. These lines should each be commented using the /* text */ C comment convention. Just to give you an idea of how different two versions of wtmpx can be, let’s compare the relevant excerpts on these two operating systems. Here’s an excerpt from Solaris 10’s utmpx.h: /* * This data structure describes the utmp *file* contents using * fixed-width data types. It should only be used by the implementation. * * Applications should use the getutxent(3c) family of routines to interact * with this database. */ struct futmpx { char ut_user[32]; /* user login name */ char ut_id[4]; /* inittab id */ char ut_line[32]; /* device name (console, lnxx) */ pid32_t ut_pid; /* process id */ int16_t ut_type; /* type of entry */ struct { int16_t e_termination; /* process termination status */ int16_t e_exit; /* process exit status */ } ut_exit; /* exit status of a process */ struct timeval32 ut_tv; /* time entry was made */ int32_t ut_session; /* session ID, user for windowing */ int32_t pad[5]; /* reserved for future use */ int16_t ut_syslen; /* significant length of ut_host */ Reading Binary Log Files | 379 char ut_host[257]; /* remote host name */ }; And here’s an excerpt from Linux 2.6’s bits/utmp.h: struct utmp { short int ut_type; /* Type of login. */ pid_t ut_pid; /* Process ID of login process. */ char ut_line[UT_LINESIZE]; /* Device name. */ char ut_id[4]; /* Inittab ID. */ char ut_user[UT_NAMESIZE]; /* Username. */ char ut_host[UT_HOSTSIZE]; /* Hostname for remote login. */ struct exit_status ut_exit; /* Exit status of a process marked as DEAD_PROCESS. */ /* The ut_session and ut_tv fields must be the same size when compiled 32- and 64-bit. This allows data files and shared memory to be shared between 32- and 64-bit applications. */ #if __WORDSIZE == 64 && defined __WORDSIZE_COMPAT32 int32_t ut_session; /* Session ID, used for windowing. */ struct { int32_t tv_sec; /* Seconds. */ int32_t tv_usec; /* Microseconds. */ } ut_tv; /* Time entry was made. */ #else long int ut_session; /* Session ID, used for windowing. */ struct timeval ut_tv; /* Time entry was made. */ #endif int32_t ut_addr_v6[4]; /* Internet address of remote host. */ char __unused[20]; /* Reserved for future use. */ }; These files provide all the clues we need to compose the necessary unpack() statement. unpack() takes a data format template as its first argument. It uses this template to determine how to disassemble the (usually) binary data it receives in its second argu- ment. unpack() will take apart the data as instructed, returning a list in which each element corresponds to an element of your template. Let’s construct our template piece by piece, based on the C structure from the Solaris utmpx.h include file. There are many possible template letters we can use. I’ve translated the ones we’ll use in Table 10-1, but you should check the pack() section of the perlfunc manual page for more information. Constructing these templates is not always straightforward; C compilers occasionally pad out values to satisfy alignment con- straints. The command pstruct that ships with Perl can often help with quirks like these. 380 | Chapter 10: Log Files Table 10-1. Translating the utmpx.h C code to an unpack() template C code unpack() template Template letter/repeat # translation char ut_user[32]; A32 ASCII string (space-padded), 32 bytes long char ut_id[4]; A4 ASCII string (space-padded), 4 bytes long char ut_line[32]; A32 ASCII string (space-padded), 32 bytes long pid32_t ut_pid; l A signed “long” value (4 bytes, which may not be the same as the size of a true long value on some machines) int16_t ut_type; s A signed “short” value struct { int16_t e_termination; s A signed “short” value int16_t e_exit; } ut_exit; s A signed “short” value x2 Compiler-inserted padding struct { time32_t tv_sec; l A signed “long” value int32_t tv_usec } ut_tv; l A signed “long” value int32_t ut_session; l A signed “long” value int32_t pad[5]; x20 Skip 20 bytes for padding int16_t ut_syslen; s A signed “short” value char ut_host[257]; Z257 ASCII string, null-terminated up to 257 bytes including \0 x Compiler-inserted padding Having constructed our template, let’s use it in a real piece of code: # template for Solaris 9/10 wtmpx my $template = 'A32 A4 A32 l s s s x2 l l l x20 s Z257 x'; my $recordsize = length( pack( $template, () ) ); open my $WTMP, '<', '/var/adm/wtmpx' or die "Unable to open wtmpx:$!\n"; my ($ut_user, $ut_id, $ut_line, $ut_pid, $ut_type, $ut_e_termination, $ut_e_exit, $tv_sec, $tv_usec, $ut_session, $ut_syslen, $ut_host, ) = (); # read wtmpx one record at a time my $record; Reading Binary Log Files | 381 while ( read( $WTMP, $record, $recordsize ) ) { # unpack it using our template ( $ut_user, $ut_id, $ut_line, $ut_pid, $ut_type, $ut_e_termination, $ut_e_exit, $tv_sec, $tv_usec, $ut_session, $ut_syslen, $ut_host ) = unpack( $template, $record ); # this makes the output more readable - the value 8 comes # from /usr/include/utmp.h: # #define DEAD_PROCESS 8 if ( $ut_type == 8 ) { $ut_host = '(exit)'; } print "$ut_line:$ut_user:$ut_host:" . scalar localtime($tv_sec) . "\n"; } close $WTMP; Here’s the output of this little program: pts/ Apr 16 06:37:44 2008 pts/176:vezt:(exit):Wed Apr 16 06:38:03 2008 pts/ Apr 16 08:09:27 2008 pts/ Apr 16 08:34:18 2008 pts/17:croche:(exit):Wed Apr 16 08:34:45 2008 pts/ Apr 16 09:45:57 2008 pts/139:hermd:(exit):Wed Apr 16 09:58:55 2008 One small comment on the code before we move on: read() takes a number of bytes to read as its third argument. Rather than hardcoding in a record size like “32”, we use a handy property of the pack() function. When handed an empty list, pack() returns a null or space-padded string the size of a record. This allows us to feed pack() an arbitrary record template and have it tell us how big a record it is: my $recordsize = length( pack( $template, () ) ); You Know You’re a Power User When... Of all the methods we’ll look at for accessing logging information, the unpack() method is the one with the greatest potential to leave you feeling like a power user. This is the one you’ll need to use if you find the other methods are failing due to data corruption. For example, I’ve heard of cases where the wtmpx file became damaged in a way that left the last executable just sputtering error messages. When that happens it’s some- times possible to write code that will skip the damaged part (via sysread() in concert with unpack()) and allow you to recover the rest of the file. You’ll definitely have earned your superhero cape the first time you succeed in one of these situations. The unpack() method is not the only intra-Perl method for accessing the wtmp/x data. At least one module uses the vendor-approved system calls (getutxent(), etc.) for read- ing these files. We’ll use that module in an example a little later. 382 | Chapter 10: Log Files Calling an OS (or Someone Else’s) Binary Sifting through wtmp files is such a common task that Unix systems ship with a com- mand called last for printing a human-readable dump of the binary file. Here’s some sample output showing approximately the same data as the output of our previous example: vezt pts/176 c-61-212-209-21. Wed Apr 16 06:37 - 06:38 (00:00) birnou pts/147 pool-50-29-232-8 Wed Apr 16 08:09 still logged croche pts/17 ce-23-213-189-15 Wed Apr 16 08:34 - 08:34 (00:00) hermd pts/139 d-66-249-250-270 Wed Apr 16 09:45 - 09:58 (00:12) We can easily call binaries like last from Perl. This code will show all the unique user- names found in our current wtmpx file: # location of the last command binary my $lastexec = '/bin/last'; open my $LAST, '-|', "$lastexec" or die "Unable to run $lastexec:$!\n"; my %seen; while(my $line = <$LAST>){ last if $line =~ /^$/; my $user = (split(' ', $line))[0]; print "$user\n" unless exists $seen{$user}; $seen{$user}=''; } close $LAST or die "Unable to properly close pipe:$!\n"; So why use this method when unpack() looked like it could serve all your needs? Port- ability. As you’ve seen, the format of the wtmp/x file differs from Unix variant to Unix variant. On top of this, a single vendor may change the format of wtmp/x between OS releases, rendering your perfectly good unpack() template invalid. However, one thing you can reasonably depend on is the continued presence of a last command that will read this format, independent of any underlying format changes. If you use the unpack() method, you have to create and maintain separate template strings for each different wtmp format you plan to parse.† The biggest disadvantage of using this method rather than unpack() is the increased sophistication of the field parsing you need to do in the program. With unpack(), all the fields are automatically extracted from the data for you. Using our last example, you may find yourself with split() or regular expression-resistant output like this, all in the same output: user console Wed Oct 14 20:35 - 20:37 (00:01) user pts/12 Wed Oct 14 09:19 - 18:12 (08:53) user pts/17 Tue Oct 13 13:36 - 17:09 (03:33) reboot system boot Tue Oct 6 14:13 † There’s a bit of hand waving going on here, since you still have to track where the last executable is found in each Unix environment and compensate for any differences in the format of each program’s output. Reading Binary Log Files | 383 Your eye has little trouble picking out the columns, but any program that parses this output will have to deal with the missing information in lines 1 and 4. unpack() can still be used to tease apart this output because it has fixed field widths, but that’s not always possible. There are other techniques for writing more sophisticated parsers, but that’s probably more work than you desire. Using the OS’s Logging API For this approach, let’s switch our focus to the Windows Event Log Service. As men- tioned earlier, Windows machines unfortunately do not log to plain-text files. The only supported way to get to the log file data is through a set of special API calls. Most users rely on the Event Viewer program, shown in Figure 10-1, to retrieve this data for them. Figure 10-1. The Windows Event Viewer Luckily, there is a Perl module (written by Jesse Dougherty and later updated by Martin Pauley and Bret Giddings) that allows easy access to the Event Log API calls.‡ We’ll walk through a more complex version of this program later in this chapter, but for now ‡ Log information in Windows can also be retrieved using the Window Management Instrumentation (WMI) framework we touched on in Chapter 4, but Win32::EventLog is easier to use and understand. If you need to parse Event Log data stored on a non-Windows machine, Parse::EventLog by John Eaglesham makes a valiant attempt. 384 | Chapter 10: Log Files here’s a simple program that dumps a listing of events in the System event log in a syslog- like format: use Win32::EventLog; # each event has a type - this is a translation of the common types my %type = (1 => 'ERROR', 2 => 'WARNING', 4 => 'INFORMATION', 8 => 'AUDIT_SUCCESS', 16 => 'AUDIT_FAILURE'); # if this is set, we also retrieve the full text of every # message on each Read() $Win32::EventLog::GetMessageText = 1; # open the System event log my $log = new Win32::EventLog('System') or die "Unable to open system log:$^E\n"; my $event = ''; # read through it one record at a time, starting with the first entry while ($log->Read((EVENTLOG_SEQUENTIAL_READ|EVENTLOG_FORWARDS_READ), 1,$entry)){ print scalar localtime($entry->{TimeGenerated}).' '; print $entry->{Computer}.'['.($entry->{EventID} & 0xffff).'] '; print $entry->{Source}.':'.$type{$entry->{EventType}}.': '; print $entry->{Message}; } Command-line utilities like last that dump event logs into plain ASCII format also exist for Windows. We’ll see one of these utilities in action later in this chapter, and shortly we’ll see an example of using the Unix equivalent of an OS logging API for wtmp data. Structure of Log File Data In addition to the format in which log files present their data, it is important to think about the contents of these files, because what the data represents and how it is repre- sented both contribute to our plan of attack when programming. With log file contents, often a distinction can be made between data that is stateful and data that is stateless. Let’s take a look at a couple of examples that will make this distinction clear. Here’s a three-line snippet from an Apache web server log. Each line represents a request answered by the web server: - - [13/Dec/2008:00:04:20 −0500] "GET home/u1/tux/ tuxedo05.gif HTTP/1.0" 200 18666 - - [13/Dec/2008:00:04:21 −0500] "GET home/u2/news.htm HTTP/1.0" 200 6748 - - [13/Dec/2008:00:04:22 −0500] "GET home/u1/bgc.jpg HTTP/1.1" 304 - Structure of Log File Data | 385 Here are a few lines from a printer daemon log file: Aug 14 12:58:46 warhol printer: cover/door open Aug 14 12:58:58 warhol printer: error cleared Aug 14 17:16:26 warhol printer: offline or intervention needed Aug 14 17:16:43 warhol printer: error cleared Aug 15 20:40:45 warhol printer: paper out Aug 15 20:40:48 warhol printer: error cleared In both cases, each line of the log file is independent of every other line in the file. We can find patterns or aggregate lines together to gather statistics, but there’s nothing inherent in the data that connects the log file entries to each other. Now consider some slightly doctored entries from a sendmail mail log: Dec 13 05:28:27 mailhub sendmail[26690]: FAA26690: from=, size=643, class=0, pri=30643, nrcpts=1, msgid=<>, proto=ESMTP, [] Dec 13 05:29:13 mailhub sendmail[26695]: FAA26695: from=, size=9600, class=0, pri=39600, nrcpts=1, msgid=<>, proto=ESMTP, [] Dec 13 05:29:15 mailhub sendmail[26691]: FAA26690: to=, delay=00:00:02, xdelay=00:00:01, mailer=local, stat=Sent Dec 13 05:29:19 mailhub sendmail[26696]: FAA26695: to="|IFS=' '&&exec /usr/bin/ procmail -f-||exit 75 #user", ctladdr=user (6603/104), delay=00:00:06, xdelay=00:00:06, mailer=prog, stat=Sent Unlike in the previous examples, there is a definite connection between the lines in this file. Figure 10-2 makes that connection explicit. Dec 13 05:28:27 mailhub sendmail[26690]: FAA26690: from=, size=643, class=0, pri=30643, nrcpts=1, msgid=<>, proto=ESMTP, [] Dec 13 05:29:13 mailhub sendmail[26695]: FAA26695: from=, size=9600, class=0,pri=39600 nrcpts=1,msgid=<>, proto=ESMTP, [192.168.16] Dec 13 05:29:15 mailhub sendmail[26691]: FAA26690: to=, delay=00:00:02, xdelay=00:00:01, mailer=local, stat=Sent Dec 13 05:29:29 mailhub sendmail[26696]: FAA26695: to="|IFS=' '&&exec /usr/bin/procmail -f-||exit 75 #user", ctladdr=user pri=30643, nrcpts=1, (6603/104), delay=00:00:06, xdelay=00:00:06, mailer=prog, stat=Sent Figure 10-2. Related entries in the sendmail log 386 | Chapter 10: Log Files Each line has at least one partner entry that shows the source and destination(s) of each message. When a message enters the system it is assigned a unique “Message-ID,” highlighted in the figure, which identifies that message while it is in play. This message ID allows us to associate related lines in an interleaved log file, essentially giving a message an existence or “state” in between entries in the log file. Sometimes we care about the “distance” between state transitions. Take, for instance, the wtmpx file we looked at earlier in this chapter: in that file we’re interested not only in when a user logs in and out (the two state transitions in the log), but also in the time between these two events (i.e., how long the user was logged in). The most sophisticated log files can add another twist. Here are some excerpts from a POP (Post Office Protocol) server’s log file while the server is in debug mode. The names and IP addresses have been changed to protect the innocent: Jan 14 15:53:45 mailhub popper[20243]: Debugging turned on Jan 14 15:53:45 mailhub popper[20243]: (v2.53) Servicing request from "client" at 129.X.X.X Jan 14 15:53:45 mailhub popper[20243]: +OK QPOP (version 2.53) at mailhub starting. Jan 14 15:53:45 mailhub popper[20243]: Received: "USER username" Jan 14 15:53:45 mailhub popper[20243]: +OK Password required for username. Jan 14 15:53:45 mailhub popper[20243]: Received: "pass xxxxxxxxx" Jan 14 15:53:45 mailhub popper[20243]: +OK username has 1 message (26627 octets). Jan 14 15:53:46 mailhub popper[20243]: Received: "LIST" Jan 14 15:53:46 mailhub popper[20243]: +OK 1 messages (26627 octets) Jan 14 15:53:46 mailhub popper[20243]: Received: "RETR 1" Jan 14 15:53:46 mailhub popper[20243]: +OK 26627 octets Jan 14 15:53:56 mailhub popper[20243]: Received: "DELE 1" Jan 14 15:53:56 mailhub popper[20243]: Deleting message 1 at offset 0 of length 26627 Jan 14 15:53:56 mailhub popper[20243]: +OK Message 1 has been deleted. Jan 14 15:53:56 mailhub popper[20243]: Received: "QUIT" Jan 14 15:53:56 mailhub popper[20243]: +OK Pop server at mailhub signing off. Jan 14 15:53:56 mailhub popper[20243]: (v2.53) Ending request from "user" at (client) 129.X.X.X Not only do we encounter connections (“Servicing request from...”) and disconnec- tions (“Ending request from...”), but we have information detailing what took place in between these state transitions. Each of the middle events also provides potentially useful “distance” information. If there was a problem with our POP server, we might look to see how long each step in the output took. In the case of an FTP server, you may be able to draw some conclusions from this data about how people interact with your site. On average, how long do people stay con- nected before they transfer files? Do they pause between commands for a long time? Do they always travel from one part of your site to another before downloading the same file? The interstitial data can be a rich source of information. Structure of Log File Data | 387 Dealing with Log File Information Once you’ve learned how to access your logging data programmatically, two important applications start begging to be addressed: logging information space management and log analysis. Let’s look at each in turn. Space Management of Logging Information The downside to having programs that can provide useful or verbose logging output is the amount of disk space this output can consume. This is a concern for all three op- erating systems covered in this book: Unix, Mac OS X, and Windows. Windows is probably the least troublesome of the lot, because its central logging facility has built- in autotrimming support. Usually, the task of keeping the log files down to a reasonable size is handed off to the system administrator. Most Unix vendors provide some sort of automated log size management mechanism with the OS, but it often handles only the select set of log files shipped with the machine. As soon as you add another service to a machine that creates a separate log file, it becomes necessary to tweak (or even toss) the vendor-supplied solution. Log rotation The usual solution to the space problem is to rotate the log files. (We’ll explore an unusual solution in the next subsection.) After a specific interval has passed or a certain file size has been reached, we rename the current log file (e.g., logfile becomes logfile.0). The logging process is then continued into an empty file. The next time the specified interval or limit is reached, we repeat the process, first renaming the original backup file (e.g., renaming logfile.0 to logfile.1) and then renaming the current log file to logfile.0. This process is repeated until a set number backup files have been created, at which point the oldest backup file is deleted. Figure 10-3 illustrates this process. This method allows us to keep on hand a reasonable, finite amount of log data. Ta- ble 10-2 provides one recipe for log rotation and the Perl functions needed to perform each step. Table 10-2. A recipe for log rotation in Perl Process Perl Move the older backup logs out of the way (i.e., move each one to a new name in the sequence). rename(), or File::Copy::move() if moving files across filesystems. If necessary, signal the process creating this particular log file to close the current file and cease logging to disk until told otherwise. kill() for programs that take signals; system() or `` (back- ticks) if another administrative program has to be called for this purpose. Copy or move the log file that was just in use to another file. File::Copy to copy, rename() to rename (or File::Copy::move() if moving files across filesystems). 388 | Chapter 10: Log Files Process Perl If necessary, truncate the current log file. truncate() or open my $FILE,'>','filename'. If necessary, signal the logging process to resume logging. See row 2 of this table. If desired, compress or post-process the copied file. system() or `` (backticks) to run a compression program; Compress::Zlib or other code for post-processing. Delete other, older log file copies. stat() to examine file sizes and dates; unlink() to delete files. logfile logfile logfile logfile logfile rotate rotate rotate rotate rotate logfile.0 logfile.0 logfile.0 logfile.0 rotate rotate rotate rotate logfile.0 logfile.1 logfile.1 rotate logfile.1 rotate logfile.1 rotate logfile.2 logfile.2 rotate logfile.2 rotate logfile.3 logfile.3 rotate logfile.n rotate POOF! Figure 10-3. A pictorial representation of log rotation There are many variations on this theme. Everyone and their aunt’s vendors have writ- ten their own scripts for log rotation. Thus, it should come as no surprise that there’s a Perl module to handle log rotation. Let’s look at Logfile::Rotate, by Paul Gampe. Logfile::Rotate uses the object-oriented programming convention of first creating a new log file object instance and then running a method of that instance. First, we create a new instance with the parameters found in Table 10-3. Table 10-3. Logfile::Rotate parameters Parameter Purpose File Name of log file to rotate Count (optional, default: 7) Number of backup files to keep around Gzip (optional, default: Perl’s default gzip executable name as found during the Perl build—must be in your path) Full path to gzip compression program executable Post Code to be executed after the rotation has been completed, as in row 5 of Table 10-2 Dealing with Log File Information | 389 Here’s some example code that uses these parameters: use Logfile::Rotate; my $logfile = new Logfile::Rotate( File => '/var/adm/log/syslog', Count => 5, Gzip => '/usr/local/bin/gzip', Post => sub { open my $PID, '<', '/etc/' or die "Unable to open pid file:$!\n"; chomp(my $pid = <$PID>); close $PID; kill 'HUP', $pid; } ); # Log file locked (really) and loaded. Now let's rotate it. $logfile->rotate(); # make sure the log file is unlocked (destroying object unlocks file) undef $logfile; The preceding code has three potential security flaws. See if you can pick them out before looking at the sidebar “Identifying and Fixing Insecure Code” for the answers and tips on how to avoid all three. Identifying and Fixing Insecure Code Now that you’ve pored over the Logfile::Rotate code looking for security holes, let’s talk about them. Since this module is often run by a privileged user (such as root), there are a few concerns: 1. The /usr/local/bin/gzip command will be run as that privileged user. We’ve done the right thing by calling the command with a full path (important!), but it be- hooves you to check just who has filesystem permissions to modify/replace that executable. One perhaps slightly safer way to sidestep this problem (presuming you retain total control over who can install Perl modules) is to change the line to Gzip => 'lib'. This causes Logfile::Rotate to call Compress::Zlib instead of call- ing out to a separate binary to do the compression. 2. In the Post section, the code happily reads /etc/ without seeing if that file could be tampered with by a malicious party. Is the file world-writable? Is it a link to something else? Does the right user own the file? Our code doesn’t care, but it should. It would be easy to check its permissions via stat() before proceeding. 3. In the same section, the code blithely sends a HUP signal to the PID number it read from the file just mentioned. It makes no attempt to determine if that process ID actually refers to a running syslog process. More defensive coding would check the process table first (perhaps with one of the process table listing strategies we discussed in Chapter 4) before sending the signal. 390 | Chapter 10: Log Files These are the most blatant problems with the code. Be sure to read the section on safe scripting in Chapter 1 for more thoughts on the matter. Circular buffering We’ve just discussed the traditional log rotation method for dealing with storage of ever-growing logs. Now let me show you a more unusual approach that you can add to your toolkit. Here’s a common scenario: you’re trying to debug a server daemon that provides a torrent of log output. You’re only interested in a small part of the total output, perhaps just the lines the server produces after you run some sort of test with a special client. Saving all of the log output to disk as usual would fill your disk quickly. Rotating the logs as often as would be needed with this volume of output would slow down the server. What do you do? I wrote a program called bigbuffy to deal with this conundrum. The approach is pretty straightforward. bigbuffy reads from its usual “standard” or “console” input one line at a time. These lines are stored in a circular buffer of a set size (see Figure 10-4). When the buffer is full, it starts filling from the top again. This read/store process continues until bigbuffy receives a signal from the user. Upon receiving this signal, it dumps the current contents of the buffer to a file and returns to its normal cycle. What’s left behind on disk is essentially a window into the log stream, showing just the data you need. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 É Figure 10-4. Logging to a circular buffer bigbuffy can be paired with a service-monitoring program like those found in Chap- ter 13. As soon as the monitor detects a problem, it can signal bigbuffy to dump its log Dealing with Log File Information | 391 buffer, leaving you with a snapshot of the log localized to the failure instance (assuming your buffer is large enough and your monitor noticed the problem in time). Here’s a simplified version of bigbuffy. The code is longer than the examples we’ve seen so far in this chapter, but it’s not very complex. We’ll use it in a moment as a spring- board for addressing some important issues, such as input blocking and security: use Getopt::Long; my @buffer; # buffer for storing input my $dbuffsize = 200; # default circular buffer size (in lines) my $whatline = 0; # start line in circular buffer my $dumpnow = 0; # flag to indicate dump requested # parse the options my ( $buffsize, $dumpfile ); GetOptions( 'buffsize=i' => \$buffsize, 'dumpfile=s' => \$dumpfile, ); $buffsize ||= $dbuffsize; # set up the signal handler and initialize a counter die "USAGE: $0 [--buffsize=] --dumpfile=" unless ( length($dumpfile) ); $SIG{'USR1'} = \&dumpnow; # set a signal handler for dump # and away we go! (with just a simple # read line-store line loop) while ( defined( $_ = <> ) ) { # Insert line into data structure. # Note: we do this first, even if we've caught a signal. # Better to dump an extra line than lose a line of data if # something goes wrong in the dumping process. $buffer[$whatline] = $_; # where should the next line go? $whatline = ++$whatline % $buffsize; # if we receive a signal, dump the current buffer if ($dumpnow) { dodump(); } } # simple signal handler that just sets an exception flag, # see perlipc(1) sub dumpnow { $dumpnow = 1; } 392 | Chapter 10: Log Files # dump the circular buffer out to a file, appending to file if # it exists sub dodump { my $line; # counter for line dump my $exists; # flag, does the output file exist already? my $DUMP_FH; # filehandle for dump file my ( @firststat, @secondstat ); # to hold output of lstats $dumpnow = 0; # reset the flag and signal handler $SIG{'USR1'} = \&dumpnow; if ( -e $dumpfile and ( ! -f $dumpfile or -l $dumpfile ) ) { warn 'ALERT: dumpfile exists and is not a plain file, '. "skipping dump.\n"; return undef; } # We have to take special precautions when we're doing an # append. The next set of "if" statements performs a set of # security checks while opening the file for appending. if ( -e $dumpfile ) { $exists = 1; unless ( @firststat = lstat $dumpfile ) { warn "Unable to lstat $dumpfile, skipping dump.\n"; return undef; } if ( $firststat[3] != 1 ) { warn "$dumpfile is a hard link, skipping dump.\n"; return undef; } } unless ( open $DUMP_FH, '>>', $dumpfile ) { warn "Unable to open $dumpfile for append, skipping dump:$!.\n"; return undef; } if ($exists) { unless ( @secondstat = lstat $DUMP_FH ) { warn "Unable to lstat opened $dumpfile, skipping dump.\n"; return undef; } if ( $firststat[0] != $secondstat[0] or # check dev num $firststat[1] != $secondstat[1] or # check inode $firststat[7] != $secondstat[7] # check sizes ) { warn "SECURITY PROBLEM: lstats don't match, skipping dump.\n"; return undef; } } $line = $whatline; print {$DUMP_FH} '-' . scalar(localtime) . ( '-' x 50 ) . "\n"; Dealing with Log File Information | 393 do { # print only valid lines in case buffer was not full print {$DUMP_FH} $buffer[$line] if defined $buffer[$line]; $line = ++$line % $buffsize; } until $line == $whatline; close $DUMP_FH; # zorch the active buffer to avoid leftovers # in future dumps $whatline = 1; @buffer = (); return 1; } A program like this can stir up interesting implementation issues. We’ll look at a few of them here. I mentioned earlier that this is a simplified version of bigbuffy. For ease of implementation, especially across platforms, this version has an unsavory characteristic: while dumping data to disk, it can’t continue reading input. During a buffer dump, the OS may tell the program sending output to bigbuffy to pause operation pending the drain of its output buffer. Luckily, the dump is fast, so the win- dow where this could happen is very small, but this is still less passive than you might like. Possible solutions to this problem include: • Rewriting bigbuffy to use a double-buffered, multitasking approach. Instead of using a single storage buffer, it would use two. Upon receiving the signal, the pro- gram would begin to log to a second buffer while a child process or another thread handled dumping the first buffer. At the next signal, the buffers would be swapped again. • Rewriting bigbuffy to interleave reading and writing while it is dumping. The sim- plest version of this approach would involve writing some number of lines to the output file each time a new line is read. This gets a bit tricky if the log output being read is “bursty” instead of arriving as constant flow, though, as you wouldn’t want to have to wait for a new line of output before you could receive the requested log buffer dump. You’d have to use some sort of timeout or internal clock mechanism to get around this problem. Both approaches are hard to pull off portably in a cross-platform environment, hence the simplified version shown in this book. You may have noticed that bigbuffy takes considerable care with the opening and writing of its output file. This is an example of the defensive coding style mentioned earlier, in the section “Log rotation” on page 388. If this pro- gram is to be used to debug server daemons, it is likely to be run by privileged users on Input blocking in log-processing programs. Security in log-processing programs. 394 | Chapter 10: Log Files a system. It is therefore important to think about unpleasant situations that might allow the program to be abused. One possible scenario would be swapping the link to the output file with a link to another file. If we opened and wrote to the file without checking its identity, we might find ourselves inadvertently stomping on an important file like /etc/passwd. Even if we check the output file before opening it, it might be possible for a malicious party to switch it on us before we begin writing to it. To avoid this scenario: • We check if the output file exists already. If it does, we lstat() it to get the file- system information. • We open the file in append mode. • Before we actually write to the file, we lstat() the open filehandle and check that it is still the same file we expect it to be and that it hasn’t been switched since we initially checked it. If it is not the same file (e.g., if someone swapped the file with a link right before the open), we do not write to the file and we complain loudly. This last step avoids the potential race condition mentioned in Chapter 1. If we didn’t have to append, we could instead open a temporary file with a randomized name (so it couldn’t be guessed ahead of time) and then rename the temporary file into place. Perl ships with Tim Jenness’s File::Temp module to help you do things like this. These sorts of gyrations are necessary on most Unix systems because Unix was not originally designed with security as a high priority. Windows also has “junctions,”* the rough equivalent of symbolic links, but I have yet to see any indication that they pose the same sort of security threat due to their implementation. Log Parsing and Analysis Some system administrators never get past the rotation phase in their relationships with their log files. As long as the necessary information exists on disk when it is needed for debugging, they never put any thought into using their log file information for any other purpose. I’d like to suggest that this is a shortsighted view, and that a little log file analysis can go a long way. We’re going to look at a few approaches you can use for performing log file analysis in Perl, starting with the most simple and getting more complex as we go along. Most of the examples in this section use Unix log files for demonstration purposes, since the average Unix system has ample logs just waiting to be analyzed, but the approaches offered here are not OS-specific. * You’ll also hear the term “reparse point” in this context, because Microsoft has been refining its terminology about these sorts of things over the course of several OS releases. At the time of this writing, junctions are considered to be created from reparse points. Dealing with Log File Information | 395 Stream read-count The easiest approach is the simple “read and count,” where we read through a stream of log data looking for interesting data, and increment a counter when we find it. Here’s a simple example that counts the number of times a machine has rebooted based on the contents of a Solaris 10 wtmpx file: # template for Solaris 10 wtmpx my $template = 'A32 A4 A32 l s s s x2 l l l x20 s Z257 x'; # determine the size of a record my $recordsize = length( pack( $template, () ) ); # open the file open my $WTMP, '<', '/var/adm/wtmpx' or die "Unable to open wtmpx:$!\n"; my ($ut_user, $ut_id, $ut_line, $ut_pid, $ut_type, $ut_e_termination, $ut_e_exit, $tv_sec, $tv_usec, $ut_session_pad, $ut_syslen, $ut_host ) = (); my $reboots = 0; # read through it one record at a time while ( read( $WTMP, $record, $recordsize ) ) { ( $ut_user, $ut_id, $ut_line, $ut_pid, $ut_type, $ut_e_termination, $ut_e_exit, $tv_sec, $tv_usec, $ut_session, $ut_syslen, $ut_host ) = unpack( $template, $record ); if ( $ut_line eq 'system boot' ) { print "rebooted " . scalar localtime($tv_sec) . "\n"; $reboots++; } } close $WTMP; print "Total reboots: $reboots\n"; Let’s extend this methodology and explore an example of statistics gathering using the Windows Event Log facility. As mentioned before, Windows has a well-developed and fairly sophisticated system-logging mechanism. This sophistication makes it a bit trickier for the beginning Perl programmer. We’ll have to use some Windows-specific Perl module routines to get at the basic log information. Windows programs and operating system components log their activities by posting “events” to one of several different event logs. The OS records in the log basic infor- mation such as when the event was posted, which program or OS function posted it, what kind of event it is (informational or something more serious), etc. 396 | Chapter 10: Log Files Unlike in Unix, the actual description of the event, or log message, is not actually stored with the event entry. Instead, an EventID is posted to the log. This EventID contains a reference to a specific message compiled into a program library (.dll). Retrieving a log message given an EventID is tricky. The process involves looking up the proper library in the registry and loading the library by hand. Luckily, the current version of Win32::EventLog performs this process for us automatically (see $Win32::EventLog::GetMessageText in our first Win32::Eventlog example, in the section “Using the OS’s Logging API” on page 384). For our next example, we’re going to generate some simple statistics on the number of entries currently in the System log, where they have come from, and their level of severity. We’ll write this program in a slightly different manner from how we wrote the first Windows logging example in this chapter. Our first step is to load the Win32::EventLog module, which contains the glue between Perl and the Windows event log routines. We then initialize a hash table that will be used to contain the results of our calls to the log-reading routines. Perl would normally take care of this for us, but sometimes it is good to add code like this for the benefit of others who will be reading the program. Finally, we set up a small list of event types that we will use later for printing statistics: use Win32::EventLog; # this is the equivalent of $event{Length => NULL, RecordNumber =>NULL, ...} my %event; my @fields = qw(Length RecordNumber TimeGenerated TimeWritten EventID EventType Category ClosingRecordNumber Source Computer Strings Data); @event{@fields} = (NULL) x @fields; # partial list of event types: Type 1 is "Error", # 2 is "Warning", etc. my @types = ('','Error','Warning','','Information"); Our next step is to open up the System event log. The Open() call places an EventLog handle into $EventLog that we can use as our connection to this particular log: my $EventLog = ''; # the handle to the event Log my $event = ''; # the event we'll be returning my $numevents = 0; # total number of events in log my $oldestevent = 0; # oldest event in the log Win32::EventLog::Open($EventLog,'System','') or die "Could not open System log:$^E\n"; Once we have this handle, we can use it to retrieve the number of events in the log and the ID of the oldest record: $EventLog->GetNumber($numevents); $EventLog->GetOldest($oldestevent); We use this information as part of our first Read() statement, which positions us at the place in the log right before the first record. This is the equivalent of seek()ing to the beginning of a file: Dealing with Log File Information | 397 $EventLog->Read( ( EVENTLOG_SEEK_READ | EVENTLOG_FORWARDS_READ ), $numevents + $oldestevent, $event ); From here on in, we use a simple loop to read each log entry in turn. The EVENTLOG_SEQUENTIAL_READ flag says “continue reading from the position of the last record read.” The EVENTLOG_FORWARDS_READ flag moves us forward in chronological order.† The third argument to Read() is the record offset: in this case it’s 0, because we want to pick up right where we left off. As we read each record, we record its Source and EventType in a hash table of counters: my %source; my %types; for ( my $i = 0; $i < $numevents; $i++ ) { $EventLog->Read( ( EVENTLOG_SEQUENTIAL_READ | EVENTLOG_FORWARDS_READ ), 0, $event ); $source{ $event->{Source} }++; $types{ $event->{EventType} }++; } # now print out the totals print "--> Event Log Source Totals:\n"; for ( sort keys %source ) { print "$_: $source{$_}\n"; } print '-' x 30, "\n"; print "--> Event Log Type Totals:\n"; for ( sort keys %types ) { print "$types[$_]: $types{$_}\n"; } print '-' x 30, "\n"; print "Total number of events: $numevents\n"; My results look like this: --> Event Log Source Totals: Application Popup: 4 BROWSER: 228 DCOM: 12 Dhcp: 12 EventLog: 351 Mouclass: 6 NWCWorkstation: 2 Print: 27 Rdr: 12 RemoteAccess: 108 SNMP: 350 Serial: 175 Service Control Manager: 248 Sparrow: 5 Srv: 201 msbusmou: 162 † Here’s another place where the Win32 event log routines are more flexible than usual. Our code could have moved to the end of the log and read backward in time if we wanted to do that for some reason. 398 | Chapter 10: Log Files msi8042: 3 msinport: 162 mssermou: 151 qic117: 2 ------------------------------ --> Event Log Type Totals: Error: 493 Warning: 714 Information: 1014 ------------------------------ Total number of events: 2220 As promised, here’s some sample code that relies on a last-like program to dump the contents of the event log. It uses a program called ElDump by Jesper Lauritsen, which you can download from ElDump is similar to DumpEl, which can be found in several of the resource kits (and online at http://www my $eldump = 'c:\bin\eldump'; # path to ElDump # output data field separated by ~ and without full message # text (faster) my $dumpflags = '-l system -c ~ -M'; open my $ELDUMP, '-|', "$eldump $dumpflags" or die "Unable to run $eldump:$!\n"; print 'Reading system log.'; my ( $date, $time, $source, $type, $category, $event, $user, $computer ); while ( defined ($_ = <$ELDUMP>) ) { ( $date, $time, $source, $type, $category, $event, $user, $computer ) = split('~'); $$type{$source}++; print '.'; } print "done.\n"; close $ELDUMP; # for each type of event, print out the sources and number of # events per source foreach $type (qw(Error Warning Information AuditSuccess AuditFailure)) { print '-' x 65, "\n"; print uc($type) . "s by source:\n"; for ( sort keys %$type ) { print "$_ ($$type{$_})\n"; } } print '-' x 65, "\n"; Here’s a snippet from the output: ERRORs by source: BROWSER (8) Dealing with Log File Information | 399 Cdrom (2) DCOM (15) Dhcp (2524) Disk (1) EventLog (5) RemoteAccess (30) Serial (24) Service Control Manager (100) Sparrow (2) atapi (2) i8042prt (4) ----------------------------------------------------------------- WARNINGs by source: BROWSER (80) Cdrom (22) Dhcp (76) Print (8) Srv (82) A simple stream read-count variation A simple variation of the stream read-count approach involves making multiple passes through the data. This is sometimes necessary for large data sets and cases where it takes an initial scan to determine the difference between interesting and uninteresting data. Programmatically, this means that after the first pass through the input, you do one of the following: • Move back to the beginning of the data stream (which could just be a file) using seek() or an API-specific call. • Close and reopen the filehandle. This is often the only choice when you are reading the output of a program like last. Here’s an example where a multiple-pass read-count approach might be useful. Imagine you have to deal with a security breach where an account on your system has been compromised. One of the first questions you might want to ask is, “Has any other account been compromised from the same source machine?” Finding a comprehensive answer to this seemingly simple question turns out to be trickier than you might expect. Let’s take a first shot at the problem. This code takes the name of a user as its first argument and an optional regular expression as a second argument for filtering out hosts we wish to ignore: use Perl6::Form; use User::Utmp qw(:constants); my ( $user, $ignore ) = @ARGV; my $format = '{<<<<<<<} {<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<} {<<<<<<<<<<<<<<<<<<<<<<<}'; User::Utmp::utmpxname('/var/adm/wtmpx'); print "-- scanning for first host contacts from $user --\n"; 400 | Chapter 10: Log Files my %contacts = ();# hostnames that have contacted us for specified user while ( my $entry = User::Utmp::getutxent() ) { if ( $entry->{ut_user} eq $user ) { next if ( defined $ignore and $entry->{ut_host} =~ /$ignore/o ); if ( $entry->{ut_type} == USER_PROCESS and !exists $contacts{ $entry->{ut_host} } ) { $contacts{ $entry->{ut_host} } = $entry->{ut_time}; print form $format, $entry->{ut_user}, $entry->{ut_host}, scalar localtime( $entry->{ut_time} ); } } } print "-- scanning for other contacts from those hosts --\n"; User::Utmp::setutxent(); # reset to start of database while ( my $entry = User::Utmp::getutxent() ) { # if it is a user's process, and we're looking for this host, # and this is a connection from a user *other* than the # compromised account, then output this record if ( $entry->{ut_type} == USER_PROCESS and exists $contacts{ $entry->{ut_host} } and $entry->{ut_user} ne $user ) { print form $format, $entry->{ut_user}, $entry->{ut_host}, scalar localtime( $entry->{ut_time} ); } } User::Utmp::endutxent(); # close database (not strictly necessary) First the program scans through the wtmpx data looking for all logins from the com- promised user. As it finds them, it compiles a hash of all the hosts from which these logins took place. It then resets the database so the next scan will start at the beginning of the file. The second scan looks for connections from that host list, printing matches as it finds them. We could easily modify this program to scan all of the files in a directory of rotated wtmp log files. To do that, we’d just have to loop over the list, calling User::Utmp::utmpxname() with each filename in turn. One problem with this program is that it is too specific. That is, it will only match exact hostnames. If the intruder is coming in from an ISP’s pool of DSL or cable modem addresses (which they often are), chances are the hostnames could change with each connection. Still, partial solutions like this often help a great deal. Besides its simplicity, the stream read-count approach we’ve been discussing has the advantage of being faster and less memory-intensive than the method we’ll consider next. It works well with the stateless type of log files we discussed early on in the chapter. But sometimes, especially when dealing with stateful data, we need to use a different plan of attack. Dealing with Log File Information | 401 Read-remember-process The opposite extreme of our previous approach, where we passed by the data as fast as possible, is to read it into memory and deal with it after reading. Let’s look at a few versions of this strategy. First, an easy example: let’s say you have an FTP transfer log and you want to know which files have been transferred the most often. Here are some sample lines from a wu-ftpd FTP server transfer log. Blank lines have been added to make it easier to see where these long lines begin and end: Sun Dec 27 05:18:57 2008 1 11868 /net/ a _ o a ftp 0 * Sun Dec 27 05:52:28 2008 25 269273 /CPAN/doc/FAQs/FAQ/PerlFAQ.html a _ o a mozilla@ ftp 0 * Sun Dec 27 06:15:04 2008 1 11868 /CPAN/MIRRORING.FROM b _ o a ftp 0 * Sun Dec 27 06:15:05 2008 1 35993 /CPAN/RECENT.html b _ o a ftp 0 * Table 10-4 lists the fields in each line of the output (please see the wu-ftpd server man- page xferlog(5) for details on each field). Table 10-4. Fields in a wu-ftpd server transfer log Field # Field name 0 current-time 1 transfer-time (in seconds) 2 remote-host 3 filesize 4 filename 5 transfer-type 6 special-action-flag 7 direction 8 access-mode 9 username 10 service-name 11 authentication-method 12 authenticated-user-id 402 | Chapter 10: Log Files Here’s some code to show which files have been transferred most often: my $xferlog = '/var/adm/log/xferlog'; my %files = (); open my $XFERLOG, '<', $xferlog or die "Unable to open $xferlog:$!\n"; while (defined ($line = <$XFERLOG>)){ $files{(split(' ',$line))[8]}++; } close $XFERLOG; for (sort {$files{$b} <=> $files{$a}||$a cmp $b} keys %files){ print "$_:$files{$_}\n"; } We read each line of the file, using the name of the file as a hash key and incrementing the value for that key. The name of the file is extracted from each log line using an array index that references a specific element of the list returned by the split() function:‡ $files{(split)[8]}++; You may notice that the specific element we reference (8) is different from the eighth field listed in Table 10-4. This is an unfortunate result of the lack of field delimiters in the original file. We are splitting on whitespace (the default for split()), so the date field becomes five separate list items. One subtle trick in this code sample is in the anonymous sort function we use to sort the values: for (sort {$files{$b} <=> $files{$a}||$a cmp $b} keys %files){ Note that the places of $a and $b have been switched from their alphabetical order in the first portion. This causes sort to return the items in descending order, thus showing us the more frequently transferred files first. The second portion of the anonymous sort function (||$a cmp $b) assures that we list files with the same number of transfers in a sorted order. If we wanted to limit this script to counting only certain files or directories, we could let the user specify a regular expression as the first argument to this script. For example, adding this: next unless /$ARGV[0]/o; to the while loop allows you to specify a regular expression to limit which log lines will be counted. ‡ Just FYI, a split() is likely to break if you have filenames with whitespace in them. In that case you’d probably need to use a regexp to take apart the line instead. Dealing with Log File Information | 403 Regular Expressions Crafting regular expressions is often one of the most important parts of log parsing. Regexps are used like programmatic sieves to extract the interesting data from the uninteresting data in the logs. The regular expressions used in this chapter are very basic, but you’ll probably be creating more sophisticated regexps for your own use. To use them more efficiently, you may wish to use the regular expression techniques introduced in Chapter 8. One of the best resources for learning about regular expressions is Jeffrey Friedl’s book, Mastering Regular Expressions ( (O’Reilly). Any time you spend learning how to wield the power of regexps will benefit you in many ways. Let’s take a look at another example of the read-remember-process approach, using our “breach-finder” program from the previous section. Our earlier code only showed us successful logins from the intruder sites. What if we want to find out about unsuc- cessful attempts? For that information, we’re going to have to bring in another log file. This scenario exposes one of Unix’s flaws: Unix systems tend to store log information in a number of different places and formats. Few tools are provided for dealing with these disparities (luckily, we have Perl). It is not uncommon to need more than one data source to solve problems like this one. The log file that will be the most help to us in this endeavor is the one generated through syslog by Wietse Venema’s Unix security tool tcpwrappers. tcpwrappers pro- vides gatekeeper programs and libraries that can be used to control access to network services. An individual network service like telnet can be configured so that a tcpwrap- pers program handles all network connections. When a connection attempt is made, the tcpwrappers program will syslog it and then either pass the connection off to the real service or take some action (like dropping the connection). The choice of whether to let the connection through is based on some simple user-provided rules (e.g., allow connections from only certain hosts). tcpwrappers can also take preliminary precau- tions to make sure the connection is coming from the place it purports to come from using a DNS reverse-lookup. It can even be configured to log the name of the user who made the connection (via the RFC 931 ident protocol) if possible. For a more detailed description of tcpwrappers, see Simson Garfinkel, Gene Spafford, and Alan Schwartz’s book Practical Unix & Internet Security ( (O’Reilly). For our purposes, we can just add some code to our previous breach-finder program that scans the tcpwrappers log (tcpdlog in this case) for connections from the suspect hosts we found in our scan of wtmp. If we add the following code to the end of our previous code sample: 404 | Chapter 10: Log Files # tcpd log file location my $tcpdlog = '/var/log/tcpd/tcpdlog'; print "-- connections found in tcpdlog --\n"; open my $TCPDLOG, '<', $tcpdlog or die "Unable to read $tcpdlog:$!\n"; my ( $connecto, $connectfrom ); while ( defined( $_ = <$TCPDLOG> ) ) { next if !/connect from /; # we only care about connections ( $connecto, $connectfrom ) = /(.+):\s+connect from\s+(.+)/; $connectfrom =~ s/^.+@//; print if ( exists $contacts{$connectfrom} and $connectfrom !~ /$ignore/o ); } close $TCPDLOG; we get output that looks like this: -- first host contacts from baduser -- user Thu Apr 3 13:41:47 2008 -- other connects from suspect machines -- user2 Thu Oct 9 17:06:49 2008 user2 Thu Oct 9 17:44:31 2008 user2 Fri Oct 10 22:00:41 2008 user2 Wed Oct 15 07:32:50 2008 user2 Wed Oct 22 16:24:12 2008 -- connections found in tcpdlog -- Jan 12 13:16:29 host2 in.rshd[866]: connect from Jan 13 14:38:54 host3 in.rlogind[4761]: connect from Jan 15 14:30:17 host4 in.ftpd[18799]: connect from Jan 16 19:48:19 host5 in.ftpd[5131]: connect from You may have noticed that this output contains connections from two different time ranges: we found connections in wtmpx from April 3 to October 22, while the tcpwrappers data appeared to show only January connections. The difference in these dates is an indication that our wtmpx files and our tcpwrappers files are rotated at different speeds. You need to be aware of these details when writing code that tacitly assumes the two log files being correlated refer to the same time period. For a final and more sophisticated example of the read-remember-process approach, let’s look at a task that requires combining stateful and stateless data. If you wanted a more comprehensive picture of the activity on a wu-ftpd server, you might want to use code to correlate the login and logout activity logged in a machine’s wtmp file with the file transfer information recorded by wu-ftpd in its xferlog file. It might be nice if you could see output that showed when an FTP session started and finished, and what transfers took place during that session. Here’s a snippet of sample output from the code we’re about to assemble. It shows four FTP sessions in March. The first session shows one file being transferred to the machine, the next two show files being transferred from that machine, and the last shows a connection without any transfers: Dealing with Log File Information | 405 Thu Mar 12 18:14:30 2008-Wed Mar 12 18:14:38 2008 -> /home/dnb/makemod Sat Mar 14 23:28:08 2008-Fri Mar 14 23:28:56 2008 <- /home/dnb/.emacs19 Sat Mar 14 23:14:05 2008-Fri Mar 14 23:34:28 2008 <- /home/dnb/lib/emacs19/cperl-mode.el <- /home/dnb/lib/emacs19/filladapt.el Wed Mar 25 21:21:15 2008-Tue Mar 25 21:36:15 2008 (no transfers in xferlog) Producing this output turns out to be nontrivial, since we need to pigeonhole stateless data into a stateful log. The xferlog transfer log shows only the time and the host that initiated the transfer. The wtmpx log shows the connections and disconnections from other hosts to the server. Let’s walk through how to combine the two types of data using a read-remember-process approach. First, we’ll define some variables for the program and load some supporting modules: use Time::Local; # for date->Unix time (secs from Epoch) conversion use User::Utmp qw(:constants); use Readonly; # to create read-only constants for legibility # location of transfer log my $xferlog = '/var/log/xferlog'; # location of wtmpx my $wtmpx = '/var/adm/wtmpx'; # month name to number mapping my %month = qw{Jan 0 Feb 1 Mar 2 Apr 3 May 4 Jun 5 Jul 6 Aug 7 Sep 8 Oct 9 Nov 10 Dec 11}; Now let’s look at the procedure that reads the wu-ftpd xferlog log file: # scans a wu-ftpd transfer log and populates the %transfers # data structure print 'Scanning $xferlog...'; open my $XFERLOG, '<', $xferlog or die "Unable to open $xferlog:$!\n"; # fields we will parse from the log my ( $time, $rhost, $fname, $direction ); my ( $sec, $min, $hours, $mday, $mon, $year ); my $unixdate; # the converted date my %transfers; # our data structure for holding transfer info while (<$XFERLOG>) { # using an array slice to select the fields we want ( $mon, $mday, $time, $year, $rhost, $fname, $direction ) = (split)[ 1, 2, 3, 4, 6, 8, 11 ]; $fname =~ tr/ -~//cd; # remove "bad" chars $rhost =~ tr/ -~//cd; # remove "bad" chars 406 | Chapter 10: Log Files # 'i' is "transferred in" $fname = ( $direction eq 'i' ? '-> ' : '<- ') . $fname; # convert the transfer time to Unix epoch format ( $hours, $min, $sec ) = split( ':', $time ); $unixdate = timelocal( $sec, $min, $hours, $mday, $month{$mon}, $year ); # put the data into a hash of lists of lists, i.e.: # $transfers{hostname} = ( [time1, $filename1], # [time2, $filename2], # ...) push( @{ $transfers{$rhost} }, [ $unixdate, $fname ] ); } close $XFERLOG; print "done.\n"; Three lines of Perl in the previous code deserve a little explanation. The first two are: $fname =~ tr/ -~//cd; # remove "bad" chars $rhost =~ tr/ -~//cd; # remove "bad" chars This is just a primitive attempt to prevent nasty things from showing up in our output later in the program. This line strips control characters from the filename, so if a user has transferred a file with a funky name (either unintentionally or maliciously) we don’t have to suffer later when we go to print the name and our terminal program freaks out. You’ll see a similar cleanup take place to the hostnames we read in later in the code. If we wanted to be really thorough (and we should want this), we could write a regular expression to accept only “legitimate” filenames and use what it captured. But for now, this will do. A more complicated piece of Perl is the push() statement: push( @{ $transfers{$rhost} }, [ $unixdate, $fname ] ); This line creates a hash of lists of lists that looks something like this: $transfers{hostname} = ([time1, filename1], [time2, filename2],[time3, filename3]...) The %transfers hash is keyed on the name of the host that initiated the transfer. For each host, we store a list of transfer pairs, each pair recording when a file was transferred and the name of that file. We’re choosing to store the time in “seconds since the epoch”* for ease of comparison later. The subroutine timelocal() from the module Time::Local helps us convert to that standard. Because we’re scanning a file transfer log written in chronological order, these lists of pairs are built in chronological order as well (a property that will come in handy later). * This is seconds since some arbitrary starting point. For example, the epoch on Unix machines is 00:00:00 GMT on January 1, 1970. Dealing with Log File Information | 407 Let’s move on to scanning wtmpx: # scans the wtmpx file and populates the @sessions structure with ftp sessions my ( %connections, @sessions ); print "Scanning $wtmpx...\n"; User::Utmp::utmpxname($wtmpx); while ( my $entry = User::Utmp::getutxent() ) { next if ( $entry->{ut_id} ne 'ftp' ); # ignore non-ftp sessions # "open" connection record using a hash of lists of lists (where the LoL # is used like a a stack stored in a hash, keyed on the device name) if ( $entry->{ut_user} and $entry->{ut_type} == USER_PROCESS ) { $entry->{ut_host} =~ tr/ -~//cd; # remove "bad" chars push( @{ $connections{ $entry->{ut_line} } }, [ $entry->{ut_host}, $entry->{ut_time} ] ); } # found close connection entry, try to pair with open if ( $entry->{ut_type} == DEAD_PROCESS ) { if ( !exists $connections{ $entry->{ut_line} } ) { warn "found lone logout on $entry->{ut_line}:" . scalar localtime( $entry->{ut_time} ) . "\n"; next; } # create a list of sessions, where each session is represented by # a list of this form: (hostname, login, logout) push( @sessions, [ @{ shift @{ $connections{ $entry->{ut_line} } } }, $entry->{ut_time} ] ); # if there are no more connections under that tty, remove it from hash delete $connections{ $entry->{ut_line} } unless ( @{ $connections{ $entry->{ut_line} } }); } } User::Utmp::endutxent(); print "done.\n"; Let’s look at what’s going on in this code. We read through wtmpx one record at a time. If the current record takes place on the special device name ftp, we know that this is an FTP session. Given an entry in the wtmpx database for ftp, we see if it describes the opening (ut_type is USER_PROCESS) or closing (ut_type is DEAD_PROCESS) of an FTP session. If it describes the opening of a connection, we record that info in a data structure that keeps tabs on all the open sessions, called %connections. Like %transfers in our previous 408 | Chapter 10: Log Files subroutine, it is a hash of lists of lists, this time keyed on the device (i.e., tty/pty) of each connection. Each of the values in this hash is a set of pairs detailing the name of the host from which the connection originated and the time. Why use such a complicated data structure to keep track of the open connections? Unfortunately, there isn’t a simple “open-close open-close open-close” pairing of lines in wtmpx. For instance, take a look at these lines from wtmpx (as printed by our first wtmpx program earlier in this chapter): Mar 27 14:04:47 2008 ttyp7:dnb:(exit):Thu Mar 27 14:05:11 2008 Mar 27 14:05:20 2008 ftpd1833:dnb:(exit):Thu Mar 27 14:06:20 2008 ftpd1833:dnb:(exit):Thu Mar 27 14:06:43 2008 Notice the two open FTP connection records on the same device (lines 1 and 3). If we just stored a single connection per device in a plain hash, we’d lose the first connection record when we found the second one. Instead, we use the list of lists keyed off every device in %connections as a stack. When we see a connection opening, we add a (host, login-time) pair for the connection to the stack kept for that device. Each time we see a close connection line for this device, we “pop” one of the open connection records off the stack and store our complete information about the session as a whole in another data structure called @sessions. That’s the purpose of this statement: push( @sessions, [ @{ shift @{ $connections{ $entry->{ut_line} } } }, $entry->{ut_time} ] ); Let’s untangle this statement from the inside out to make sure everything is clear. The part in bold type returns a reference to the stack/list of open connection pairs for a specific device (ut_line): push( @sessions, [ @{ shift @{ $connections{ $entry->{ut_line} } } }, $entry->{ut_time} ] ); This pops the reference to the first connection pair off that stack: push( @sessions, [ @{ shift @{ $connections{ $entry->{ut_line} } } }, $entry->{ut_time} ] ); Dealing with Log File Information | 409 We dereference it to get at the actual (host, login-time) connection pair list. If we place this pair at the beginning of another list that ends with the connection time, Perl will interpolate the connection pair and we’ll have a single, three-element list. This gives us a triad of (host, login-time, logout-time): push( @sessions, [ @{ shift @{ $connections{ $entry->{ut_line} } } }, $entry->{ut_time} ] ); Now that we have all of the parts of an FTP session (initiating host, connection start time, and end time) in a single list, we can push a reference to a new anonymous array containing that list on to the @sessions list of lists for future use: push( @sessions, [ @{ shift @{ $connections{ $entry->{ut_line} } } }, $entry->{ut_time} ] ); We have a list of sessions thanks to this one very busy statement. To finish the job, we check if the stack is empty for a device (i.e., if there are no more open connection requests pending). If this is the case, we can delete that device’s entry from the hash, as we know the connection has ended: delete $connections{ $entry->{ut_line} } unless ( @{ $connections{ $entry->{ut_line} } }); Now it’s time to do the actual correlation between our two data sets. For each session, we want to print out the connection triad, and then the files transferred during that session: # constants to make the connection triad data structure more readable; # the list consists of ($HOSTNAME,$LOGIN,$LOGOUT) in those positions Readonly my $HOSTNAME => 0; Readonly my $LOGIN => 1; Readonly my $LOGOUT => 2; # iterate over the session log, pairing sessions with transfers foreach my $session (@sessions) { # print session times print scalar localtime( $session->[$LOGIN] ) . '-' . scalar localtime( $session->[$LOGOUT] ) . ' ' . $session->[$HOSTNAME] . "\n "; # returns all of the files transferred for a given connect session # easy case, no transfers in this login if ( !exists $transfers{ $session->[$HOSTNAME] } ) { 410 | Chapter 10: Log Files print " \t( no transfers in xferlog ) \n "; next; } # easy case, first transfer we have on record is after this login if ( $transfers{ $session->[$HOSTNAME] }->[0]->[0] > $session->[$LOGOUT] ) { print " \t( no transfers in xferlog ) \n "; next; } my (@found) = (); # to hold the transfers we find per each session # find any files transferred in this session foreach my $transfer ( @{ $transfers{ $session->[$HOSTNAME] } } ) { # if transfer happened before login next if ( $transfer->[0] < $session->[$LOGIN] ); # if transfer happened after logout next if ( $transfer->[0] > $session->[$LOGOUT] ); # if we've already reported on this entry next if ( !defined $transfer->[1] ); # record that transfer and mark as used by undef'ing the filename push( @found, " \t " . $transfer->[1] . " \n " ); undef $transfer->[1]; } print( scalar @found ? @found : " \t( no transfers in xferlog ) \n" ) . " \n "; } The code starts by eliminating the easy cases: if we haven’t seen any transfers initiated by this host, or if the first transfer associated with this host occurs after the session triad we are checking has ended, we know no files have been transferred during this session. If we can’t eliminate the easy cases, we need to look through our lists of transfers. We check each transfer made from the host in question to see if it occurred after the session started but before the session ended. If either of these conditions isn’t true, we skip to the next transfer. Also, as soon as we’ve found a transfer that takes place after the session has ended, we avoid testing the other transfers for the host. Remember I mentioned that all of the transfers are added to the data structure in chronological order? Here’s where that pays off. The last test we make before considering a transfer entry to be valid may look a little peculiar: # if we've already used this entry next if ( !defined $transfer->[1] ); If two anonymous FTP sessions from the same host overlap in time, we have no way of knowing which session is responsible for initiating the transfer of any files uploaded Dealing with Log File Information | 411 or downloaded during that window. There is no information in either of our logs that can help us make that determination. The best we can do in this case is make up a standard and keep to it. The standard used here is “attribute the transfer to the first session possible.” The preceding test line and the subsequent undefing of the filename value as a flag enforce that standard. If this final test passes, we declare victory and add the filename to the list of files trans- ferred in the current session (@found). The session and its accompanying file transfers are then printed. Read-remember-process programs that have to do this sort of correlation can get fairly sophisticated, especially when they are bringing together data sources where the cor- relation is a bit fuzzy. So, in good Perl spirit, let’s see if we can take an easier approach. Black boxes In the Perl world, if you are trying to write something generally useful, it’s always possible that another person has beaten you to it and published his code for the task. In that case, you can simply feed your data into that person’s module in a prescribed way and receive results without having to know how the task was actually performed. This is often known as a “black box approach.” This approach can have its perils, though, so be sure to heed the warning in the following sidebar. Congratulations! You Are the New Module Maintainer! Though I tend to go for the black box approach more often than not, it is not without its perils. Let me tell you a cautionary tale. In the first edition of this book I gushed about a module called SyslogScan. This was a swell module for the parsing of syslog with especially good support for the mail logs the sendmail mail transfer agent produced. It handled the drudgework of parsing a raw sendmail log and pairing up the two log lines associated with the handling of a single mail message. It provided a lovely, simple interface for iterating through the log file one message at a time. These iterators could then be handed to other parts of the package, and it would produce summary reports and summary objects. Those objects could in turn be handed to yet another part of the package, and even more impressive reports would be generated. It was beautiful. But at some point, the developers of sendmail made a few small changes to the format of their log file. SyslogScan ceased being able to parse the log file as well as it did before. In time, it stopped working entirely. In most cases this sort of change wouldn’t be too much of a hassle, because the module author would notice the problem and issue a new release to address the log format change. Unfortunately, the author of SyslogScan seems to have disappeared from the Perl world some time in 1997. And that’s where the module sits as of this writing on CPAN: frozen in time and broken. If you depended on the module after the log format change, you had three choices: 412 | Chapter 10: Log Files 1. Start using another module (perhaps not viable if this was the only module for that purpose). 2. Write your own replacement module (could be lots of work). 3. Try to patch SyslogScan yourself to deal with the format change. Of the three choices, #3 probably involves the least work. Chances are the changes necessary to get it working again are small. But from this point on, congratulations, you are now the maintainer of the module (at least for your small world)! If it breaks again for some reason, the onus will be on you to fix it again. This may not be a big deal for you, but it is a potential drawback worth knowing about before you commit to relying on somebody else’s code. One of the strengths of the Perl community is its generosity in sharing code. There are many log-parsing modules available on CPAN. Most of them are designed to perform very specific tasks. For example, the Log::Procmail module by Philippe “BooK” Bruhat makes iterating through the log produced by the procmail mail filter and parsing it as we go easy. To print a list of addresses we received mail from and where each of those messages were filed, we can just write code like this: use Log::Procmail; my $procl = new Log::Procmail '/var/log/procmail'; while (my $entry = $procl->next){ print $entry->from . ' => ' . $entry->folder . "\n"; } There are a number of Apache log file parsers (for example, Apache::ParseLog, Parse::AccessLogEntry, and Apache::LogRegex) that perform similar heavy lifting for that log format. Several modules are also available for building your own special-purpose parsers. Some of these are themselves more “black box” than others. On the Unix side of the house, Parse::Syslog continues to be a good black-box choice for taking apart syslog-style lines. As an added spiffy feature, Parse::Syslog’s new() method will also take a File::Tail object instead of just your average, boring filehandle. Given this object, Parse::Syslog will operate on a log file that is still being written to, like so: use File::Tail; use Parse::Syslog; my $file = File::Tail->new( name => '/var/log/mail/mail.log' ); my $syslg = Parse::Syslog->new( $file ); while ( my $parsed = $syslg->next ) { print $parsed->{host} . ':' . $parsed->{program} . ':' . $parsed->{text} . "\n"; } Dealing with Log File Information | 413 If you’d like to build a parser using more basic building blocks, you may want to look at the set of modules that help in the construction of regular expressions. For exam- ple, Dmitry Karasik’s Regexp::Log::DateRange module helps you construct the gnarly regular expression necessary for selecting a date range in syslog files: use Regexp::Log::DateRange; # construct a regexp for May 31 8:00a to May 31 11:00a my $regexp = Regexp::Log::DateRange->new('syslog', [ qw(5 31 8 00) ], [ qw(5 31 11 00) ]); # $regexp now contains: 'may\s+31\s+(?:(?:0?[8-9]|10)\:|11\:0?0\:)' # compile that regular expression for better performance $regexp = qr/$regexp/i; # now use that regexp if ($input =~ /$regexp/) { print "$input matched\n" }; If you want to go up one level of meta, Philippe “BooK” Bruhat’s Regexp::Log module allows you to build other modules that build regular expressions for you. The easiest way to see how these derived modules function is to look at one of the modules built using it. Regexp::Log::Common, a parser module for the Common Log Format (used by packages like Apache) by Barbie, is a good example of a derived module. Here’s how a derived module like Regexp::Log::Common is used: use Regexp::Log::Common; my $rlc = Regexp::Log::Common->new( format => ':extended' ); $rlc->capture( qw(:none referer) ); my $regexp = $rlc->regexp; # now we have a regexp that will capture the referer field # from each line in the Extended Common Log Format # as in # ($referer) = $logline =~ /$regexp/ After loading the module, we tell it that we will be dealing with a file that has lines following the Extended Common Log Format. (:extended is just a shortcut for speci- fying all of field names found in that format; we could have listed them by hand if we really wanted.) We then tell the module which of these fields we want to capture using capture(). capture() may look like a simple method call to set the list of fields to capture, but it actually adds those fields to the current capture list. This list starts off defaulting to the entire set of fields, so we need to use the special :none keyword to zero out the list before telling it the one field we are looking to capture (“referer”). To end this section on using the black box method of programming, we’re going to look at one of the black box analysis modules that can help make the writing of log analysis modules considerably easier. Alex White has written a module called Log::Statistics that can perform simple (i.e., count-based) analyses of log files. Let’s take a look at how it works. 414 | Chapter 10: Log Files The first step after the usual loading of the module and creation of a new object is to teach the module how to parse your log file into fields. For this example, we’ll use the stats log file format generated by the PureFtpd server ( It has the following fields: Here are three example lines (with extra separator lines) so you can get a sense of what they look like: 1151826125 44a778cc.1a41 ftp bb.67.1333.static. D 29 0 /home/ftp/net/mirrors/ languages/perl/CPAN/authors/02STAMP 1151826483 44a77a32.1cf4 ftp D 11 0 /home/ftp/net/mirrors/ 1151829011 44a78408.1eca ftp D 1809 0 /home/ftp/net /mirrors/ To parse this sort of line, we tell Log::Statistics to use a custom regular expression that will capture each field: use Log::Statistics; my $ls = Log::Statistics->new(); $ls->add_line_regexp( '^(\d+)\s+(.*)\s+(\w+)\s(.*)\s+(U|D)\s+(\d+)\s+(\d+)\s+(.*)'); At this point, we tell the module which fields it should summarize and at which posi- tions they are found in the regular expression: $ls->add_field( 3, 'ip' ); $ls->add_field( 4, 'direction' ); All that remains is the actual reading of the log file and its parsing: open my $LOG, '<', 'pureftpd.log'; my $line = ''; while ( defined ($line = <$LOG>) ) { $ls->parse_line($line); } close($LOG); print $ls->get_xml(); The end result is an XML-based report that looks something like this: Dealing with Log File Information | 415 ... In it, we can see that 4674 downloads have been recorded; we also get a list of the IP addresses or hostnames that did the downloading and how many downloads each per- formed. If we wanted to get fancier and show the files each host downloaded, we could change the add_field() section to: $ls->register_field( 'ip', 3 ); $ls->register_field( 'file', 7 ); $ls->add_group(['ip','file']); The first two lines associate names to those positions in the regexp (without generating statistics for them, like add_field() does); the last line specifies the two fields to group on when calculating the statistics. Now the XML output looks like this: ... Prefer to see who downloaded which files? Simply reverse the last line of the preceding code so it reads: $ls->add_group(['file','ip']); The output will now have a section like this: ... 416 | Chapter 10: Log Files ... This XML output can be transformed for display in a pretty table (perhaps using an XSLT stylesheet) or parsed and graphed to make pretty pictures. If you like modules that do this sort of statistical work for you, there are a few worth looking at (including Algorithm::Accounting and Logfile). Be sure to check them all out before embarking on your next project in this vein. As a way of ending this section, let me remind you that the black box approach should be used carefully. The plus side of this approach is that you can often get a great deal done with very little code of your own, thanks to the hard work of the module or script author. The minus side to using the black box approach is that you have to place your trust in another author’s code. It may have subtle bugs or use an approach that does not scale for your needs. It is best to look over the code carefully before you drop it into production in your site. Using databases The last approach we’ll discuss requires the most knowledge outside of the Perl domain to implement. As a result, we’ll only take a very simple look at a technique that over time will probably become more prevalent. The previous examples we’ve seen work fine on reasonably sized data sets when run on machines with a reasonable amount of memory, but they don’t scale. For situations where you have lots of data, especially if the data comes from different sources, data- bases are the natural tool. There are at least two ways to make use of databases from Perl. The first is one I’ll call a “Perl-only” method. With this method, all of the database activity takes place in Perl, or libraries tightly coupled to Perl. The second way uses Perl modules like the DBI family to make Perl a client of another database, such as MySQL, Oracle, or Microsoft SQL Server. Let’s look at an example of using both of these approaches for log pro- cessing and analysis. As long as the data set is not too large, we can probably stick to a Perl-only solution. We’ll extend our ubiquitous breach-finder program for an exam- ple. So far our code just dealt with connections on a single machine. If we wanted to find out about logins from intruders on any of our machines, how would we do it? Our first step is to drop all of the wtmpx data for our machines into a database of some sort. For the purpose of this example, assume that all the machines in question have direct access to some shared directory via some network filesystem, like NFS. Before we proceed, we need to choose a database format. Using Perl-only databases. Dealing with Log File Information | 417 My “Perl database format” of choice is the Berkeley DB format. I use quotes around “Perl database format” because, while the support for DB is shipped with the Perl sources, the actually DB libraries must be procured from another source (http://www and installed before the Perl support can be built. Table 10-5 provides a comparison between the different supported database formats. Table 10-5. Comparison of the supported Perl database formats Name Unix support Windows support Mac OS X support Key or value size limits Byte-order independent “old” dbm Yes No No 1K No “new” dbm Yes No Yes 4K No Sdbm Yes Yes Yes 1K (default) No Gdbm Yesa Yesb Yesa None No DB Yesa Yesb Yesa None Yes a Actual database libraries may have to be downloaded separately. b Database library and Perl module must be downloaded from the Web ( has an old version, or you’ll need to use the Cygwin distribution of Perl). At some point, you may be able to use Strawberry Perl as well (see Chapter 1). I like the Berkeley DB format because it can handle larger data sets and is byte-order- independent. Byte-order independence is particularly important for the Perl code we’re about to see, since we’ll want to read and write to the same file from different machines, which may have different architectures. If byte-order independence is important to you but you don’t want to build and link in external libraries, the module DBM::Deep is another good option. We’ll start by populating the database. For the sake of simplicity and portability, we’re calling the last program to avoid having to unpack() several different wtmpx files our- selves. Here’s the code, with an explanation to follow: use DB_File; use FreezeThaw qw(freeze thaw); use Sys::Hostname; use Fcntl; use strict; # note for Solaris, if you don't want the hostnames truncated you can use # last -a, but that requires a change to the field parsing code below my $lastex = '/bin/last' if ( -x '/bin/last' ); $lastex = '/usr/ucb/last' if ( -x '/usr/ucb/last' ); my $userdb = 'userdata'; my $connectdb = 'connectdata'; my $thishost = &hostname; open my $LAST, '-|', "$lastex" or die "Can't run the program $lastex:$!\n"; 418 | Chapter 10: Log Files my ( $user, $tty, $host, $day, $mon, $date, $time, $when ); my ( %users, %connects ); while ( defined( $_ = <$LAST> ) ) { next if /^reboot/ or /^shutdown/ or /^ftp/ or /^account/ or /^wtmp/; ( $user, $tty, $host, $day, $mon, $date, $time ) = split; next if $tty =~ /^:0/ or $tty =~ /^console$/; next if ( length($host) < 4 ); $when = $mon . ' ' . $date . ' ' . $time; push( @{ $users{$user} }, [ $thishost, $host, $when ] ); push( @{ $connects{$host} }, [ $thishost, $user, $when ] ); } close $LAST; tie my %userdb, 'DB_File', $userdb, O_CREAT | O_RDWR, 0600, $DB_BTREE or die "Unable to open $userdb database for r/w:$!\n"; my $userinfo; for my $user ( keys %users ) { if ( exists $userdb{$user} ) { ($userinfo) = thaw( $userdb{$user} ); push( @{$userinfo}, @{ $users{$user} } ); $userdb{$user} = freeze $userinfo; } else { $userdb{$user} = freeze $users{$user}; } } untie %userdb; tie my %connectdb, 'DB_File', $connectdb, O_CREAT | O_RDWR, 0600, $DB_BTREE or die "Unable to open $connectdb database for r/w:$!\n"; my $connectinfo; for my $connect ( keys %connects ) { if ( exists $connectdb{$connect} ) { ($connectinfo) = thaw( $connectdb{$connect} ); push( @{$connectinfo}, @{ $connects{$connect} } ); $connectdb{$connect} = freeze($connectinfo); } else { $connectdb{$connect} = freeze( $connects{$connect} ); } } untie %connectdb; Our code takes the output from the last program and does the following: 1. Filters out the lines that are not useful. 2. Squirrels away the output in two hashes of lists of lists data structures that look like this: Dealing with Log File Information | 419 $users{username} = [[current host, connecting host, connect time], [current host, connecting host, connect time] ... ]; $connects{host} = [[current host, username1, connect time], [current host, username2, connect time], ... ]; 3. Takes this data structure in memory and attempts to merge it into a database. This last step is the most interesting, so let’s explore it more carefully. We tie the hashes %userdb and %connectdb to database files.† This magic allows us to access those hashes transparently, while Perl handles storing data in and retrieving it from the database files behind the scenes. But hashes only store simple strings, so how do we get our “hashes of list of lists” into a single hash value for storage? Ilya Zakharevich’s FreezeThaw module is used to store our complex data structure in a single scalar that can be used as a hash value. FreezeThaw can take an arbitrary Perl data structure and encode it as a string. There are other modules like this, including Data::Dumper by Gurusamy Sarathy (shipped with Perl) and Storable by Raphael Man- fredi, but FreezeThaw offers the most compact representation of a complex data struc- ture (hence its use here). Each of these modules has its strong points, so be sure to investigate all three if you have a task like this one to perform. In our program, we check whether an entry for this user or host exists. If it doesn’t, we simply “freeze” the data structure into a string and store that string in the database using our tied hash. If it does exist, we “thaw” the existing data structure found in the database back into memory, add our data, then re-freeze and re-store it. If we run this code on several machines, we’ll have a database with some potentially useful information to feed to the next version of our breach-finder program. An excellent time to populate a database like this is just after a log ro- tation of a wtmp file has taken place. The database population code presented here is too bare-bones for pro- duction use. One glaring deficiency is the lack of a mechanism to prevent multiple instances of the program from updating the database at the same time. Given that file locking over NFS is known to be dicey at best, it might be easier to call code like this from a larger program that seri- alizes the process of collecting information from each machine in turn. † You don’t usually have to use the BTree form of storage when using DB_File, but this program can store some very long values. Those values caused the version 1.85 DB_HASH storage method to croak in testing (causing corrupted data), while the BTree storage method seemed to handle the pounding. Later versions of the DB libraries may not have this bug. 420 | Chapter 10: Log Files Now that we have a database full of data, let’s walk through our new improved breach- finder program that uses this information: use DB_File; use FreezeThaw qw(freeze thaw); use Perl6::Form; use Fcntl; my ( $user, $ignore ) = @ARGV; my $userdb = 'userdata'; my $connectdb = 'connectdata'; my $hostformat = '{<<<<<<<<<<<<<<<} -> {<<<<<<<<<<<<<<<} on {<<<<<<<<<<<}'; my $userformat = '{<<<<<<<<}: {<<<<<<<<<<<<<<<} -> {<<<<<<<<<<<<<<<} on {<<<<<<<<<<<}'; tie my %userdb, 'DB_File', $userdb, O_RDONLY, 666, $DB_BTREE or die "Unable to open $userdb database for reading:$!\n"; tie my %connectdb, 'DB_File', $connectdb, O_RDONLY, 666, $DB_BTREE or die "Unable to open $connectdb database for reading:$!\n"; We’ve loaded the modules we need, taken our input, set a few variables, and tied them to our database files. Now it’s time to do some work: # we can exit if we've never seen a connect from this user if ( !exists $userdb{$user} ) { print "No logins from that user\n"; untie %userdb; untie %connectdb; exit; } my ($userinfo) = thaw( $userdb{$user} ); print "-- first host contacts from $user --\n"; my %otherhosts; foreach my $contact ( @{$userinfo} ) { next if ( $ignore and $contact->[1] =~ /$ignore/o ); print form $hostformat, $contact->[1], $contact->[0], $contact->[2]; $otherhosts{ $contact->[1] } = 1; } This code says: if we’ve seen this user at all, we reconstitute the user’s contact records in memory using thaw(). For each contact, we test to see if we’ve been asked to ignore the host from which it came. If not, we print a line for that contact and record the originating host in the %otherhosts hash. We use a hash here as a simple way of collecting the unique list of hosts from all of the contact records. Now that we have the list of hosts from which the intruder may have connected, we need to identify all the other users who have connected from these potentially compromising hosts. Finding this information will be easy, because when we recorded which users logged into which machines, we also recorded the inverse (i.e., which machines were logged Dealing with Log File Information | 421 into by which users) in another database file. We can now look at all of the records from the hosts we identified in the previous step. If we are not told to ignore a host, and we have connection records for it, we capture a unique list of users who have logged into that host using the %userseen hash: print "-- other connects from suspect machines --\n"; my %userseen; foreach my $host ( keys %otherhosts ) { next if ( $ignore and $host =~ /$ignore/o ); next if ( !exists $connectdb{$host} ); my ($connectinfo) = thaw( $connectdb{$host} ); foreach my $connect ( @{$connectinfo} ) { next if ( $ignore and $connect->[0] =~ /$ignore/o ); $userseen{ $connect->[1] } = 1; } } The final act of this three-step drama has a nice circular flair. We return to our original user database to find all of the connections made by suspect users from suspect machines: foreach my $user ( sort keys %userseen ) { next if ( !exists $userdb{$user} ); ($userinfo) = thaw( $userdb{$user} ); foreach my $contact ( @{$userinfo} ) { next if ( $ignore and $contact->[1] =~ /$ignore/o ); print form $userformat, $user, $contact->[1], $contact->[0], $contact->[2] if ( exists $otherhosts{ $contact->[1] } ); } } All that’s left to do then is sweep up the theater and go home: untie %userdb; untie %connectdb; Here’s some example output from the program (again, with the user- and hostnames changed to protect the innocent): -- first host contacts from baduser -- badhost1.example -> machine1.hogwarts.ed on Jan 18 09:55 badhost2.example -> machine2.hogwarts.ed on Jan 19 11:53 -- other connects from suspect machines -- baduser2: badhost1.example -> machine2.hogwarts.e on Dec 15 13:26 baduser2: badhost2.example -> machine2.hogwarts.e on Dec 11 12:45 baduser3: badhost1.example -> machine1.hogwarts.e on Jul 13 16:20 baduser4: badhost1.example -> machine1.hogwarts.e on Jun 9 11:53 baduser: badhost1.example -> machine1.hogwarts.e on Jan 18 09:55 baduser: badhost2.example -> machine2.hogwarts.e on Jan 19 11:53 422 | Chapter 10: Log Files This is a lovely example program, but it doesn’t really scale past a small cluster of machines. For every subsequent run of the program, it may have to read a record from the database, thaw() it back into memory, add some new data to the record, freeze() it again, and store it back in the database. This can be CPU time- and memory-intensive. The whole process potentially happens once per user and machine connection, so things slow down very quickly. If you have a very large data set, you may need to load your data into a more sophisticated SQL database (commercial or otherwise) and query the information you need from it using SQL. If you’re not familiar with SQL, I recom- mend you take a quick peek at Appendix D before looking at this section. Populating the database could be done with code that looks like the following. This example uses SQLite as the backend, but swapping in most other database backends (e.g., MySQL, Microsoft SQL Server, Oracle, DB2, etc.) would be easy; the only things you’d need to change are the DBI connect string and the code for making sure a table with that name exists/is created. That said, let’s dive in: use DBI; use Sys::Hostname; use strict; my $db = 'lastdata'; my $table = 'lastinfo'; # field names we'll use in that table my @fields = qw( username localhost otherhost whenl ); my $lastex = '/bin/last' if ( -x '/bin/last' ); $lastex = '/usr/ucb/last' if ( -x '/usr/ucb/last' ); # database-specific code (note: no username/pwd used, unusual) # RaiseError is used so we don't have to check that each operation succeeds my $dbh = DBI->connect( 'dbi:SQLite:dbname=$db.sql3', '', '', { PrintError => 0, RaiseError => 1, ShowErrorStatement => 1, } ); # Determine the names of the tables currently in the database. # This code is mildly database engine-specific because of the # need to map() to strip off the quotes DBD::SQLite returns around # table names. Most database engines don't require that handholding, # so $dbh->tables()'s results can be used directly. my %dbtables; @dbtables{ map { /\"(.*)\"/, $1 } $dbh->tables() } = (); if ( !exists $dbtables{$table} ) { # More database engine-specific code. Using Perl-cliented SQL databases. Dealing with Log File Information | 423 # This creates the table with all fields of type text. With other database # engines, you might want to use char and varchar as appropriate. $dbh->do( "CREATE TABLE $table (" . join( ' text, ', @fields ) . ' text)' ); } my $thishost = &hostname; # this constructs and prepares a SQL statement with placeholders, as in: # "INSERT INTO lastinfo(username,localhost,otherhost,whenl) # VALUES (?, ?, ?, ?)" my $sth = $dbh->prepare( "INSERT INTO $table (" . join( ', ', @fields ) . ') VALUES (' . join( ', ', ('?') x @fields ) . ')' ); open my $LAST, '-|', "$lastex" or die "Can't run the program $lastex:$!\n"; my ( $user, $tty, $host, $day, $mon, $date, $time, $whenl ); my ( %users, %connects ); while ( defined( $_ = <$LAST> ) ) { next if /^reboot/ or /^shutdown/ or /^ftp/ or /^account/ or /^wtmp/; ( $user, $tty, $host, $day, $mon, $date, $time ) = split; next if $tty =~ /^:0/ or $tty =~ /^console$/; next if ( length($host) < 4 ); $whenl = $mon . ' ' . $date . ' ' . $time; # actually insert the data into the database $sth->execute( $user, $thishost, $host, $whenl ); } close $LAST; $dbh->disconnect; This code creates a table called lastinfo with username, localhost, otherhost, and whenl columns. We iterate over the output of last, inserting non-bogus entries into this table. Now we can use our databases to do what they do so well. Here is a set of sample SQL queries that could easily be wrapped in Perl using the DBI or ODBC interfaces we explored in Chapter 7: -- how many entries in the database? select count (*) from lastinfo; ----------- 10068 -- how many users have logged in? select count (distinct username) from lastinfo; ----------- 237 424 | Chapter 10: Log Files -- how many separate hosts have connected to our machines? select count (distinct otherhost) from lastinfo; ----------- 1000 -- which local hosts has the user "dnb" logged into? select distinct localhost from lastinfo where username = "dnb"; localhost ---------------------------------------- host1 host2 These examples should give you a taste of the sort of “data mining” you can do once all of the data is in a real database. Each of those queries took only a second or so to run. Databases can be fast, powerful tools for system administration. Writing Your Own Log Files I’ve intentionally held back any discussion of how to create your own log files until the very end of this chapter for one simple reason: if you have a good understanding of how to read, parse, and analyze random log files, you are much more likely to write code that will produce log files that are easy to read, parse, and analyze. The actual mechanics of writing log files is pretty easy, as you’ll see in a moment, but knowing how to write good/useful log files is a learned art. There are a relatively large number of Perl modules available to help you with log file production. In the interest of saving space, we’ll look at three options that do a good job of representing the varying levels of functionality and complexity the cornucopia of modules has to offer. One simple admission before we look at some modules: you don’t ac- tually need any modules at all to write to a log file. This process can be as simple as: open my $LOGFILE, '>>', 'logfile' or die "can't open logfile for append: $!\n"; print $LOGFILE 'began logfile example: ' . scalar localtime . "\n"; close $LOGFILE; But as you’ve probably guessed, we’re going to get much spiffier than that.... Logging Shortcuts and Formatting Help The first option is to use log modules that try to make writing the actual lines of a log easier, more structured, or both. For example, Tie::LogFile by Chris Reinhardt makes it easy to write lines to a log file with a preset format. Here’s a piece of sample code: Writing Your Own Log Files | 425 use Tie::LogFile; tie( *LOG, 'Tie::LogFile', 'filename', format => '(%p) [%d] %m' ); print LOG 'message'; # (pid) [dt] message close(LOG); The tie() line creates a tie()d filehandle with special properties. Each time we print to that filehandle, it will format the output using the format string specified and then add it to the file. In this case, we’re specifying that each line contain: (%p) The PID of the running process [%d] A date/time stamp %m The actual message If we run the preceding program three times, we get a file that looks like this: (19064) [Wed Jun 21 12:01:46 2008] message (19719) [Wed Jun 21 12:09:02 2008] message (19725) [Wed Jun 21 12:10:12 2008] message Basic/Intermediate Logging Frameworks Eventually, your desires for more sophisticated logging functionality may outgrow the types of modules we’ve seen so far. At that point you’ll find yourself looking for a module with at least a basic framework for handling logging tasks. Two such frame- works that have found favor in the Perl community are Log::Dispatch, by Dave Rolsky, and Log::Agent, originally by Raphael Manfredi and now maintained by Mark Rogaski. We’ll take a look at the first one, but you should feel free to compare the two and see which one appeals to you. Here’s how Log::Dispatch works. First, you create a log dispatch object through which all logging is done: use Log::Dispatch; my $ld = Log::Dispatch->new; That object isn’t particularly useful to start, but (and here comes the fun part) it acts as the hub for a set of modules that handle the disposition of every log message. For example, if you wanted log messages to go to a file, you would use a line like this to add it to the dispatch object: $ld->add( Log::Dispatch::File->new( name => 'to_file', filename => 'filename', min_level => 'info', 426 | Chapter 10: Log Files max_level => 'alert', mode => 'append' ) ); This line says that output should go to an object called to_file whose job it will be to log data to the file filename. That object will log any message it receives that has a log level anywhere from info to alert. But why stop with logging to a file? How about configuring it to send out messages via an email message? You can do that as follows: $ld->add( Log::Dispatch::Email::MailSend->new( name => 'to_email', min_level => 'alert', to => [qw ( )], subject => 'log alert' ) ); Similarly, you might want to send messages to a syslog server for further aggregation and processing: $ld->add( Log::Dispatch::Syslog->new( name => 'to_syslog', min_level => 'warning', facility => 'local2' ) ); One of the great things about Log::Dispatch is that it has so many of these dispatch objects available. The module ships with other modules that can write to a file, send email, or log to the screen. Others have created modules for logging to a database via DBI, writing to files that are automatically rotated for you, sending messages via a Jabber IM server, and so on. Observant readers are probably getting a bit impatient at this point, because they’ve noticed we haven’t actually logged anything yet. No problem, that’s easy: $ld->log( level => 'notice', message => 'here is a log message' ); Or, we could use a shortcut to send a message at the notice level: $ld->notice( 'here is a log message' ); This code will send that message to each of the dispatch objects we’ve add()ed that are set to listen to messages at this log level. If at this point we did the equivalent of screaming bloody murder: $ld->emergency( 'printer on fire!' ); Writing Your Own Log Files | 427 a message would get recorded to the file and sent to syslog, and an email message would be dispatched. To send a message to a specific dispatch object, the log_to method is used: $ld->log_to( name => 'to_syslog', level => 'debug', message => 'sneeble component is failing' ); A basic/intermediate logging framework like this gives us significantly more control over when messages are logged, and where. This is often all of the flexibility and control one needs for a project. But there are cases where a really large project demands even more control. For those cases, there is an advanced logging framework available. Advanced Logging Framework The next step up in complexity and power is the log4perl logging framework, by Mike Schilli and Kevin Goess. This is a direct port of the log4j framework that is so popular in the Java community. The log4perl package is so compatible with its progenitor that it will even parse and use many log4j configuration files without modification. Code examples and a deeper exploration of log4perl’s functionality would take con- siderably more space in this chapter than makes sense, especially considering that one of its authors has created an excellent tutorial (see the references section at the end of this chapter for a pointer). Instead, let me give you a quick rundown of the features of the framework and how they can benefit you. Let’s start with the features we’ve already seen: • log4perl offers the same ability to multiplex logging messages out to different output destinations as Log::Dispatch. In fact, it actually uses the same Log::Dispatch::* output modules as Log::Dispatch, so all of that flexibility comes along for the ride. • log4perl supports logging levels (and has the ability to tell parts of the framework to pay attention to only messages of a certain level). • log4perl has similar convenience methods ($object->error(), $object->warn(), $object->debug(), etc.) for logging at all of the standard levels. Now let’s add the exciting parts: • log4perl has something called “categories” that let you name a particular section of your code for logging purposes. For example, for an online banking application, you might have GetBalance, MakeWithdrawal, and MakeDeposit categories for each section of your code. The logging for each of these categories can be turned off/on and have its level set independently. Only want logging information about with- drawals at debug level? No problem with log4perl. • If you are building big systems with lots of code, chances are they are written in an OOP-ish style with classes and subclasses, objects and sub-objects, and all that 428 | Chapter 10: Log Files other good stuff. log4perl handles all of this complexity, because its categories are actually hierarchical in nature. Each category can correspond to a class in your system; you can have a Withdrawal, Withdrawal::CheckBalance, Withdrawal::Check Balance::Overdraft, and so on. Log levels can be set at a place high in the tree of categories, and all subcategories under that level will inherit the setting. Want logging enabled for only a piece of your complex code hierarchy? Easily done. • As I alluded to earlier, log4perl can read configuration files that describe precisely how logging should be enabled for your complex code jungle. As an added bonus, log4perl can be set to periodically check this file for modifications and load a new configuration if it changes. This means you can change the kind of logging your massive system is doing while it’s running. Pretty slick. If this description has piqued your interest, visit for more details. The subject of log creation, manipulation, and analysis is a vast one. Hopefully this chapter has given you a grasp of a few tools and a little inspiration. Module Information for This Chapter Modules CPAN ID Version Win32::EventLog (ships with ActivePerl) 0.074 File::Copy (ships with ActivePerl) 2.09 Logfile::Rotate PAULG 1.04 File::Temp (ships with Perl) 0.17 Getopt::Long (ships with Perl) 2.35 Time::Local (ships with Perl) 1.13 Perl6::Form DCONWAY 0.04 User::Utmp MPIOTR 1.8 Readonly ROODE 1.03 Log::Procmail BOOK 0.11 SyslogScan RHNELSON 0.32 File::Tail MGRABNAR 0.99.3 Parse::Syslog DSCHWEI 1.09 Regexp::Log::DateRange KARASIK 0.01 Regexp::Log BOOK 0.04 Regexp::Log::Common BARBIE 0.04 Log::Statistics VVU 0.047 DB_File (ships with Perl) PMQS 1.72 Module Information for This Chapter | 429 Modules CPAN ID Version DBM::Deep RKINYON 0.983 FreezeThaw ILYAZ 0.3 Sys::Hostname (ships with Perl) 1.11 Fcntl (ships with Perl) 1.05 DBI TIMB 1.52 DBD::Sqlite MSERGEANT 1.13 Tie::LogFile CREIN 0.1 Log::Dispatch DROLSKY 2.13 Log4perl MSCHILLI 1.07 References for More Information Essential System Administration, Third Edition ( 9780596003432/), by Æleen Frisch (O’Reilly) has a good, short intro to syslog. is the home of Frank Heyne software, a provider of Win32 Event Log-parsing software. It also has a good Event Log FAQ list. is Philippe Le Berre’s home page; it contains an excellent write- up on the use of Win32::EventLog and other Windows packages. Practical Unix & Internet Security, Third Edition ( 9780596003234/), by Simson Garfinkel, Gene Spafford, and Alan Schwartz (O’Reilly), is another good (and slightly more detailed) intro to syslog; also includes tcpwrappers information. is the home of the Log::Statistics package and contains some good documentation on the project. is the home of the log4perl project. Be sure to see the tutorial linked off that site at is the site set up by Tina Bird and Marcus Ranum, two security researchers who are working to bring more attention to log analysis issues as they relate to security. They also host a mailing list on the subject at http://www.loga USENIX held a workshop on analysis of system logs in 2008 (WASL ’08). More infor- mation can be found at For interactive log analysis, the products provided by Splunk (http://www.splunk .com) are pretty phenomenal. They also allow for free usage when analyzing data under a certain size. 430 | Chapter 10: Log Files Microsoft makes a (poorly publicized but very cool) package called Log Parser. I last found it on the download site at ilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07, but given how often Microsoft shuffles URLs, you may have to search for it at Microsoft describes it like this: Log parser is a powerful, versatile tool that provides universal query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Win- dows® operating system such as the Event Log, the Registry, the file system, and Active Directory®. You tell Log Parser what information you need and how you want it pro- cessed. The results of your query can be custom-formatted in text based output, or they can be persisted to more specialty targets like SQL, SYSLOG, or a chart. References for More Information | 431 CHAPTER 11 Security Any discussion of security is fraught with peril, for at least three reasons: • Security means different things to different people. If you walked into a conference of Greco-Roman scholars and asked about Rome, the first scholar might rise dramatically to her feet and begin to lecture about aqueducts (infrastructure and delivery), while the second focused on Pax Romana (ideology and policies), a third expounded on the Roman legions (enforcement), a fourth on the Roman Senate (administration), and so on. The need to deal with every facet of security at once is security’s first trap. • Security is a continuum, not a binary. People often mistakenly think that a program, a computer, a network, etc. can be “secure.” This chapter will never claim to show you how to make anything secure, though it will try to help you to make something more secure, or at least to recognize when something is less secure. • Finally, one of the most deadly traps in this business is specificity. It is true that you can often address security issues by paying attention to the details, but the set of details is ever-shifting. Patching security holes A, B, and C only ensures that those particular holes will not be a problem—if the patches work as promised. It does nothing to help when hole D is found. That’s why this chapter will focus on general principles and tools for improving security, rather than telling you how to fix any particular buffer overflow, vulnerable registry key, or world-writable system file. One good way to lead into a discussion of these principles is to examine how security manifests itself in the physical world. In both the real and virtual worlds, it all comes down to fear. Will something I care about be damaged, lost, or revealed? Is there some- thing I can do to prevent this from happening? What is the likelihood of something happening, and what are the consequences if it does? Is it happening right now? If we look at how we face fear in the physical world, we can learn ways to deal with it in the system administration domain as well. When we want to protect real-world objects, we invent stronger ways of partitioning physical space (e.g., bank vaults) so that only certain people can get to their contents. When we want to protect real-world 433 intellectual property and secrets, we create methods of restricting access, like top-secret clearance policies or, in spy-vs.-spy situations, data encryption. The computer equiv- alents of these things are remarkably similar; they too include permission systems, access lists, encryption, etc. But both on and off the computer, security is a never-ending pursuit. For every hour spent designing a security system, there is at least an hour spent looking for a way to evade it. In our case, threats may come from hordes of bored teenagers with computers looking for something to do with their excess energy, or disgruntled former employees with vengeance on their minds. One approach to improving security that has persisted over the ages is appointing a designated person to allay the public’s fears. Once upon a time, there was nothing so comforting as the sound of the night watchman’s footsteps as he walked through the town, jiggling door handles. We’ll use this quaint image as the jumping-off point for our exploration of security and network monitoring with Perl. Noticing Unexpected or Unauthorized Changes A good watchman notices change. She knows when things are in the wrong place or go missing. If your precious Maltese Falcon gets replaced with a forgery, the watchman is the first person who should notice. Similarly, if someone modifies or replaces key files on your system, you want sirens to blare and klaxons to sound. More often than not, the change will be harmless. But the first time someone breaches your security and mucks with /bin/login, system32/*.dll, or Finder, you’ll be so glad you noticed that you will excuse any prior false alarms.* Local Filesystem Changes Filesystems are an excellent place to begin our exploration of change-checking pro- grams. We’re going to investigate ways to check whether important files, like operating system binaries and security-related files (e.g., /etc/passwd or system32/*.dll), have changed. Changes to these files made without the administrator’s knowledge are often signs of an intruder. Some relatively sophisticated cracker toolkits available on the Web install Trojan versions of important files, then cover their tracks. That’s a malevolent kind of change that we have the ability to detect.† On the other end of the spectrum, sometimes it is just nice to know when important files have been changed (especially in environments where multiple people administer the same systems). The techniques we’re about to explore will work equally well in both cases. * This is not to say that you shouldn’t work hard to reduce false positives. If you get too many alerts, you’ll start ignoring them or (worse) automatically sending them to the bitbucket. † Though if you are dealing with a particularly nasty rootkit that changes the OS-level functions that Perl calls, all bets are off. Sorry. 434 | Chapter 11: Security The easiest way to tell if a file has changed is to use the Perl functions stat() and lstat(). These functions take a filename or a filehandle and return an array containing information about that file. The only difference between the two functions manifests itself on operating systems such as Unix that support symbolic links. In these cases, lstat() returns information about the symbolic link itself, while stat() returns info about the target of the link. On all other operating systems, the information lstat() returns should be the same as that returned by stat(). Using stat() or lstat() is easy: my @information = stat('filename'); As demonstrated in Chapter 2, we can also use Tom Christiansen’s File::Stat module to provide this information using an object-oriented syntax. The information stat() or lstat() returns is operating system-dependent. stat() and lstat() began as Unix system calls, so the Perl documentation for these calls is skewed toward the return values for Unix systems. Table 11-1 shows how these values compare to those returned by stat() on Windows-based operating systems. The first two columns show the Unix field number and description. Table 11-1. stat() return value comparisona Field # Unix field description Valid for Windows-based operating systems? 0 Device number of filesystem Yes (drive #) 1 Inode number No (always 0) 2 File mode (type and permissions) Yes 3 Number of (hard) links to the file Yes (for NTFS) 4 Numeric user ID of file’s owner No (always 0) 5 Numeric group ID of file’s owner No (always 0) 6 Device identifier (special files only) Yes (drive #) 7 Total size of file, in bytes Yes (but does not include the size of any alternate data streams) 8 Last access time since the epoch Yes 9 Last modify time since the epoch Yes 10 Inode change time since the epoch Yes (but is file creation time) 11 Preferred block size for filesystem I/O No (always null) 12 Actual number of blocks allocated No (always null) a Fans of the first edition of this book might notice that this chart has lost a column. The transition from Mac OS to Mac OS X brought along a massive amount of compatibility changes (unusually, making it more compatible), rendering the Mac OS column unnecessary. Noticing Unexpected or Unauthorized Changes | 435 If dealing with time values on Windows systems in a way that is con- sistent with Unix systems is important to you, you will want to install the module Win32::UTCFileTime, by Steve Hay, and read its documen- tation carefully. Windows systems have some issues reporting file times as they relate to daylight savings time. This module can override the standard Perl stat() and other calls to fix the problems. In addition to stat() and lstat(), some versions of Perl have special mechanisms for returning attributes of a file that are particular to a specific OS. See Chapter 2 for dis- cussions of functions like Win32::FileSecurity::Get(). Once you have queried the stat() values for a file, the next step is to compare the “interesting” values against a known set of values for that file that you’ve pre-generated and kept secure. If the values have changed, something about the file must have changed. Here’s a program that both generates a string of lstat() values and checks files against a known set of those values. We intentionally exclude field #8 from Ta- ble 11-1 (last access time) because it changes every time a file is read. This program takes either a -p filename argument to print lstat() values for a given file or a -c filename argument to check the lstat() values for all of the files recorded in filename: use Getopt::Std; # we use this for prettier output later in PrintChanged() my @statnames = qw(dev ino mode nlink uid gid rdev size mtime ctime blksize blocks); getopt( 'p:c:', \my %opt ); die "Usage: $0 [-p |-c ]\n" unless ( $opt{p} or $opt{c} ); if ( $opt{p} ) { die "Unable to stat file $opt{p}:$!\n" unless ( -e $opt{p} );