[使用Python进行自然语言处理].(Natural.Language.Processing.with.Python)

Today I was chatting with three of our visiting graduate students f' >>> nltk.word_tokenize(nltk.html_clean(content)) >>> nltk.word_tokenize(nltk.clean_html(llog.entries[2].content[0].value)) [u'Today', u'I', u'was', u'chatting', u'with', u'three', u'of', u'our', u'visiting', u'graduate', u'students', u'from', u'the', u'PRC', u'.', u'Thinking', u'that', u'I', 3.1 Accessing Text from the Web and from Disk | 83 u'was', u'being', u'au', u'courant', u',', u'I', u'mentioned', u'the', u'expression', u'DUI4XIANG4', u'\u5c0d\u8c61', u'("', u'boy', u'/', u'girl', u'friend', u'"', ...] Note that the resulting strings have a u prefix to indicate that they are Unicode strings (see Section 3.3). With some further work, we can write programs to create a small corpus of blog posts, and use this as the basis for our NLP work. Reading Local Files In order to read a local file, we need to use Python’s built-in open() function, followed by the read() method. Supposing you have a file document.txt, you can load its contents like this: >>> f = open('document.txt') >>> raw = f.read() Your Turn: Create a file called document.txt using a text editor, and type in a few lines of text, and save it as plain text. If you are using IDLE, select the New Window command in the File menu, typing the required text into this window, and then saving the file as document.txt inside the directory that IDLE offers in the pop-up dialogue box. Next, in the Python interpreter, open the file using f = open('document.txt'), then inspect its contents using print f.read(). Various things might have gone wrong when you tried this. If the interpreter couldn’t find your file, you would have seen an error like this: >>> f = open('document.txt') Traceback (most recent call last): File "", line 1, in -toplevel- f = open('document.txt') IOError: [Errno 2] No such file or directory: 'document.txt' To check that the file that you are trying to open is really in the right directory, use IDLE’s Open command in the File menu; this will display a list of all the files in the directory where IDLE is running. An alternative is to examine the current directory from within Python: >>> import os >>> os.listdir('.') Another possible problem you might have encountered when accessing a text file is the newline conventions, which are different for different operating systems. The built-in open() function has a second parameter for controlling how the file is opened: open('do cument.txt', 'rU'). 'r' means to open the file for reading (the default), and 'U' stands for “Universal”, which lets us ignore the different conventions used for marking new- lines. Assuming that you can open the file, there are several methods for reading it. The read() method creates a string with the contents of the entire file: 84 | Chapter 3: Processing Raw Text >>> f.read() 'Time flies like an arrow.\nFruit flies like a banana.\n' Recall that the '\n' characters are newlines; this is equivalent to pressing Enter on a keyboard and starting a new line. We can also read a file one line at a time using a for loop: >>> f = open('document.txt', 'rU') >>> for line in f: ... print line.strip() Time flies like an arrow. Fruit flies like a banana. Here we use the strip() method to remove the newline character at the end of the input line. NLTK’s corpus files can also be accessed using these methods. We simply have to use nltk.data.find() to get the filename for any corpus item. Then we can open and read it in the way we just demonstrated: >>> path = nltk.data.find('corpora/gutenberg/melville-moby_dick.txt') >>> raw = open(path, 'rU').read() Extracting Text from PDF, MSWord, and Other Binary Formats ASCII text and HTML text are human-readable formats. Text often comes in binary formats—such as PDF and MSWord—that can only be opened using specialized soft- ware. Third-party libraries such as pypdf and pywin32 provide access to these formats. Extracting text from multicolumn documents is particularly challenging. For one-off conversion of a few documents, it is simpler to open the document with a suitable application, then save it as text to your local drive, and access it as described below. If the document is already on the Web, you can enter its URL in Google’s search box. The search result often includes a link to an HTML version of the document, which you can save as text. Capturing User Input Sometimes we want to capture the text that a user inputs when she is interacting with our program. To prompt the user to type a line of input, call the Python function raw_input(). After saving the input to a variable, we can manipulate it just as we have done for other strings. >>> s = raw_input("Enter some text: ") Enter some text: On an exceptionally hot evening early in July >>> print "You typed", len(nltk.word_tokenize(s)), "words." You typed 8 words. 3.1 Accessing Text from the Web and from Disk | 85 The NLP Pipeline Figure 3-1 summarizes what we have covered in this section, including the process of building a vocabulary that we saw in Chapter 1. (One step, normalization, will be discussed in Section 3.6.) Figure 3-1. The processing pipeline: We open a URL and read its HTML content, remove the markup and select a slice of characters; this is then tokenized and optionally converted into an nltk.Text object; we can also lowercase all the words and extract the vocabulary. There’s a lot going on in this pipeline. To understand it properly, it helps to be clear about the type of each variable that it mentions. We find out the type of any Python object x using type(x); e.g., type(1) is since 1 is an integer. When we load the contents of a URL or file, and when we strip out HTML markup, we are dealing with strings, Python’s data type (we will learn more about strings in Section 3.2): >>> raw = open('document.txt').read() >>> type(raw) When we tokenize a string we produce a list (of words), and this is Python’s type. Normalizing and sorting lists produces other lists: >>> tokens = nltk.word_tokenize(raw) >>> type(tokens) >>> words = [w.lower() for w in tokens] >>> type(words) >>> vocab = sorted(set(words)) >>> type(vocab) The type of an object determines what operations you can perform on it. So, for ex- ample, we can append to a list but not to a string: 86 | Chapter 3: Processing Raw Text >>> vocab.append('blog') >>> raw.append('blog') Traceback (most recent call last): File "", line 1, in AttributeError: 'str' object has no attribute 'append' Similarly, we can concatenate strings with strings, and lists with lists, but we cannot concatenate strings with lists: >>> query = 'Who knows?' >>> beatles = ['john', 'paul', 'george', 'ringo'] >>> query + beatles Traceback (most recent call last): File "", line 1, in TypeError: cannot concatenate 'str' and 'list' objects In the next section, we examine strings more closely and further explore the relationship between strings and lists. 3.2 Strings: Text Processing at the Lowest Level It’s time to study a fundamental data type that we’ve been studiously avoiding so far. In earlier chapters we focused on a text as a list of words. We didn’t look too closely at words and how they are handled in the programming language. By using NLTK’s corpus interface we were able to ignore the files that these texts had come from. The contents of a word, and of a file, are represented by programming languages as a fun- damental data type known as a string. In this section, we explore strings in detail, and show the connection between strings, words, texts, and files. Basic Operations with Strings Strings are specified using single quotes or double quotes , as shown in the fol- lowing code example. If a string contains a single quote, we must backslash-escape the quote so Python knows a literal quote character is intended, or else put the string in double quotes . Otherwise, the quote inside the string will be interpreted as a close quote, and the Python interpreter will report a syntax error: >>> monty = 'Monty Python' >>> monty 'Monty Python' >>> circus = "Monty Python's Flying Circus" >>> circus "Monty Python's Flying Circus" >>> circus = 'Monty Python\'s Flying Circus' >>> circus "Monty Python's Flying Circus" >>> circus = 'Monty Python's Flying Circus' File "", line 1 circus = 'Monty Python's Flying Circus' ^ SyntaxError: invalid syntax 3.2 Strings: Text Processing at the Lowest Level | 87 Sometimes strings go over several lines. Python provides us with various ways of en- tering them. In the next example, a sequence of two strings is joined into a single string. We need to use backslash or parentheses so that the interpreter knows that the statement is not complete after the first line. >>> couplet = "Shall I compare thee to a Summer's day?"\ ... "Thou are more lovely and more temperate:" >>> print couplet Shall I compare thee to a Summer's day?Thou are more lovely and more temperate: >>> couplet = ("Rough winds do shake the darling buds of May," ... "And Summer's lease hath all too short a date:") >>> print couplet Rough winds do shake the darling buds of May,And Summer's lease hath all too short a date: Unfortunately these methods do not give us a newline between the two lines of the sonnet. Instead, we can use a triple-quoted string as follows: >>> couplet = """Shall I compare thee to a Summer's day? ... Thou are more lovely and more temperate:""" >>> print couplet Shall I compare thee to a Summer's day? Thou are more lovely and more temperate: >>> couplet = '''Rough winds do shake the darling buds of May, ... And Summer's lease hath all too short a date:''' >>> print couplet Rough winds do shake the darling buds of May, And Summer's lease hath all too short a date: Now that we can define strings, we can try some simple operations on them. First let’s look at the + operation, known as concatenation . It produces a new string that is a copy of the two original strings pasted together end-to-end. Notice that concatenation doesn’t do anything clever like insert a space between the words. We can even multiply strings : >>> 'very' + 'very' + 'very' 'veryveryvery' >>> 'very' * 3 'veryveryvery' Your Turn: Try running the following code, then try to use your un- derstanding of the string + and * operations to figure out how it works. Be careful to distinguish between the string ' ', which is a single white- space character, and '', which is the empty string. >>> a = [1, 2, 3, 4, 5, 6, 7, 6, 5, 4, 3, 2, 1] >>> b = [' ' * 2 * (7 - i) + 'very' * i for i in a] >>> for line in b: ... print b We’ve seen that the addition and multiplication operations apply to strings, not just numbers. However, note that we cannot use subtraction or division with strings: 88 | Chapter 3: Processing Raw Text >>> 'very' - 'y' Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for -: 'str' and 'str' >>> 'very' / 2 Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for /: 'str' and 'int' These error messages are another example of Python telling us that we have got our data types in a muddle. In the first case, we are told that the operation of subtraction (i.e., -) cannot apply to objects of type str (strings), while in the second, we are told that division cannot take str and int as its two operands. Printing Strings So far, when we have wanted to look at the contents of a variable or see the result of a calculation, we have just typed the variable name into the interpreter. We can also see the contents of a variable using the print statement: >>> print monty Monty Python Notice that there are no quotation marks this time. When we inspect a variable by typing its name in the interpreter, the interpreter prints the Python representation of its value. Since it’s a string, the result is quoted. However, when we tell the interpreter to print the contents of the variable, we don’t see quotation characters, since there are none inside the string. The print statement allows us to display more than one item on a line in various ways, as shown here: >>> grail = 'Holy Grail' >>> print monty + grail Monty PythonHoly Grail >>> print monty, grail Monty Python Holy Grail >>> print monty, "and the", grail Monty Python and the Holy Grail Accessing Individual Characters As we saw in Section 1.2 for lists, strings are indexed, starting from zero. When we index a string, we get one of its characters (or letters). A single character is nothing special—it’s just a string of length 1. >>> monty[0] 'M' >>> monty[3] 't' >>> monty[5] ' ' 3.2 Strings: Text Processing at the Lowest Level | 89 As with lists, if we try to access an index that is outside of the string, we get an error: >>> monty[20] Traceback (most recent call last): File "", line 1, in ? IndexError: string index out of range Again as with lists, we can use negative indexes for strings, where -1 is the index of the last character . Positive and negative indexes give us two ways to refer to any position in a string. In this case, when the string had a length of 12, indexes 5 and -7 both refer to the same character (a space). (Notice that 5 = len(monty) - 7.) >>> monty[-1] 'n' >>> monty[5] ' ' >>> monty[-7] ' ' We can write for loops to iterate over the characters in strings. This print statement ends with a trailing comma, which is how we tell Python not to print a newline at the end. >>> sent = 'colorless green ideas sleep furiously' >>> for char in sent: ... print char, ... c o l o r l e s s g r e e n i d e a s s l e e p f u r i o u s l y We can count individual characters as well. We should ignore the case distinction by normalizing everything to lowercase, and filter out non-alphabetic characters: >>> from nltk.corpus import gutenberg >>> raw = gutenberg.raw('melville-moby_dick.txt') >>> fdist = nltk.FreqDist(ch.lower() for ch in raw if ch.isalpha()) >>> fdist.keys() ['e', 't', 'a', 'o', 'n', 'i', 's', 'h', 'r', 'l', 'd', 'u', 'm', 'c', 'w', 'f', 'g', 'p', 'b', 'y', 'v', 'k', 'q', 'j', 'x', 'z'] This gives us the letters of the alphabet, with the most frequently occurring letters listed first (this is quite complicated and we’ll explain it more carefully later). You might like to visualize the distribution using fdist.plot(). The relative character frequencies of a text can be used in automatically identifying the language of the text. Accessing Substrings A substring is any continuous section of a string that we want to pull out for further processing. We can easily access substrings using the same slice notation we used for lists (see Figure 3-2). For example, the following code accesses the substring starting at index 6, up to (but not including) index 10: >>> monty[6:10] 'Pyth' 90 | Chapter 3: Processing Raw Text Here we see the characters are 'P', 'y', 't', and 'h', which correspond to monty[6] ... monty[9] but not monty[10]. This is because a slice starts at the first index but finishes one before the end index. We can also slice with negative indexes—the same basic rule of starting from the start index and stopping one before the end index applies; here we stop before the space character. >>> monty[-12:-7] 'Monty' As with list slices, if we omit the first value, the substring begins at the start of the string. If we omit the second value, the substring continues to the end of the string: >>> monty[:5] 'Monty' >>> monty[6:] 'Python' We test if a string contains a particular substring using the in operator, as follows: >>> phrase = 'And now for something completely different' >>> if 'thing' in phrase: ... print 'found "thing"' found "thing" We can also find the position of a substring within a string, using find(): >>> monty.find('Python') 6 Your Turn: Make up a sentence and assign it to a variable, e.g., sent = 'my sentence...'. Now write slice expressions to pull out individual words. (This is obviously not a convenient way to process the words of a text!) Figure 3-2. String slicing: The string Monty Python is shown along with its positive and negative indexes; two substrings are selected using “slice” notation. The slice [m,n] contains the characters from position m through n-1. 3.2 Strings: Text Processing at the Lowest Level | 91 More Operations on Strings Python has comprehensive support for processing strings. A summary, including some operations we haven’t seen yet, is shown in Table 3-2. For more information on strings, type help(str) at the Python prompt. Table 3-2. Useful string methods: Operations on strings in addition to the string tests shown in Table 1-4; all methods produce a new string or list Method Functionality s.find(t) Index of first instance of string t inside s (-1 if not found) s.rfind(t) Index of last instance of string t inside s (-1 if not found) s.index(t) Like s.find(t), except it raises ValueError if not found s.rindex(t) Like s.rfind(t), except it raises ValueError if not found s.join(text) Combine the words of the text into a string using s as the glue s.split(t) Split s into a list wherever a t is found (whitespace by default) s.splitlines() Split s into a list of strings, one per line s.lower() A lowercased version of the string s s.upper() An uppercased version of the string s s.titlecase() A titlecased version of the string s s.strip() A copy of s without leading or trailing whitespace s.replace(t, u) Replace instances of t with u inside s The Difference Between Lists and Strings Strings and lists are both kinds of sequence. We can pull them apart by indexing and slicing them, and we can join them together by concatenating them. However, we can- not join strings and lists: >>> query = 'Who knows?' >>> beatles = ['John', 'Paul', 'George', 'Ringo'] >>> query[2] 'o' >>> beatles[2] 'George' >>> query[:2] 'Wh' >>> beatles[:2] ['John', 'Paul'] >>> query + " I don't" "Who knows? I don't" >>> beatles + 'Brian' Traceback (most recent call last): File "", line 1, in TypeError: can only concatenate list (not "str") to list >>> beatles + ['Brian'] ['John', 'Paul', 'George', 'Ringo', 'Brian'] 92 | Chapter 3: Processing Raw Text When we open a file for reading into a Python program, we get a string corresponding to the contents of the whole file. If we use a for loop to process the elements of this string, all we can pick out are the individual characters—we don’t get to choose the granularity. By contrast, the elements of a list can be as big or small as we like: for example, they could be paragraphs, sentences, phrases, words, characters. So lists have the advantage that we can be flexible about the elements they contain, and corre- spondingly flexible about any downstream processing. Consequently, one of the first things we are likely to do in a piece of NLP code is tokenize a string into a list of strings (Section 3.7). Conversely, when we want to write our results to a file, or to a terminal, we will usually format them as a string (Section 3.9). Lists and strings do not have exactly the same functionality. Lists have the added power that you can change their elements: >>> beatles[0] = "John Lennon" >>> del beatles[-1] >>> beatles ['John Lennon', 'Paul', 'George'] On the other hand, if we try to do that with a string—changing the 0th character in query to 'F'—we get: >>> query[0] = 'F' Traceback (most recent call last): File "", line 1, in ? TypeError: object does not support item assignment This is because strings are immutable: you can’t change a string once you have created it. However, lists are mutable, and their contents can be modified at any time. As a result, lists support operations that modify the original value rather than producing a new value. Your Turn: Consolidate your knowledge of strings by trying some of the exercises on strings at the end of this chapter. 3.3 Text Processing with Unicode Our programs will often need to deal with different languages, and different character sets. The concept of “plain text” is a fiction. If you live in the English-speaking world you probably use ASCII, possibly without realizing it. If you live in Europe you might use one of the extended Latin character sets, containing such characters as “ø” for Danish and Norwegian, “ő” for Hungarian, “ñ” for Spanish and Breton, and “ň” for Czech and Slovak. In this section, we will give an overview of how to use Unicode for processing texts that use non-ASCII character sets. 3.3 Text Processing with Unicode | 93 What Is Unicode? Unicode supports over a million characters. Each character is assigned a number, called a code point. In Python, code points are written in the form \uXXXX, where XXXX is the number in four-digit hexadecimal form. Within a program, we can manipulate Unicode strings just like normal strings. How- ever, when Unicode characters are stored in files or displayed on a terminal, they must be encoded as a stream of bytes. Some encodings (such as ASCII and Latin-2) use a single byte per code point, so they can support only a small subset of Unicode, enough for a single language. Other encodings (such as UTF-8) use multiple bytes and can represent the full range of Unicode characters. Text in files will be in a particular encoding, so we need some mechanism for translating it into Unicode—translation into Unicode is called decoding. Conversely, to write out Unicode to a file or a terminal, we first need to translate it into a suitable encoding— this translation out of Unicode is called encoding, and is illustrated in Figure 3-3. Figure 3-3. Unicode decoding and encoding. From a Unicode perspective, characters are abstract entities that can be realized as one or more glyphs. Only glyphs can appear on a screen or be printed on paper. A font is a mapping from characters to glyphs. Extracting Encoded Text from Files Let’s assume that we have a small text file, and that we know how it is encoded. For example, polish-lat2.txt, as the name suggests, is a snippet of Polish text (from the Polish Wikipedia; see http://pl.wikipedia.org/wiki/Biblioteka_Pruska). This file is encoded as Latin-2, also known as ISO-8859-2. The function nltk.data.find() locates the file for us. 94 | Chapter 3: Processing Raw Text >>> path = nltk.data.find('corpora/unicode_samples/polish-lat2.txt') The Python codecs module provides functions to read encoded data into Unicode strings, and to write out Unicode strings in encoded form. The codecs.open() function takes an encoding parameter to specify the encoding of the file being read or written. So let’s import the codecs module, and call it with the encoding 'latin2' to open our Polish file as Unicode: >>> import codecs >>> f = codecs.open(path, encoding='latin2') For a list of encoding parameters allowed by codecs, see http://docs.python.org/lib/ standard-encodings.html. Note that we can write Unicode-encoded data to a file using f = codecs.open(path, 'w', encoding='utf-8'). Text read from the file object f will be returned in Unicode. As we pointed out earlier, in order to view this text on a terminal, we need to encode it, using a suitable encoding. The Python-specific encoding unicode_escape is a dummy encoding that converts all non-ASCII characters into their \uXXXX representations. Code points above the ASCII 0–127 range but below 256 are represented in the two-digit form \xXX. >>> for line in f: ... line = line.strip() ... print line.encode('unicode_escape') Pruska Biblioteka Pa\u0144stwowa. Jej dawne zbiory znane pod nazw\u0105 "Berlinka" to skarb kultury i sztuki niemieckiej. Przewiezione przez Niemc\xf3w pod koniec II wojny \u015bwiatowej na Dolny \u015al\u0105sk, zosta\u0142y odnalezione po 1945 r. na terytorium Polski. Trafi\u0142y do Biblioteki Jagiello\u0144skiej w Krakowie, obejmuj\u0105 ponad 500 tys. zabytkowych archiwali\xf3w, m.in. manuskrypty Goethego, Mozarta, Beethovena, Bacha. The first line in this output illustrates a Unicode escape string preceded by the \u escape string, namely \u0144. The relevant Unicode character will be displayed on the screen as the glyph ń. In the third line of the preceding example, we see \xf3, which corre- sponds to the glyph ó, and is within the 128–255 range. In Python, a Unicode string literal can be specified by preceding an ordinary string literal with a u, as in u'hello'. Arbitrary Unicode characters are defined using the \uXXXX escape sequence inside a Unicode string literal. We find the integer ordinal of a character using ord(). For example: >>> ord('a') 97 The hexadecimal four-digit notation for 97 is 0061, so we can define a Unicode string literal with the appropriate escape sequence: >>> a = u'\u0061' >>> a u'a' >>> print a a 3.3 Text Processing with Unicode | 95 Notice that the Python print statement is assuming a default encoding of the Unicode character, namely ASCII. However, ń is outside the ASCII range, so cannot be printed unless we specify an encoding. In the following example, we have specified that print should use the repr() of the string, which outputs the UTF-8 escape sequences (of the form \xXX) rather than trying to render the glyphs. >>> nacute = u'\u0144' >>> nacute u'\u0144' >>> nacute_utf = nacute.encode('utf8') >>> print repr(nacute_utf) '\xc5\x84' If your operating system and locale are set up to render UTF-8 encoded characters, you ought to be able to give the Python command print nacute_utf and see ń on your screen. There are many factors determining what glyphs are rendered on your screen. If you are sure that you have the correct encoding, but your Python code is still failing to produce the glyphs you expected, you should also check that you have the necessary fonts installed on your system. The module unicodedata lets us inspect the properties of Unicode characters. In the following example, we select all characters in the third line of our Polish text outside the ASCII range and print their UTF-8 escaped value, followed by their code point integer using the standard Unicode convention (i.e., prefixing the hex digits with U+), followed by their Unicode name. >>> import unicodedata >>> lines = codecs.open(path, encoding='latin2').readlines() >>> line = lines[2] >>> print line.encode('unicode_escape') Niemc\xf3w pod koniec II wojny \u015bwiatowej na Dolny \u015al\u0105sk, zosta\u0142y\n >>> for c in line: ... if ord(c) > 127: ... print '%r U+%04x %s' % (c.encode('utf8'), ord(c), unicodedata.name(c)) '\xc3\xb3' U+00f3 LATIN SMALL LETTER O WITH ACUTE '\xc5\x9b' U+015b LATIN SMALL LETTER S WITH ACUTE '\xc5\x9a' U+015a LATIN CAPITAL LETTER S WITH ACUTE '\xc4\x85' U+0105 LATIN SMALL LETTER A WITH OGONEK '\xc5\x82' U+0142 LATIN SMALL LETTER L WITH STROKE If you replace the %r (which yields the repr() value) by %s in the format string of the preceding code sample, and if your system supports UTF-8, you should see an output like the following: ó U+00f3 LATIN SMALL LETTER O WITH ACUTE ś U+015b LATIN SMALL LETTER S WITH ACUTE Ś U+015a LATIN CAPITAL LETTER S WITH ACUTE 96 | Chapter 3: Processing Raw Text ą U+0105 LATIN SMALL LETTER A WITH OGONEK ł U+0142 LATIN SMALL LETTER L WITH STROKE Alternatively, you may need to replace the encoding 'utf8' in the example by 'latin2', again depending on the details of your system. The next examples illustrate how Python string methods and the re module accept Unicode strings. >>> line.find(u'zosta\u0142y') 54 >>> line = line.lower() >>> print line.encode('unicode_escape') niemc\xf3w pod koniec ii wojny \u015bwiatowej na dolny \u015bl\u0105sk, zosta\u0142y\n >>> import re >>> m = re.search(u'\u015b\w*', line) >>> m.group() u'\u015bwiatowej' NLTK tokenizers allow Unicode strings as input, and correspondingly yield Unicode strings as output. >>> nltk.word_tokenize(line) [u'niemc\xf3w', u'pod', u'koniec', u'ii', u'wojny', u'\u015bwiatowej', u'na', u'dolny', u'\u015bl\u0105sk', u'zosta\u0142y'] Using Your Local Encoding in Python If you are used to working with characters in a particular local encoding, you probably want to be able to use your standard methods for inputting and editing strings in a Python file. In order to do this, you need to include the string '# -*- coding: -*-' as the first or second line of your file. Note that has to be a string like 'latin-1', 'big5', or 'utf-8' (see Figure 3-4). Figure 3-4 also illustrates how regular expressions can use encoded strings. 3.4 Regular Expressions for Detecting Word Patterns Many linguistic processing tasks involve pattern matching. For example, we can find words ending with ed using endswith('ed'). We saw a variety of such “word tests” in Table 1-4. Regular expressions give us a more powerful and flexible method for de- scribing the character patterns we are interested in. There are many other published introductions to regular expressions, organized around the syntax of regular expressions and applied to searching text files. Instead of doing this again, we focus on the use of regular expressions at different stages of linguistic processing. As usual, we’ll adopt a problem-based approach and present new features only as they are needed to solve practical problems. In our discussion we will mark regular expressions using chevrons like this: «patt». 3.4 Regular Expressions for Detecting Word Patterns | 97 To use regular expressions in Python, we need to import the re library using: import re. We also need a list of words to search; we’ll use the Words Corpus again (Sec- tion 2.4). We will preprocess it to remove any proper names. >>> import re >>> wordlist = [w for w in nltk.corpus.words.words('en') if w.islower()] Using Basic Metacharacters Let’s find words ending with ed using the regular expression «ed$». We will use the re.search(p, s) function to check whether the pattern p can be found somewhere inside the string s. We need to specify the characters of interest, and use the dollar sign, which has a special behavior in the context of regular expressions in that it matches the end of the word: >>> [w for w in wordlist if re.search('ed$', w)] ['abaissed', 'abandoned', 'abased', 'abashed', 'abatised', 'abed', 'aborted', ...] The . wildcard symbol matches any single character. Suppose we have room in a crossword puzzle for an eight-letter word, with j as its third letter and t as its sixth letter. In place of each blank cell we use a period: >>> [w for w in wordlist if re.search('^..j..t..$', w)] ['abjectly', 'adjuster', 'dejected', 'dejectly', 'injector', 'majestic', ...] Figure 3-4. Unicode and IDLE: UTF-8 encoded string literals in the IDLE editor; this requires that an appropriate font is set in IDLE’s preferences; here we have chosen Courier CE. 98 | Chapter 3: Processing Raw Text Your Turn: The caret symbol ^ matches the start of a string, just like the $ matches the end. What results do we get with the example just shown if we leave out both of these, and search for «..j..t..»? Finally, the ? symbol specifies that the previous character is optional. Thus «^e-?mail $» will match both email and e-mail. We could count the total number of occurrences of this word (in either spelling) in a text using sum(1 for w in text if re.search('^e-? mail$', w)). Ranges and Closures The T9 system is used for entering text on mobile phones (see Figure 3-5). Two or more words that are entered with the same sequence of keystrokes are known as textonyms. For example, both hole and golf are entered by pressing the sequence 4653. What other words could be produced with the same sequence? Here we use the regular expression «^[ghi][mno][jlk][def]$»: >>> [w for w in wordlist if re.search('^[ghi][mno][jlk][def]$', w)] ['gold', 'golf', 'hold', 'hole'] The first part of the expression, «^[ghi]», matches the start of a word followed by g, h, or i. The next part of the expression, «[mno]», constrains the second character to be m, n, or o. The third and fourth characters are also constrained. Only four words satisfy all these constraints. Note that the order of characters inside the square brackets is not significant, so we could have written «^[hig][nom][ljk][fed]$» and matched the same words. Figure 3-5. T9: Text on 9 keys. Your Turn: Look for some “finger-twisters,” by searching for words that use only part of the number-pad. For example «^[ghijklmno]+$», or more concisely, «^[g-o]+$», will match words that only use keys 4, 5, 6 in the center row, and «^[a-fj-o]+$» will match words that use keys 2, 3, 5, 6 in the top-right corner. What do - and + mean? 3.4 Regular Expressions for Detecting Word Patterns | 99 Let’s explore the + symbol a bit further. Notice that it can be applied to individual letters, or to bracketed sets of letters: >>> chat_words = sorted(set(w for w in nltk.corpus.nps_chat.words())) >>> [w for w in chat_words if re.search('^m+i+n+e+$', w)] ['miiiiiiiiiiiiinnnnnnnnnnneeeeeeeeee', 'miiiiiinnnnnnnnnneeeeeeee', 'mine', 'mmmmmmmmiiiiiiiiinnnnnnnnneeeeeeee'] >>> [w for w in chat_words if re.search('^[ha]+$', w)] ['a', 'aaaaaaaaaaaaaaaaa', 'aaahhhh', 'ah', 'ahah', 'ahahah', 'ahh', 'ahhahahaha', 'ahhh', 'ahhhh', 'ahhhhhh', 'ahhhhhhhhhhhhhh', 'h', 'ha', 'haaa', 'hah', 'haha', 'hahaaa', 'hahah', 'hahaha', 'hahahaa', 'hahahah', 'hahahaha', ...] It should be clear that + simply means “one or more instances of the preceding item,” which could be an individual character like m, a set like [fed], or a range like [d-f]. Now let’s replace + with *, which means “zero or more instances of the preceding item.” The regular expression «^m*i*n*e*$» will match everything that we found using «^m+i +n+e+$», but also words where some of the letters don’t appear at all, e.g., me, min, and mmmmm. Note that the + and * symbols are sometimes referred to as Kleene clo- sures, or simply closures. The ^ operator has another function when it appears as the first character inside square brackets. For example, «[^aeiouAEIOU]» matches any character other than a vowel. We can search the NPS Chat Corpus for words that are made up entirely of non-vowel characters using «^[^aeiouAEIOU]+$» to find items like these: :):):), grrr, cyb3r, and zzzzzzzz. Notice this includes non-alphabetic characters. Here are some more examples of regular expressions being used to find tokens that match a particular pattern, illustrating the use of some new symbols: \, {}, (), and |. >>> wsj = sorted(set(nltk.corpus.treebank.words())) >>> [w for w in wsj if re.search('^[0-9]+\.[0-9]+$', w)] ['0.0085', '0.05', '0.1', '0.16', '0.2', '0.25', '0.28', '0.3', '0.4', '0.5', '0.50', '0.54', '0.56', '0.60', '0.7', '0.82', '0.84', '0.9', '0.95', '0.99', '1.01', '1.1', '1.125', '1.14', '1.1650', '1.17', '1.18', '1.19', '1.2', ...] >>> [w for w in wsj if re.search('^[A-Z]+\$$', w)] ['C$', 'US$'] >>> [w for w in wsj if re.search('^[0-9]{4}$', w)] ['1614', '1637', '1787', '1901', '1903', '1917', '1925', '1929', '1933', ...] >>> [w for w in wsj if re.search('^[0-9]+-[a-z]{3,5}$', w)] ['10-day', '10-lap', '10-year', '100-share', '12-point', '12-year', ...] >>> [w for w in wsj if re.search('^[a-z]{5,}-[a-z]{2,3}-[a-z]{,6}$', w)] ['black-and-white', 'bread-and-butter', 'father-in-law', 'machine-gun-toting', 'savings-and-loan'] >>> [w for w in wsj if re.search('(ed|ing)$', w)] ['62%-owned', 'Absorbed', 'According', 'Adopting', 'Advanced', 'Advancing', ...] Your Turn: Study the previous examples and try to work out what the \, {}, (), and | notations mean before you read on. 100 | Chapter 3: Processing Raw Text You probably worked out that a backslash means that the following character is de- prived of its special powers and must literally match a specific character in the word. Thus, while . is special, \. only matches a period. The braced expressions, like {3,5}, specify the number of repeats of the previous item. The pipe character indicates a choice between the material on its left or its right. Parentheses indicate the scope of an oper- ator, and they can be used together with the pipe (or disjunction) symbol like this: «w(i|e|ai|oo)t», matching wit, wet, wait, and woot. It is instructive to see what happens when you omit the parentheses from the last expression in the example, and search for «ed|ing$». The metacharacters we have seen are summarized in Table 3-3. Table 3-3. Basic regular expression metacharacters, including wildcards, ranges, and closures Operator Behavior . Wildcard, matches any character ^abc Matches some pattern abc at the start of a string abc$ Matches some pattern abc at the end of a string [abc] Matches one of a set of characters [A-Z0-9] Matches one of a range of characters ed|ing|s Matches one of the specified strings (disjunction) * Zero or more of previous item, e.g., a*, [a-z]* (also known as Kleene Closure) + One or more of previous item, e.g., a+, [a-z]+ ? Zero or one of the previous item (i.e., optional), e.g., a?, [a-z]? {n} Exactly n repeats where n is a non-negative integer {n,} At least n repeats {,n} No more than n repeats {m,n} At least m and no more than n repeats a(b|c)+ Parentheses that indicate the scope of the operators To the Python interpreter, a regular expression is just like any other string. If the string contains a backslash followed by particular characters, it will interpret these specially. For example, \b would be interpreted as the backspace character. In general, when using regular expressions containing backslash, we should instruct the interpreter not to look inside the string at all, but simply to pass it directly to the re library for pro- cessing. We do this by prefixing the string with the letter r, to indicate that it is a raw string. For example, the raw string r'\band\b' contains two \b symbols that are interpreted by the re library as matching word boundaries instead of backspace char- acters. If you get into the habit of using r'...' for regular expressions—as we will do from now on—you will avoid having to think about these complications. 3.4 Regular Expressions for Detecting Word Patterns | 101 3.5 Useful Applications of Regular Expressions The previous examples all involved searching for words w that match some regular expression regexp using re.search(regexp, w). Apart from checking whether a regular expression matches a word, we can use regular expressions to extract material from words, or to modify words in specific ways. Extracting Word Pieces The re.findall() (“find all”) method finds all (non-overlapping) matches of the given regular expression. Let’s find all the vowels in a word, then count them: >>> word = 'supercalifragilisticexpialidocious' >>> re.findall(r'[aeiou]', word) ['u', 'e', 'a', 'i', 'a', 'i', 'i', 'i', 'e', 'i', 'a', 'i', 'o', 'i', 'o', 'u'] >>> len(re.findall(r'[aeiou]', word)) 16 Let’s look for all sequences of two or more vowels in some text, and determine their relative frequency: >>> wsj = sorted(set(nltk.corpus.treebank.words())) >>> fd = nltk.FreqDist(vs for word in wsj ... for vs in re.findall(r'[aeiou]{2,}', word)) >>> fd.items() [('io', 549), ('ea', 476), ('ie', 331), ('ou', 329), ('ai', 261), ('ia', 253), ('ee', 217), ('oo', 174), ('ua', 109), ('au', 106), ('ue', 105), ('ui', 95), ('ei', 86), ('oi', 65), ('oa', 59), ('eo', 39), ('iou', 27), ('eu', 18), ...] Your Turn: In the W3C Date Time Format, dates are represented like this: 2009-12-31. Replace the ? in the following Python code with a regular expression, in order to convert the string '2009-12-31' to a list of integers [2009, 12, 31]: [int(n) for n in re.findall(?, '2009-12-31')] Doing More with Word Pieces Once we can use re.findall() to extract material from words, there are interesting things to do with the pieces, such as glue them back together or plot them. It is sometimes noted that English text is highly redundant, and it is still easy to read when word-internal vowels are left out. For example, declaration becomes dclrtn, and inalienable becomes inlnble, retaining any initial or final vowel sequences. The regular expression in our next example matches initial vowel sequences, final vowel sequences, and all consonants; everything else is ignored. This three-way disjunction is processed left-to-right, and if one of the three parts matches the word, any later parts of the regular expression are ignored. We use re.findall() to extract all the matching pieces, and ''.join() to join them together (see Section 3.9 for more about the join operation). 102 | Chapter 3: Processing Raw Text >>> regexp = r'^[AEIOUaeiou]+|[AEIOUaeiou]+$|[^AEIOUaeiou]' >>> def compress(word): ... pieces = re.findall(regexp, word) ... return ''.join(pieces) ... >>> english_udhr = nltk.corpus.udhr.words('English-Latin1') >>> print nltk.tokenwrap(compress(w) for w in english_udhr[:75]) Unvrsl Dclrtn of Hmn Rghts Prmble Whrs rcgntn of the inhrnt dgnty and of the eql and inlnble rghts of all mmbrs of the hmn fmly is the fndtn of frdm , jstce and pce in the wrld , Whrs dsrgrd and cntmpt fr hmn rghts hve rsltd in brbrs acts whch hve outrgd the cnscnce of mnknd , and the advnt of a wrld in whch hmn bngs shll enjy frdm of spch and Next, let’s combine regular expressions with conditional frequency distributions. Here we will extract all consonant-vowel sequences from the words of Rotokas, such as ka and si. Since each of these is a pair, it can be used to initialize a conditional frequency distribution. We then tabulate the frequency of each pair: >>> rotokas_words = nltk.corpus.toolbox.words('rotokas.dic') >>> cvs = [cv for w in rotokas_words for cv in re.findall(r'[ptksvr][aeiou]', w)] >>> cfd = nltk.ConditionalFreqDist(cvs) >>> cfd.tabulate() a e i o u k 418 148 94 420 173 p 83 31 105 34 51 r 187 63 84 89 79 s 0 0 100 2 1 t 47 8 0 148 37 v 93 27 105 48 49 Examining the rows for s and t, we see they are in partial “complementary distribution,” which is evidence that they are not distinct phonemes in the language. Thus, we could conceivably drop s from the Rotokas alphabet and simply have a pronunciation rule that the letter t is pronounced s when followed by i. (Note that the single entry having su, namely kasuari, ‘cassowary’ is borrowed from English). If we want to be able to inspect the words behind the numbers in that table, it would be helpful to have an index, allowing us to quickly find the list of words that contains a given consonant-vowel pair. For example, cv_index['su'] should give us all words containing su. Here’s how we can do this: >>> cv_word_pairs = [(cv, w) for w in rotokas_words ... for cv in re.findall(r'[ptksvr][aeiou]', w)] >>> cv_index = nltk.Index(cv_word_pairs) >>> cv_index['su'] ['kasuari'] >>> cv_index['po'] ['kaapo', 'kaapopato', 'kaipori', 'kaiporipie', 'kaiporivira', 'kapo', 'kapoa', 'kapokao', 'kapokapo', 'kapokapo', 'kapokapoa', 'kapokapoa', 'kapokapora', ...] This program processes each word w in turn, and for each one, finds every substring that matches the regular expression «[ptksvr][aeiou]». In the case of the word ka- suari, it finds ka, su, and ri. Therefore, the cv_word_pairs list will contain ('ka', 'ka 3.5 Useful Applications of Regular Expressions | 103 suari'), ('su', 'kasuari'), and ('ri', 'kasuari'). One further step, using nltk.Index(), converts this into a useful index. Finding Word Stems When we use a web search engine, we usually don’t mind (or even notice) if the words in the document differ from our search terms in having different endings. A query for laptops finds documents containing laptop and vice versa. Indeed, laptop and laptops are just two forms of the same dictionary word (or lemma). For some language pro- cessing tasks we want to ignore word endings, and just deal with word stems. There are various ways we can pull out the stem of a word. Here’s a simple-minded approach that just strips off anything that looks like a suffix: >>> def stem(word): ... for suffix in ['ing', 'ly', 'ed', 'ious', 'ies', 'ive', 'es', 's', 'ment']: ... if word.endswith(suffix): ... return word[:-len(suffix)] ... return word Although we will ultimately use NLTK’s built-in stemmers, it’s interesting to see how we can use regular expressions for this task. Our first step is to build up a disjunction of all the suffixes. We need to enclose it in parentheses in order to limit the scope of the disjunction. >>> re.findall(r'^.*(ing|ly|ed|ious|ies|ive|es|s|ment)$', 'processing') ['ing'] Here, re.findall() just gave us the suffix even though the regular expression matched the entire word. This is because the parentheses have a second function, to select sub- strings to be extracted. If we want to use the parentheses to specify the scope of the disjunction, but not to select the material to be output, we have to add ?:, which is just one of many arcane subtleties of regular expressions. Here’s the revised version. >>> re.findall(r'^.*(?:ing|ly|ed|ious|ies|ive|es|s|ment)$', 'processing') ['processing'] However, we’d actually like to split the word into stem and suffix. So we should just parenthesize both parts of the regular expression: >>> re.findall(r'^(.*)(ing|ly|ed|ious|ies|ive|es|s|ment)$', 'processing') [('process', 'ing')] This looks promising, but still has a problem. Let’s look at a different word, processes: >>> re.findall(r'^(.*)(ing|ly|ed|ious|ies|ive|es|s|ment)$', 'processes') [('processe', 's')] The regular expression incorrectly found an -s suffix instead of an -es suffix. This dem- onstrates another subtlety: the star operator is “greedy” and so the .* part of the ex- pression tries to consume as much of the input as possible. If we use the “non-greedy” version of the star operator, written *?, we get what we want: 104 | Chapter 3: Processing Raw Text >>> re.findall(r'^(.*?)(ing|ly|ed|ious|ies|ive|es|s|ment)$', 'processes') [('process', 'es')] This works even when we allow an empty suffix, by making the content of the second parentheses optional: >>> re.findall(r'^(.*?)(ing|ly|ed|ious|ies|ive|es|s|ment)?$', 'language') [('language', '')] This approach still has many problems (can you spot them?), but we will move on to define a function to perform stemming, and apply it to a whole text: >>> def stem(word): ... regexp = r'^(.*?)(ing|ly|ed|ious|ies|ive|es|s|ment)?$' ... stem, suffix = re.findall(regexp, word)[0] ... return stem ... >>> raw = """DENNIS: Listen, strange women lying in ponds distributing swords ... is no basis for a system of government. Supreme executive power derives from ... a mandate from the masses, not from some farcical aquatic ceremony.""" >>> tokens = nltk.word_tokenize(raw) >>> [stem(t) for t in tokens] ['DENNIS', ':', 'Listen', ',', 'strange', 'women', 'ly', 'in', 'pond', 'distribut', 'sword', 'i', 'no', 'basi', 'for', 'a', 'system', 'of', 'govern', '.', 'Supreme', 'execut', 'power', 'deriv', 'from', 'a', 'mandate', 'from', 'the', 'mass', ',', 'not', 'from', 'some', 'farcical', 'aquatic', 'ceremony', '.'] Notice that our regular expression removed the s from ponds but also from is and basis. It produced some non-words, such as distribut and deriv, but these are acceptable stems in some applications. Searching Tokenized Text You can use a special kind of regular expression for searching across multiple words in a text (where a text is a list of tokens). For example, " " finds all instances of a man in the text. The angle brackets are used to mark token boundaries, and any whitespace between the angle brackets is ignored (behaviors that are unique to NLTK’s findall() method for texts). In the following example, we include <.*> , which will match any single token, and enclose it in parentheses so only the matched word (e.g., monied) and not the matched phrase (e.g., a monied man) is produced. The second example finds three-word phrases ending with the word bro . The last example finds sequences of three or more words starting with the letter l . >>> from nltk.corpus import gutenberg, nps_chat >>> moby = nltk.Text(gutenberg.words('melville-moby_dick.txt')) >>> moby.findall(r" (<.*>) ") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> chat = nltk.Text(nps_chat.words()) >>> chat.findall(r"<.*> <.*> ") you rule bro; telling you bro; u twizted bro 3.5 Useful Applications of Regular Expressions | 105 >>> chat.findall(r"{3,}") lol lol lol; lmao lol lol; lol lol lol; la la la la la; la la la; la la la; lovely lol lol love; lol lol lol.; la la la; la la la Your Turn: Consolidate your understanding of regular expression pat- terns and substitutions using nltk.re_show(p, s), which annotates the string s to show every place where pattern p was matched, and nltk.app.nemo(), which provides a graphical interface for exploring reg- ular expressions. For more practice, try some of the exercises on regular expressions at the end of this chapter. It is easy to build search patterns when the linguistic phenomenon we’re studying is tied to particular words. In some cases, a little creativity will go a long way. For instance, searching a large text corpus for expressions of the form x and other ys allows us to discover hypernyms (see Section 2.5): >>> from nltk.corpus import brown >>> hobbies_learned = nltk.Text(brown.words(categories=['hobbies', 'learned'])) >>> hobbies_learned.findall(r"<\w*> <\w*s>") speed and other activities; water and other liquids; tomb and other landmarks; Statues and other monuments; pearls and other jewels; charts and other items; roads and other features; figures and other objects; military and other areas; demands and other factors; abstracts and other compilations; iron and other metals With enough text, this approach would give us a useful store of information about the taxonomy of objects, without the need for any manual labor. However, our search results will usually contain false positives, i.e., cases that we would want to exclude. For example, the result demands and other factors suggests that demand is an instance of the type factor, but this sentence is actually about wage demands. Nevertheless, we could construct our own ontology of English concepts by manually correcting the out- put of such searches. This combination of automatic and manual processing is the most com- mon way for new corpora to be constructed. We will return to this in Chapter 11. Searching corpora also suffers from the problem of false negatives, i.e., omitting cases that we would want to include. It is risky to conclude that some linguistic phenomenon doesn’t exist in a corpus just because we couldn’t find any instances of a search pattern. Perhaps we just didn’t think carefully enough about suitable patterns. Your Turn: Look for instances of the pattern as x as y to discover in- formation about entities and their properties. 106 | Chapter 3: Processing Raw Text 3.6 Normalizing Text In earlier program examples we have often converted text to lowercase before doing anything with its words, e.g., set(w.lower() for w in text). By using lower(), we have normalized the text to lowercase so that the distinction between The and the is ignored. Often we want to go further than this and strip off any affixes, a task known as stem- ming. A further step is to make sure that the resulting form is a known word in a dictionary, a task known as lemmatization. We discuss each of these in turn. First, we need to define the data we will use in this section: >>> raw = """DENNIS: Listen, strange women lying in ponds distributing swords ... is no basis for a system of government. Supreme executive power derives from ... a mandate from the masses, not from some farcical aquatic ceremony.""" >>> tokens = nltk.word_tokenize(raw) Stemmers NLTK includes several off-the-shelf stemmers, and if you ever need a stemmer, you should use one of these in preference to crafting your own using regular expressions, since NLTK’s stemmers handle a wide range of irregular cases. The Porter and Lan- caster stemmers follow their own rules for stripping affixes. Observe that the Porter stemmer correctly handles the word lying (mapping it to lie), whereas the Lancaster stemmer does not. >>> porter = nltk.PorterStemmer() >>> lancaster = nltk.LancasterStemmer() >>> [porter.stem(t) for t in tokens] ['DENNI', ':', 'Listen', ',', 'strang', 'women', 'lie', 'in', 'pond', 'distribut', 'sword', 'is', 'no', 'basi', 'for', 'a', 'system', 'of', 'govern', '.', 'Suprem', 'execut', 'power', 'deriv', 'from', 'a', 'mandat', 'from', 'the', 'mass', ',', 'not', 'from', 'some', 'farcic', 'aquat', 'ceremoni', '.'] >>> [lancaster.stem(t) for t in tokens] ['den', ':', 'list', ',', 'strange', 'wom', 'lying', 'in', 'pond', 'distribut', 'sword', 'is', 'no', 'bas', 'for', 'a', 'system', 'of', 'govern', '.', 'suprem', 'execut', 'pow', 'der', 'from', 'a', 'mand', 'from', 'the', 'mass', ',', 'not', 'from', 'som', 'farc', 'aqu', 'ceremony', '.'] Stemming is not a well-defined process, and we typically pick the stemmer that best suits the application we have in mind. The Porter Stemmer is a good choice if you are indexing some texts and want to support search using alternative forms of words (il- lustrated in Example 3-1, which uses object-oriented programming techniques that are outside the scope of this book, string formatting techniques to be covered in Sec- tion 3.9, and the enumerate() function to be explained in Section 4.2). Example 3-1. Indexing a text using a stemmer. class IndexedText(object): def __init__(self, stemmer, text): self._text = text self._stemmer = stemmer 3.6 Normalizing Text | 107 self._index = nltk.Index((self._stem(word), i) for (i, word) in enumerate(text)) def concordance(self, word, width=40): key = self._stem(word) wc = width/4 # words of context for i in self._index[key]: lcontext = ' '.join(self._text[i-wc:i]) rcontext = ' '.join(self._text[i:i+wc]) ldisplay = '%*s' % (width, lcontext[-width:]) rdisplay = '%-*s' % (width, rcontext[:width]) print ldisplay, rdisplay def _stem(self, word): return self._stemmer.stem(word).lower() >>> porter = nltk.PorterStemmer() >>> grail = nltk.corpus.webtext.words('grail.txt') >>> text = IndexedText(porter, grail) >>> text.concordance('lie') r king ! DENNIS : Listen , strange women lying in ponds distributing swords is no beat a very brave retreat . ROBIN : All lies ! MINSTREL : [ singing ] Bravest of Nay . Nay . Come . Come . You may lie here . Oh , but you are wounded ! doctors immediately ! No , no , please ! Lie down . [ clap clap ] PIGLET : Well ere is much danger , for beyond the cave lies the Gorge of Eternal Peril , which you . Oh ... TIM : To the north there lies a cave -- the cave of Caerbannog -- h it and lived ! Bones of full fifty men lie strewn about its lair . So , brave k not stop our fight ' til each one of you lies dead , and the Holy Grail returns t Lemmatization The WordNet lemmatizer removes affixes only if the resulting word is in its dictionary. This additional checking process makes the lemmatizer slower than the stemmers just mentioned. Notice that it doesn’t handle lying, but it converts women to woman. >>> wnl = nltk.WordNetLemmatizer() >>> [wnl.lemmatize(t) for t in tokens] ['DENNIS', ':', 'Listen', ',', 'strange', 'woman', 'lying', 'in', 'pond', 'distributing', 'sword', 'is', 'no', 'basis', 'for', 'a', 'system', 'of', 'government', '.', 'Supreme', 'executive', 'power', 'derives', 'from', 'a', 'mandate', 'from', 'the', 'mass', ',', 'not', 'from', 'some', 'farcical', 'aquatic', 'ceremony', '.'] The WordNet lemmatizer is a good choice if you want to compile the vocabulary of some texts and want a list of valid lemmas (or lexicon headwords). Another normalization task involves identifying non-standard words, including numbers, abbreviations, and dates, and mapping any such tokens to a special vocabulary. For example, every decimal number could be mapped to a single token 0.0, and every acronym could be mapped to AAA. This keeps the vocabulary small and improves the ac- curacy of many language modeling tasks. 108 | Chapter 3: Processing Raw Text 3.7 Regular Expressions for Tokenizing Text Tokenization is the task of cutting a string into identifiable linguistic units that consti- tute a piece of language data. Although it is a fundamental task, we have been able to delay it until now because many corpora are already tokenized, and because NLTK includes some tokenizers. Now that you are familiar with regular expressions, you can learn how to use them to tokenize text, and to have much more control over the process. Simple Approaches to Tokenization The very simplest method for tokenizing text is to split on whitespace. Consider the following text from Alice’s Adventures in Wonderland: >>> raw = """'When I'M a Duchess,' she said to herself, (not in a very hopeful tone ... though), 'I won't have any pepper in my kitchen AT ALL. Soup does very ... well without--Maybe it's always pepper that makes people hot-tempered,'...""" We could split this raw text on whitespace using raw.split(). To do the same using a regular expression, it is not enough to match any space characters in the string , since this results in tokens that contain a \n newline character; instead, we need to match any number of spaces, tabs, or newlines : >>> re.split(r' ', raw) ["'When", "I'M", 'a', "Duchess,'", 'she', 'said', 'to', 'herself,', '(not', 'in', 'a', 'very', 'hopeful', 'tone\nthough),', "'I", "won't", 'have', 'any', 'pepper', 'in', 'my', 'kitchen', 'AT', 'ALL.', 'Soup', 'does', 'very\nwell', 'without--Maybe', "it's", 'always', 'pepper', 'that', 'makes', 'people', "hot-tempered,'..."] >>> re.split(r'[ \t\n]+', raw) ["'When", "I'M", 'a', "Duchess,'", 'she', 'said', 'to', 'herself,', '(not', 'in', 'a', 'very', 'hopeful', 'tone', 'though),', "'I", "won't", 'have', 'any', 'pepper', 'in', 'my', 'kitchen', 'AT', 'ALL.', 'Soup', 'does', 'very', 'well', 'without--Maybe', "it's", 'always', 'pepper', 'that', 'makes', 'people', "hot-tempered,'..."] The regular expression «[ \t\n]+» matches one or more spaces, tabs (\t), or newlines (\n). Other whitespace characters, such as carriage return and form feed, should really be included too. Instead, we will use a built-in re abbreviation, \s, which means any whitespace character. The second statement in the preceding example can be rewritten as re.split(r'\s+', raw). Important: Remember to prefix regular expressions with the letter r (meaning “raw”), which instructs the Python interpreter to treat the string literally, rather than processing any backslashed characters it contains. Splitting on whitespace gives us tokens like '(not' and 'herself,'. An alternative is to use the fact that Python provides us with a character class \w for word characters, equivalent to [a-zA-Z0-9_]. It also defines the complement of this class, \W, i.e., all 3.7 Regular Expressions for Tokenizing Text | 109 characters other than letters, digits, or underscore. We can use \W in a simple regular expression to split the input on anything other than a word character: >>> re.split(r'\W+', raw) ['', 'When', 'I', 'M', 'a', 'Duchess', 'she', 'said', 'to', 'herself', 'not', 'in', 'a', 'very', 'hopeful', 'tone', 'though', 'I', 'won', 't', 'have', 'any', 'pepper', 'in', 'my', 'kitchen', 'AT', 'ALL', 'Soup', 'does', 'very', 'well', 'without', 'Maybe', 'it', 's', 'always', 'pepper', 'that', 'makes', 'people', 'hot', 'tempered', ''] Observe that this gives us empty strings at the start and the end (to understand why, try doing 'xx'.split('x')). With re.findall(r'\w+', raw), we get the same tokens, but without the empty strings, using a pattern that matches the words instead of the spaces. Now that we’re matching the words, we’re in a position to extend the regular expression to cover a wider range of cases. The regular expression «\w+|\S\w*» will first try to match any sequence of word characters. If no match is found, it will try to match any non-whitespace character (\S is the complement of \s) followed by further word characters. This means that punctuation is grouped with any following letters (e.g., ’s) but that sequences of two or more punctuation characters are separated. >>> re.findall(r'\w+|\S\w*', raw) ["'When", 'I', "'M", 'a', 'Duchess', ',', "'", 'she', 'said', 'to', 'herself', ',', '(not', 'in', 'a', 'very', 'hopeful', 'tone', 'though', ')', ',', "'I", 'won', "'t", 'have', 'any', 'pepper', 'in', 'my', 'kitchen', 'AT', 'ALL', '.', 'Soup', 'does', 'very', 'well', 'without', '-', '-Maybe', 'it', "'s", 'always', 'pepper', 'that', 'makes', 'people', 'hot', '-tempered', ',', "'", '.', '.', '.'] Let’s generalize the \w+ in the preceding expression to permit word-internal hyphens and apostrophes: «\w+([-']\w+)*». This expression means \w+ followed by zero or more instances of [-']\w+; it would match hot-tempered and it’s. (We need to include ?: in this expression for reasons discussed earlier.) We’ll also add a pattern to match quote characters so these are kept separate from the text they enclose. >>> print re.findall(r"\w+(?:[-']\w+)*|'|[-.(]+|\S\w*", raw) ["'", 'When', "I'M", 'a', 'Duchess', ',', "'", 'she', 'said', 'to', 'herself', ',', '(', 'not', 'in', 'a', 'very', 'hopeful', 'tone', 'though', ')', ',', "'", 'I', "won't", 'have', 'any', 'pepper', 'in', 'my', 'kitchen', 'AT', 'ALL', '.', 'Soup', 'does', 'very', 'well', 'without', '--', 'Maybe', "it's", 'always', 'pepper', 'that', 'makes', 'people', 'hot-tempered', ',', "'", '...'] The expression in this example also included «[-.(]+», which causes the double hy- phen, ellipsis, and open parenthesis to be tokenized separately. Table 3-4 lists the regular expression character class symbols we have seen in this sec- tion, in addition to some other useful symbols. Table 3-4. Regular expression symbols Symbol Function \b Word boundary (zero width) \d Any decimal digit (equivalent to [0-9]) 110 | Chapter 3: Processing Raw Text Symbol Function \D Any non-digit character (equivalent to [^0-9]) \s Any whitespace character (equivalent to [ \t\n\r\f\v] \S Any non-whitespace character (equivalent to [^ \t\n\r\f\v]) \w Any alphanumeric character (equivalent to [a-zA-Z0-9_]) \W Any non-alphanumeric character (equivalent to [^a-zA-Z0-9_]) \t The tab character \n The newline character NLTK’s Regular Expression Tokenizer The function nltk.regexp_tokenize() is similar to re.findall() (as we’ve been using it for tokenization). However, nltk.regexp_tokenize() is more efficient for this task, and avoids the need for special treatment of parentheses. For readability we break up the regular expression over several lines and add a comment about each line. The special (?x) “verbose flag” tells Python to strip out the embedded whitespace and comments. >>> text = 'That U.S.A. poster-print costs $12.40...' >>> pattern = r'''(?x) # set flag to allow verbose regexps ... ([A-Z]\.)+ # abbreviations, e.g. U.S.A. ... | \w+(-\w+)* # words with optional internal hyphens ... | \$?\d+(\.\d+)?%? # currency and percentages, e.g. $12.40, 82% ... | \.\.\. # ellipsis ... | [][.,;"'?():-_`] # these are separate tokens ... ''' >>> nltk.regexp_tokenize(text, pattern) ['That', 'U.S.A.', 'poster-print', 'costs', '$12.40', '...'] When using the verbose flag, you can no longer use ' ' to match a space character; use \s instead. The regexp_tokenize() function has an optional gaps parameter. When set to True, the regular expression specifies the gaps between tokens, as with re.split(). We can evaluate a tokenizer by comparing the resulting tokens with a wordlist, and then report any tokens that don’t appear in the wordlist, using set(tokens).difference(wordlist). You’ll probably want to lowercase all the tokens first. Further Issues with Tokenization Tokenization turns out to be a far more difficult task than you might have expected. No single solution works well across the board, and we must decide what counts as a token depending on the application domain. When developing a tokenizer it helps to have access to raw text which has been man- ually tokenized, in order to compare the output of your tokenizer with high-quality (or 3.7 Regular Expressions for Tokenizing Text | 111 “gold-standard”) tokens. The NLTK corpus collection includes a sample of Penn Tree- bank data, including the raw Wall Street Journal text (nltk.corpus.tree bank_raw.raw()) and the tokenized version (nltk.corpus.treebank.words()). A final issue for tokenization is the presence of contractions, such as didn’t. If we are analyzing the meaning of a sentence, it would probably be more useful to normalize this form to two separate forms: did and n’t (or not). We can do this work with the help of a lookup table. 3.8 Segmentation This section discusses more advanced concepts, which you may prefer to skip on the first time through this chapter. Tokenization is an instance of a more general problem of segmentation. In this section, we will look at two other instances of this problem, which use radically different tech- niques to the ones we have seen so far in this chapter. Sentence Segmentation Manipulating texts at the level of individual words often presupposes the ability to divide a text into individual sentences. As we have seen, some corpora already provide access at the sentence level. In the following example, we compute the average number of words per sentence in the Brown Corpus: >>> len(nltk.corpus.brown.words()) / len(nltk.corpus.brown.sents()) 20.250994070456922 In other cases, the text is available only as a stream of characters. Before tokenizing the text into words, we need to segment it into sentences. NLTK facilitates this by including the Punkt sentence segmenter (Kiss & Strunk, 2006). Here is an example of its use in segmenting the text of a novel. (Note that if the segmenter’s internal data has been updated by the time you read this, you will see different output.) >>> sent_tokenizer=nltk.data.load('tokenizers/punkt/english.pickle') >>> text = nltk.corpus.gutenberg.raw('chesterton-thursday.txt') >>> sents = sent_tokenizer.tokenize(text) >>> pprint.pprint(sents[171:181]) ['"Nonsense!', '" said Gregory, who was very rational when anyone else\nattempted paradox.', '"Why do all the clerks and navvies in the\nrailway trains look so sad and tired,...', 'I will\ntell you.', 'It is because they know that the train is going right.', 'It\nis because they know that whatever place they have taken a ticket\nfor that ...', 'It is because after they have\npassed Sloane Square they know that the next stat...', 'Oh, their wild rapture!', 'oh,\ntheir eyes like stars and their souls again in Eden, if the next\nstation w...' '"\n\n"It is you who are unpoetical," replied the poet Syme.'] 112 | Chapter 3: Processing Raw Text Notice that this example is really a single sentence, reporting the speech of Mr. Lucian Gregory. However, the quoted speech contains several sentences, and these have been split into individual strings. This is reasonable behavior for most applications. Sentence segmentation is difficult because a period is used to mark abbreviations, and some periods simultaneously mark an abbreviation and terminate a sentence, as often happens with acronyms like U.S.A. For another approach to sentence segmentation, see Section 6.2. Word Segmentation For some writing systems, tokenizing text is made more difficult by the fact that there is no visual representation of word boundaries. For example, in Chinese, the three- character string: 爱国人 (ai4 “love” [verb], guo3 “country”, ren2 “person”) could be tokenized as 爱国 / 人, “country-loving person,” or as 爱 / 国人, “love country-person.” A similar problem arises in the processing of spoken language, where the hearer must segment a continuous speech stream into individual words. A particularly challenging version of this problem arises when we don’t know the words in advance. This is the problem faced by a language learner, such as a child hearing utterances from a parent. Consider the following artificial example, where word boundaries have been removed: (1) a. doyouseethekitty b. seethedoggy c. doyoulikethekitty d. likethedoggy Our first challenge is simply to represent the problem: we need to find a way to separate text content from the segmentation. We can do this by annotating each character with a boolean value to indicate whether or not a word-break appears after the character (an idea that will be used heavily for “chunking” in Chapter 7). Let’s assume that the learner is given the utterance breaks, since these often correspond to extended pauses. Here is a possible representation, including the initial and target segmentations: >>> text = "doyouseethekittyseethedoggydoyoulikethekittylikethedoggy" >>> seg1 = "0000000000000001000000000010000000000000000100000000000" >>> seg2 = "0100100100100001001001000010100100010010000100010010000" Observe that the segmentation strings consist of zeros and ones. They are one character shorter than the source text, since a text of length n can be broken up in only n–1 places. The segment() function in Example 3-2 demonstrates that we can get back to the orig- inal segmented text from its representation. 3.8 Segmentation | 113 Example 3-2. Reconstruct segmented text from string representation: seg1 and seg2 represent the initial and final segmentations of some hypothetical child-directed speech; the segment() function can use them to reproduce the segmented text. def segment(text, segs): words = [] last = 0 for i in range(len(segs)): if segs[i] == '1': words.append(text[last:i+1]) last = i+1 words.append(text[last:]) return words >>> text = "doyouseethekittyseethedoggydoyoulikethekittylikethedoggy" >>> seg1 = "0000000000000001000000000010000000000000000100000000000" >>> seg2 = "0100100100100001001001000010100100010010000100010010000" >>> segment(text, seg1) ['doyouseethekitty', 'seethedoggy', 'doyoulikethekitty', 'likethedoggy'] >>> segment(text, seg2) ['do', 'you', 'see', 'the', 'kitty', 'see', 'the', 'doggy', 'do', 'you', 'like', 'the', kitty', 'like', 'the', 'doggy'] Now the segmentation task becomes a search problem: find the bit string that causes the text string to be correctly segmented into words. We assume the learner is acquiring words and storing them in an internal lexicon. Given a suitable lexicon, it is possible to reconstruct the source text as a sequence of lexical items. Following (Brent & Cart- wright, 1995), we can define an objective function, a scoring function whose value we will try to optimize, based on the size of the lexicon and the amount of information needed to reconstruct the source text from the lexicon. We illustrate this in Figure 3-6. Figure 3-6. Calculation of objective function: Given a hypothetical segmentation of the source text (on the left), derive a lexicon and a derivation table that permit the source text to be reconstructed, then total up the number of characters used by each lexical item (including a boundary marker) and each derivation, to serve as a score of the quality of the segmentation; smaller values of the score indicate a better segmentation. It is a simple matter to implement this objective function, as shown in Example 3-3. 114 | Chapter 3: Processing Raw Text Example 3-3. Computing the cost of storing the lexicon and reconstructing the source text. def evaluate(text, segs): words = segment(text, segs) text_size = len(words) lexicon_size = len(' '.join(list(set(words)))) return text_size + lexicon_size >>> text = "doyouseethekittyseethedoggydoyoulikethekittylikethedoggy" >>> seg1 = "0000000000000001000000000010000000000000000100000000000" >>> seg2 = "0100100100100001001001000010100100010010000100010010000" >>> seg3 = "0000100100000011001000000110000100010000001100010000001" >>> segment(text, seg3) ['doyou', 'see', 'thekitt', 'y', 'see', 'thedogg', 'y', 'doyou', 'like', 'thekitt', 'y', 'like', 'thedogg', 'y'] >>> evaluate(text, seg3) 46 >>> evaluate(text, seg2) 47 >>> evaluate(text, seg1) 63 The final step is to search for the pattern of zeros and ones that maximizes this objective function, shown in Example 3-4. Notice that the best segmentation includes “words” like thekitty, since there’s not enough evidence in the data to split this any further. Example 3-4. Non-deterministic search using simulated annealing: Begin searching with phrase segmentations only; randomly perturb the zeros and ones proportional to the “temperature”; with each iteration the temperature is lowered and the perturbation of boundaries is reduced. from random import randint def flip(segs, pos): return segs[:pos] + str(1-int(segs[pos])) + segs[pos+1:] def flip_n(segs, n): for i in range(n): segs = flip(segs, randint(0,len(segs)-1)) return segs def anneal(text, segs, iterations, cooling_rate): temperature = float(len(segs)) while temperature > 0.5: best_segs, best = segs, evaluate(text, segs) for i in range(iterations): guess = flip_n(segs, int(round(temperature))) score = evaluate(text, guess) if score < best: best, best_segs = score, guess score, segs = best, best_segs temperature = temperature / cooling_rate print evaluate(text, segs), segment(text, segs) 3.8 Segmentation | 115 print return segs >>> text = "doyouseethekittyseethedoggydoyoulikethekittylikethedoggy" >>> seg1 = "0000000000000001000000000010000000000000000100000000000" >>> anneal(text, seg1, 5000, 1.2) 60 ['doyouseetheki', 'tty', 'see', 'thedoggy', 'doyouliketh', 'ekittylike', 'thedoggy'] 58 ['doy', 'ouseetheki', 'ttysee', 'thedoggy', 'doy', 'o', 'ulikethekittylike', 'thedoggy'] 56 ['doyou', 'seetheki', 'ttysee', 'thedoggy', 'doyou', 'liketh', 'ekittylike', 'thedoggy'] 54 ['doyou', 'seethekit', 'tysee', 'thedoggy', 'doyou', 'likethekittylike', 'thedoggy'] 53 ['doyou', 'seethekit', 'tysee', 'thedoggy', 'doyou', 'like', 'thekitty', 'like', 'thedoggy'] 51 ['doyou', 'seethekittysee', 'thedoggy', 'doyou', 'like', 'thekitty', 'like', 'thedoggy'] 42 ['doyou', 'see', 'thekitty', 'see', 'thedoggy', 'doyou', 'like', 'thekitty', 'like', 'thedoggy'] '0000100100000001001000000010000100010000000100010000000' With enough data, it is possible to automatically segment text into words with a rea- sonable degree of accuracy. Such methods can be applied to tokenization for writing systems that don’t have any visual representation of word boundaries. 3.9 Formatting: From Lists to Strings Often we write a program to report a single data item, such as a particular element in a corpus that meets some complicated criterion, or a single summary statistic such as a word-count or the performance of a tagger. More often, we write a program to produce a structured result; for example, a tabulation of numbers or linguistic forms, or a re- formatting of the original data. When the results to be presented are linguistic, textual output is usually the most natural choice. However, when the results are numerical, it may be preferable to produce graphical output. In this section, you will learn about a variety of ways to present program output. From Lists to Strings The simplest kind of structured object we use for text processing is lists of words. When we want to output these to a display or a file, we must convert these lists into strings. To do this in Python we use the join() method, and specify the string to be used as the “glue”: >>> silly = ['We', 'called', 'him', 'Tortoise', 'because', 'he', 'taught', 'us', '.'] >>> ' '.join(silly) 'We called him Tortoise because he taught us .' >>> ';'.join(silly) 'We;called;him;Tortoise;because;he;taught;us;.' >>> ''.join(silly) 'WecalledhimTortoisebecausehetaughtus.' So ' '.join(silly) means: take all the items in silly and concatenate them as one big string, using ' ' as a spacer between the items. I.e., join() is a method of the string that you want to use as the glue. (Many people find this notation for join() counter- intuitive.) The join() method only works on a list of strings—what we have been calling a text—a complex type that enjoys some privileges in Python. 116 | Chapter 3: Processing Raw Text Strings and Formats We have seen that there are two ways to display the contents of an object: >>> word = 'cat' >>> sentence = """hello ... world""" >>> print word cat >>> print sentence hello world >>> word 'cat' >>> sentence 'hello\nworld' The print command yields Python’s attempt to produce the most human-readable form of an object. The second method—naming the variable at a prompt—shows us a string that can be used to recreate this object. It is important to keep in mind that both of these are just strings, displayed for the benefit of you, the user. They do not give us any clue as to the actual internal representation of the object. There are many other useful ways to display an object as a string of characters. This may be for the benefit of a human reader, or because we want to export our data to a particular file format for use in an external program. Formatted output typically contains a combination of variables and pre-specified strings. For example, given a frequency distribution fdist, we could do: >>> fdist = nltk.FreqDist(['dog', 'cat', 'dog', 'cat', 'dog', 'snake', 'dog', 'cat']) >>> for word in fdist: ... print word, '->', fdist[word], ';', dog -> 4 ; cat -> 3 ; snake -> 1 ; Apart from the problem of unwanted whitespace, print statements that contain alter- nating variables and constants can be difficult to read and maintain. A better solution is to use string formatting expressions. >>> for word in fdist: ... print '%s->%d;' % (word, fdist[word]), dog->4; cat->3; snake->1; To understand what is going on here, let’s test out the string formatting expression on its own. (By now this will be your usual method of exploring new syntax.) >>> '%s->%d;' % ('cat', 3) 'cat->3;' >>> '%s->%d;' % 'cat' Traceback (most recent call last): File "", line 1, in TypeError: not enough arguments for format string 3.9 Formatting: From Lists to Strings | 117 The special symbols %s and %d are placeholders for strings and (decimal) integers. We can embed these inside a string, then use the % operator to combine them. Let’s unpack this code further, in order to see this behavior up close: >>> '%s->' % 'cat' 'cat->' >>> '%d' % 3 '3' >>> 'I want a %s right now' % 'coffee' 'I want a coffee right now' We can have a number of placeholders, but following the % operator we need to specify a tuple with exactly the same number of values: >>> "%s wants a %s %s" % ("Lee", "sandwich", "for lunch") 'Lee wants a sandwich for lunch' We can also provide the values for the placeholders indirectly. Here’s an example using a for loop: >>> template = 'Lee wants a %s right now' >>> menu = ['sandwich', 'spam fritter', 'pancake'] >>> for snack in menu: ... print template % snack ... Lee wants a sandwich right now Lee wants a spam fritter right now Lee wants a pancake right now The %s and %d symbols are called conversion specifiers. They start with the % character and end with a conversion character such as s (for string) or d (for decimal integer) The string containing conversion specifiers is called a format string. We combine a format string with the % operator and a tuple of values to create a complete string formatting expression. Lining Things Up So far our formatting strings generated output of arbitrary width on the page (or screen), such as %s and %d. We can specify a width as well, such as %6s, producing a string that is padded to width 6. It is right-justified by default , but we can include a minus sign to make it left-justified . In case we don’t know in advance how wide a displayed value should be, the width value can be replaced with a star in the formatting string, then specified using a variable . >>> '%6s' % 'dog' ' dog' >>> '%-6s' % 'dog' 'dog ' >>> width = 6 >>> '%-*s' % (width, 'dog') 'dog ' 118 | Chapter 3: Processing Raw Text Other control characters are used for decimal integers and floating-point numbers. Since the percent character % has a special interpretation in formatting strings, we have to precede it with another % to get it in the output. >>> count, total = 3205, 9375 >>> "accuracy for %d words: %2.4f%%" % (total, 100 * count / total) 'accuracy for 9375 words: 34.1867%' An important use of formatting strings is for tabulating data. Recall that in Sec- tion 2.1 we saw data being tabulated from a conditional frequency distribution. Let’s perform the tabulation ourselves, exercising full control of headings and column widths, as shown in Example 3-5. Note the clear separation between the language processing work, and the tabulation of results. Example 3-5. Frequency of modals in different sections of the Brown Corpus. def tabulate(cfdist, words, categories): print '%-16s' % 'Category', for word in words: # column headings print '%6s' % word, print for category in categories: print '%-16s' % category, # row heading for word in words: # for each word print '%6d' % cfdist[category][word], # print table cell print # end the row >>> from nltk.corpus import brown >>> cfd = nltk.ConditionalFreqDist( ... (genre, word) ... for genre in brown.categories() ... for word in brown.words(categories=genre)) >>> genres = ['news', 'religion', 'hobbies', 'science_fiction', 'romance', 'humor'] >>> modals = ['can', 'could', 'may', 'might', 'must', 'will'] >>> tabulate(cfd, modals, genres) Category can could may might must will news 93 86 66 38 50 389 religion 82 59 78 12 54 71 hobbies 268 58 131 22 83 264 science_fiction 16 49 4 12 8 16 romance 74 193 11 51 45 43 humor 16 30 8 8 9 13 Recall from the listing in Example 3-1 that we used a formatting string "%*s". This allows us to specify the width of a field using a variable. >>> '%*s' % (15, "Monty Python") ' Monty Python' We could use this to automatically customize the column to be just wide enough to accommodate all the words, using width = max(len(w) for w in words). Remember that the comma at the end of print statements adds an extra space, and this is sufficient to prevent the column headings from running into each other. 3.9 Formatting: From Lists to Strings | 119 Writing Results to a File We have seen how to read text from files (Section 3.1). It is often useful to write output to files as well. The following code opens a file output.txt for writing, and saves the program output to the file. >>> output_file = open('output.txt', 'w') >>> words = set(nltk.corpus.genesis.words('english-kjv.txt')) >>> for word in sorted(words): ... output_file.write(word + "\n") Your Turn: What is the effect of appending \n to each string before we write it to the file? If you’re using a Windows machine, you may want to use word + "\r\n" instead. What happens if we do output_file.write(word) When we write non-text data to a file, we must convert it to a string first. We can do this conversion using formatting strings, as we saw earlier. Let’s write the total number of words to our file, before closing it. >>> len(words) 2789 >>> str(len(words)) '2789' >>> output_file.write(str(len(words)) + "\n") >>> output_file.close() Caution! You should avoid filenames that contain space characters, such as output file.txt, or that are identical except for case distinctions, e.g., Output.txt and output.TXT. Text Wrapping When the output of our program is text-like, instead of tabular, it will usually be nec- essary to wrap it so that it can be displayed conveniently. Consider the following output, which overflows its line, and which uses a complicated print statement: >>> saying = ['After', 'all', 'is', 'said', 'and', 'done', ',', ... 'more', 'is', 'said', 'than', 'done', '.'] >>> for word in saying: ... print word, '(' + str(len(word)) + '),', After (5), all (3), is (2), said (4), and (3), done (4), , (1), more (4), is (2), said (4), We can take care of line wrapping with the help of Python’s textwrap module. For maximum clarity we will separate each step onto its own line: >>> from textwrap import fill >>> format = '%s (%d),' 120 | Chapter 3: Processing Raw Text >>> pieces = [format % (word, len(word)) for word in saying] >>> output = ' '.join(pieces) >>> wrapped = fill(output) >>> print wrapped After (5), all (3), is (2), said (4), and (3), done (4), , (1), more (4), is (2), said (4), than (4), done (4), . (1), Notice that there is a linebreak between more and its following number. If we wanted to avoid this, we could redefine the formatting string so that it contained no spaces (e.g., '%s_(%d),'), then instead of printing the value of wrapped, we could print wrap ped.replace('_', ' '). 3.10 Summary • In this book we view a text as a list of words. A “raw text” is a potentially long string containing words and whitespace formatting, and is how we typically store and visualize a text. • A string is specified in Python using single or double quotes: 'Monty Python', "Monty Python". • The characters of a string are accessed using indexes, counting from zero: 'Monty Python'[0] gives the value M. The length of a string is found using len(). • Substrings are accessed using slice notation: 'Monty Python'[1:5] gives the value onty. If the start index is omitted, the substring begins at the start of the string; if the end index is omitted, the slice continues to the end of the string. • Strings can be split into lists: 'Monty Python'.split() gives ['Monty', 'Python']. Lists can be joined into strings: '/'.join(['Monty', 'Python']) gives 'Monty/ Python'. • We can read text from a file f using text = open(f).read(). We can read text from a URL u using text = urlopen(u).read(). We can iterate over the lines of a text file using for line in open(f). • Texts found on the Web may contain unwanted material (such as headers, footers, and markup), that need to be removed before we do any linguistic processing. • Tokenization is the segmentation of a text into basic units—or tokens—such as words and punctuation. Tokenization based on whitespace is inadequate for many applications because it bundles punctuation together with words. NLTK provides an off-the-shelf tokenizer nltk.word_tokenize(). • Lemmatization is a process that maps the various forms of a word (such as ap- peared, appears) to the canonical or citation form of the word, also known as the lexeme or lemma (e.g., appear). • Regular expressions are a powerful and flexible method of specifying patterns. Once we have imported the re module, we can use re.findall() to find all sub- strings in a string that match a pattern. 3.10 Summary | 121 • If a regular expression string includes a backslash, you should tell Python not to preprocess the string, by using a raw string with an r prefix: r'regexp'. • When backslash is used before certain characters, e.g., \n, this takes on a special meaning (newline character); however, when backslash is used before regular ex- pression wildcards and operators, e.g., \., \|, \$, these characters lose their special meaning and are matched literally. • A string formatting expression template % arg_tuple consists of a format string template that contains conversion specifiers like %-6s and %0.2d. 3.11 Further Reading Extra materials for this chapter are posted at http://www.nltk.org/, including links to freely available resources on the Web. Remember to consult the Python reference ma- terials at http://docs.python.org/. (For example, this documentation covers “universal newline support,” explaining how to work with the different newline conventions used by various operating systems.) For more examples of processing words with NLTK, see the tokenization, stemming, and corpus HOWTOs at http://www.nltk.org/howto. Chapters 2 and 3 of (Jurafsky & Martin, 2008) contain more advanced material on regular expressions and morphology. For more extensive discussion of text processing with Python, see (Mertz, 2003). For information about normalizing non-standard words, see (Sproat et al., 2001). There are many references for regular expressions, both practical and theoretical. For an introductory tutorial to using regular expressions in Python, see Kuchling’s Regular Expression HOWTO, http://www.amk.ca/python/howto/regex/. For a comprehensive and detailed manual in using regular expressions, covering their syntax in most major programming languages, including Python, see (Friedl, 2002). Other presentations in- clude Section 2.1 of (Jurafsky & Martin, 2008), and Chapter 3 of (Mertz, 2003). There are many online resources for Unicode. Useful discussions of Python’s facilities for handling Unicode are: • PEP-100 http://www.python.org/dev/peps/pep-0100/ • Jason Orendorff, Unicode for Programmers, http://www.jorendorff.com/articles/uni code/ • A. M. Kuchling, Unicode HOWTO, http://www.amk.ca/python/howto/unicode • Frederik Lundh, Python Unicode Objects, http://effbot.org/zone/unicode-objects .htm • Joel Spolsky, The Absolute Minimum Every Software Developer Absolutely, Posi- tively Must Know About Unicode and Character Sets (No Excuses!), http://www.joe lonsoftware.com/articles/Unicode.html 122 | Chapter 3: Processing Raw Text The problem of tokenizing Chinese text is a major focus of SIGHAN, the ACL Special Interest Group on Chinese Language Processing (http://sighan.org/). Our method for segmenting English text follows (Brent & Cartwright, 1995); this work falls in the area of language acquisition (Niyogi, 2006). Collocations are a special case of multiword expressions. A multiword expression is a small phrase whose meaning and other properties cannot be predicted from its words alone, e.g., part-of-speech (Baldwin & Kim, 2010). Simulated annealing is a heuristic for finding a good approximation to the optimum value of a function in a large, discrete search space, based on an analogy with annealing in metallurgy. The technique is described in many Artificial Intelligence texts. The approach to discovering hyponyms in text using search patterns like x and other ys is described by (Hearst, 1992). 3.12 Exercises 1. ○ Define a string s = 'colorless'. Write a Python statement that changes this to “colourless” using only the slice and concatenation operations. 2. ○ We can use the slice notation to remove morphological endings on words. For example, 'dogs'[:-1] removes the last character of dogs, leaving dog. Use slice notation to remove the affixes from these words (we’ve inserted a hyphen to indi- cate the affix boundary, but omit this from your strings): dish-es, run-ning, nation- ality, un-do, pre-heat. 3. ○ We saw how we can generate an IndexError by indexing beyond the end of a string. Is it possible to construct an index that goes too far to the left, before the start of the string? 4. ○ We can specify a “step” size for the slice. The following returns every second character within the slice: monty[6:11:2]. It also works in the reverse direction: monty[10:5:-2]. Try these for yourself, and then experiment with different step values. 5. ○ What happens if you ask the interpreter to evaluate monty[::-1]? Explain why this is a reasonable result. 6. ○ Describe the class of strings matched by the following regular expressions: a. [a-zA-Z]+ b. [A-Z][a-z]* c. p[aeiou]{,2}t d. \d+(\.\d+)? e. ([^aeiou][aeiou][^aeiou])* f. \w+|[^\w\s]+ Test your answers using nltk.re_show(). 3.12 Exercises | 123 7. ○ Write regular expressions to match the following classes of strings: a. A single determiner (assume that a, an, and the are the only determiners) b. An arithmetic expression using integers, addition, and multiplication, such as 2*3+8 8. ○ Write a utility function that takes a URL as its argument, and returns the contents of the URL, with all HTML markup removed. Use urllib.urlopen to access the contents of the URL, e.g.: raw_contents = urllib.urlopen('http://www.nltk.org/').read() 9. ○ Save some text into a file corpus.txt. Define a function load(f) that reads from the file named in its sole argument, and returns a string containing the text of the file. a. Use nltk.regexp_tokenize() to create a tokenizer that tokenizes the various kinds of punctuation in this text. Use one multiline regular expression inline comments, using the verbose flag (?x). b. Use nltk.regexp_tokenize() to create a tokenizer that tokenizes the following kinds of expressions: monetary amounts; dates; names of people and organizations. 10. ○ Rewrite the following loop as a list comprehension: >>> sent = ['The', 'dog', 'gave', 'John', 'the', 'newspaper'] >>> result = [] >>> for word in sent: ... word_len = (word, len(word)) ... result.append(word_len) >>> result [('The', 3), ('dog', 3), ('gave', 4), ('John', 4), ('the', 3), ('newspaper', 9)] 11. ○ Define a string raw containing a sentence of your own choosing. Now, split raw on some character other than space, such as 's'. 12. ○ Write a for loop to print out the characters of a string, one per line. 13. ○ What is the difference between calling split on a string with no argument and one with ' ' as the argument, e.g., sent.split() versus sent.split(' ')? What happens when the string being split contains tab characters, consecutive space characters, or a sequence of tabs and spaces? (In IDLE you will need to use '\t' to enter a tab character.) 14. ○ Create a variable words containing a list of words. Experiment with words.sort() and sorted(words). What is the difference? 15. ○ Explore the difference between strings and integers by typing the following at a Python prompt: "3" * 7 and 3 * 7. Try converting between strings and integers using int("3") and str(3). 16. ○ Earlier, we asked you to use a text editor to create a file called test.py, containing the single line monty = 'Monty Python'. If you haven’t already done this (or can’t find the file), go ahead and do it now. Next, start up a new session with the Python 124 | Chapter 3: Processing Raw Text interpreter, and enter the expression monty at the prompt. You will get an error from the interpreter. Now, try the following (note that you have to leave off the .py part of the filename): >>> from test import msg >>> msg This time, Python should return with a value. You can also try import test, in which case Python should be able to evaluate the expression test.monty at the prompt. 17. ○ What happens when the formatting strings %6s and %-6s are used to display strings that are longer than six characters? 18. ◑ Read in some text from a corpus, tokenize it, and print the list of all wh-word types that occur. (wh-words in English are used in questions, relative clauses, and exclamations: who, which, what, and so on.) Print them in order. Are any words duplicated in this list, because of the presence of case distinctions or punctuation? 19. ◑ Create a file consisting of words and (made up) frequencies, where each line consists of a word, the space character, and a positive integer, e.g., fuzzy 53. Read the file into a Python list using open(filename).readlines(). Next, break each line into its two fields using split(), and convert the number into an integer using int(). The result should be a list of the form: [['fuzzy', 53], ...]. 20. ◑ Write code to access a favorite web page and extract some text from it. For example, access a weather site and extract the forecast top temperature for your town or city today. 21. ◑ Write a function unknown() that takes a URL as its argument, and returns a list of unknown words that occur on that web page. In order to do this, extract all substrings consisting of lowercase letters (using re.findall()) and remove any items from this set that occur in the Words Corpus (nltk.corpus.words). Try to categorize these words manually and discuss your findings. 22. ◑ Examine the results of processing the URL http://news.bbc.co.uk/ using the reg- ular expressions suggested above. You will see that there is still a fair amount of non-textual data there, particularly JavaScript commands. You may also find that sentence breaks have not been properly preserved. Define further regular expres- sions that improve the extraction of text from this web page. 23. ◑ Are you able to write a regular expression to tokenize text in such a way that the word don’t is tokenized into do and n’t? Explain why this regular expression won’t work: «n't|\w+». 24. ◑ Try to write code to convert text into hAck3r, using regular expressions and substitution, where e → 3, i → 1, o → 0, l → |, s → 5, . → 5w33t!, ate → 8. Normalize the text to lowercase before converting it. Add more substitutions of your own. Now try to map s to two different values: $ for word-initial s, and 5 for word- internal s. 3.12 Exercises | 125 25. ◑ Pig Latin is a simple transformation of English text. Each word of the text is converted as follows: move any consonant (or consonant cluster) that appears at the start of the word to the end, then append ay, e.g., string → ingstray, idle → idleay (see http://en.wikipedia.org/wiki/Pig_Latin). a. Write a function to convert a word to Pig Latin. b. Write code that converts text, instead of individual words. c. Extend it further to preserve capitalization, to keep qu together (so that quiet becomes ietquay, for example), and to detect when y is used as a con- sonant (e.g., yellow) versus a vowel (e.g., style). 26. ◑ Download some text from a language that has vowel harmony (e.g., Hungarian), extract the vowel sequences of words, and create a vowel bigram table. 27. ◑ Python’s random module includes a function choice() which randomly chooses an item from a sequence; e.g., choice("aehh ") will produce one of four possible characters, with the letter h being twice as frequent as the others. Write a generator expression that produces a sequence of 500 randomly chosen letters drawn from the string "aehh ", and put this expression inside a call to the ''.join() function, to concatenate them into one long string. You should get a result that looks like uncontrolled sneezing or maniacal laughter: he haha ee heheeh eha. Use split() and join() again to normalize the whitespace in this string. 28. ◑ Consider the numeric expressions in the following sentence from the MedLine Corpus: The corresponding free cortisol fractions in these sera were 4.53 +/- 0.15% and 8.16 +/- 0.23%, respectively. Should we say that the numeric expression 4.53 +/- 0.15% is three words? Or should we say that it’s a single compound word? Or should we say that it is actually nine words, since it’s read “four point five three, plus or minus fifteen percent”? Or should we say that it’s not a “real” word at all, since it wouldn’t appear in any dictionary? Discuss these different possibilities. Can you think of application domains that motivate at least two of these answers? 29. ◑ Readability measures are used to score the reading difficulty of a text, for the purposes of selecting texts of appropriate difficulty for language learners. Let us define μw to be the average number of letters per word, and μs to be the average number of words per sentence, in a given text. The Automated Readability Index (ARI) of the text is defined to be: 4.71 μw + 0.5 μs - 21.43. Compute the ARI score for various sections of the Brown Corpus, including section f (popular lore) and j (learned). Make use of the fact that nltk.corpus.brown.words() produces a se- quence of words, whereas nltk.corpus.brown.sents() produces a sequence of sentences. 30. ◑ Use the Porter Stemmer to normalize some tokenized text, calling the stemmer on each word. Do the same thing with the Lancaster Stemmer, and see if you ob- serve any differences. 31. ◑ Define the variable saying to contain the list ['After', 'all', 'is', 'said', 'and', 'done', ',', 'more', 'is', 'said', 'than', 'done', '.']. Process the list 126 | Chapter 3: Processing Raw Text using a for loop, and store the result in a new list lengths. Hint: begin by assigning the empty list to lengths, using lengths = []. Then each time through the loop, use append() to add another length value to the list. 32. ◑ Define a variable silly to contain the string: 'newly formed bland ideas are inexpressible in an infuriating way'. (This happens to be the legitimate inter- pretation that bilingual English-Spanish speakers can assign to Chomsky’s famous nonsense phrase colorless green ideas sleep furiously, according to Wikipedia). Now write code to perform the following tasks: a. Split silly into a list of strings, one per word, using Python’s split() opera- tion, and save this to a variable called bland. b. Extract the second letter of each word in silly and join them into a string, to get 'eoldrnnnna'. c. Combine the words in bland back into a single string, using join(). Make sure the words in the resulting string are separated with whitespace. d. Print the words of silly in alphabetical order, one per line. 33. ◑ The index() function can be used to look up items in sequences. For example, 'inexpressible'.index('e') tells us the index of the first position of the letter e. a. What happens when you look up a substring, e.g., 'inexpressi ble'.index('re')? b. Define a variable words containing a list of words. Now use words.index() to look up the position of an individual word. c. Define a variable silly as in Exercise 32. Use the index() function in combi- nation with list slicing to build a list phrase consisting of all the words up to (but not including) in in silly. 34. ◑ Write code to convert nationality adjectives such as Canadian and Australian to their corresponding nouns Canada and Australia (see http://en.wikipedia.org/wiki/ List_of_adjectival_forms_of_place_names). 35. ◑ Read the LanguageLog post on phrases of the form as best as p can and as best p can, where p is a pronoun. Investigate this phenomenon with the help of a corpus and the findall() method for searching tokenized text described in Section 3.5. The post is at http://itre.cis.upenn.edu/~myl/languagelog/archives/002733.html. 36. ◑ Study the lolcat version of the book of Genesis, accessible as nltk.corpus.gene sis.words('lolcat.txt'), and the rules for converting text into lolspeak at http:// www.lolcatbible.com/index.php?title=How_to_speak_lolcat. Define regular expres- sions to convert English words into corresponding lolspeak words. 37. ◑ Read about the re.sub() function for string substitution using regular expres- sions, using help(re.sub) and by consulting the further readings for this chapter. Use re.sub in writing code to remove HTML tags from an HTML file, and to normalize whitespace. 3.12 Exercises | 127 38. ● An interesting challenge for tokenization is words that have been split across a linebreak. E.g., if long-term is split, then we have the string long-\nterm. a. Write a regular expression that identifies words that are hyphenated at a line- break. The expression will need to include the \n character. b. Use re.sub() to remove the \n character from these words. c. How might you identify words that should not remain hyphenated once the newline is removed, e.g., 'encyclo-\npedia'? 39. ● Read the Wikipedia entry on Soundex. Implement this algorithm in Python. 40. ● Obtain raw texts from two or more genres and compute their respective reading difficulty scores as in the earlier exercise on reading difficulty. E.g., compare ABC Rural News and ABC Science News (nltk.corpus.abc). Use Punkt to perform sen- tence segmentation. 41. ● Rewrite the following nested loop as a nested list comprehension: >>> words = ['attribution', 'confabulation', 'elocution', ... 'sequoia', 'tenacious', 'unidirectional'] >>> vsequences = set() >>> for word in words: ... vowels = [] ... for char in word: ... if char in 'aeiou': ... vowels.append(char) ... vsequences.add(''.join(vowels)) >>> sorted(vsequences) ['aiuio', 'eaiou', 'eouio', 'euoia', 'oauaio', 'uiieioa'] 42. ● Use WordNet to create a semantic index for a text collection. Extend the con- cordance search program in Example 3-1, indexing each word using the offset of its first synset, e.g., wn.synsets('dog')[0].offset (and optionally the offset of some of its ancestors in the hypernym hierarchy). 43. ● With the help of a multilingual corpus such as the Universal Declaration of Human Rights Corpus (nltk.corpus.udhr), along with NLTK’s frequency distri- bution and rank correlation functionality (nltk.FreqDist, nltk.spearman_correla tion), develop a system that guesses the language of a previously unseen text. For simplicity, work with a single character encoding and just a few languages. 44. ● Write a program that processes a text and discovers cases where a word has been used with a novel sense. For each word, compute the WordNet similarity between all synsets of the word and all synsets of the words in its context. (Note that this is a crude approach; doing it well is a difficult, open research problem.) 45. ● Read the article on normalization of non-standard words (Sproat et al., 2001), and implement a similar system for text normalization. 128 | Chapter 3: Processing Raw Text CHAPTER 4 Writing Structured Programs By now you will have a sense of the capabilities of the Python programming language for processing natural language. However, if you’re new to Python or to programming, you may still be wrestling with Python and not feel like you are in full control yet. In this chapter we’ll address the following questions: 1. How can you write well-structured, readable programs that you and others will be able to reuse easily? 2. How do the fundamental building blocks work, such as loops, functions, and assignment? 3. What are some of the pitfalls with Python programming, and how can you avoid them? Along the way, you will consolidate your knowledge of fundamental programming constructs, learn more about using features of the Python language in a natural and concise way, and learn some useful techniques in visualizing natural language data. As before, this chapter contains many examples and exercises (and as before, some exer- cises introduce new material). Readers new to programming should work through them carefully and consult other introductions to programming if necessary; experienced programmers can quickly skim this chapter. In the other chapters of this book, we have organized the programming concepts as dictated by the needs of NLP. Here we revert to a more conventional approach, where the material is more closely tied to the structure of the programming language. There’s not room for a complete presentation of the language, so we’ll just focus on the language constructs and idioms that are most important for NLP. 129 4.1 Back to the Basics Assignment Assignment would seem to be the most elementary programming concept, not deserv- ing a separate discussion. However, there are some surprising subtleties here. Consider the following code fragment: >>> foo = 'Monty' >>> bar = foo >>> foo = 'Python' >>> bar 'Monty' This behaves exactly as expected. When we write bar = foo in the code , the value of foo (the string 'Monty') is assigned to bar. That is, bar is a copy of foo, so when we overwrite foo with a new string 'Python' on line , the value of bar is not affected. However, assignment statements do not always involve making copies in this way. Assignment always copies the value of an expression, but a value is not always what you might expect it to be. In particular, the “value” of a structured object such as a list is actually just a reference to the object. In the following example, assigns the refer- ence of foo to the new variable bar. Now when we modify something inside foo on line , we can see that the contents of bar have also been changed. >>> foo = ['Monty', 'Python'] >>> bar = foo >>> foo[1] = 'Bodkin' >>> bar ['Monty', 'Bodkin'] The line bar = foo does not copy the contents of the variable, only its “object refer- ence.” To understand what is going on here, we need to know how lists are stored in the computer’s memory. In Figure 4-1, we see that a list foo is a reference to an object stored at location 3133 (which is itself a series of pointers to other locations holding strings). When we assign bar = foo, it is just the object reference 3133 that gets copied. This behavior extends to other aspects of the language, such as parameter passing (Section 4.4). 130 | Chapter 4: Writing Structured Programs Let’s experiment some more, by creating a variable empty holding the empty list, then using it three times on the next line. >>> empty = [] >>> nested = [empty, empty, empty] >>> nested [[], [], []] >>> nested[1].append('Python') >>> nested [['Python'], ['Python'], ['Python']] Observe that changing one of the items inside our nested list of lists changed them all. This is because each of the three elements is actually just a reference to one and the same list in memory. Your Turn: Use multiplication to create a list of lists: nested = [[]] * 3. Now modify one of the elements of the list, and observe that all the elements are changed. Use Python’s id() function to find out the nu- merical identifier for any object, and verify that id(nested[0]), id(nested[1]), and id(nested[2]) are all the same. Now, notice that when we assign a new value to one of the elements of the list, it does not propagate to the others: >>> nested = [[]] * 3 >>> nested[1].append('Python') >>> nested[1] = ['Monty'] >>> nested [['Python'], ['Monty'], ['Python']] We began with a list containing three references to a single empty list object. Then we modified that object by appending 'Python' to it, resulting in a list containing three references to a single list object ['Python']. Next, we overwrote one of those references with a reference to a new object ['Monty']. This last step modified one of the three object references inside the nested list. However, the ['Python'] object wasn’t changed, Figure 4-1. List assignment and computer memory: Two list objects foo and bar reference the same location in the computer’s memory; updating foo will also modify bar, and vice versa. 4.1 Back to the Basics | 131 and is still referenced from two places in our nested list of lists. It is crucial to appreciate this difference between modifying an object via an object reference and overwriting an object reference. Important: To copy the items from a list foo to a new list bar, you can write bar = foo[:]. This copies the object references inside the list. To copy a structure without copying any object references, use copy.deep copy(). Equality Python provides two ways to check that a pair of items are the same. The is operator tests for object identity. We can use it to verify our earlier observations about objects. First, we create a list containing several copies of the same object, and demonstrate that they are not only identical according to ==, but also that they are one and the same object: >>> size = 5 >>> python = ['Python'] >>> snake_nest = [python] * size >>> snake_nest[0] == snake_nest[1] == snake_nest[2] == snake_nest[3] == snake_nest[4] True >>> snake_nest[0] is snake_nest[1] is snake_nest[2] is snake_nest[3] is snake_nest[4] True Now let’s put a new python in this nest. We can easily show that the objects are not all identical: >>> import random >>> position = random.choice(range(size)) >>> snake_nest[position] = ['Python'] >>> snake_nest [['Python'], ['Python'], ['Python'], ['Python'], ['Python']] >>> snake_nest[0] == snake_nest[1] == snake_nest[2] == snake_nest[3] == snake_nest[4] True >>> snake_nest[0] is snake_nest[1] is snake_nest[2] is snake_nest[3] is snake_nest[4] False You can do several pairwise tests to discover which position contains the interloper, but the id() function makes detection is easier: >>> [id(snake) for snake in snake_nest] [513528, 533168, 513528, 513528, 513528] This reveals that the second item of the list has a distinct identifier. If you try running this code snippet yourself, expect to see different numbers in the resulting list, and don’t be surprised if the interloper is in a different position. Having two kinds of equality might seem strange. However, it’s really just the type- token distinction, familiar from natural language, here showing up in a programming language. 132 | Chapter 4: Writing Structured Programs Conditionals In the condition part of an if statement, a non-empty string or list is evaluated as true, while an empty string or list evaluates as false. >>> mixed = ['cat', '', ['dog'], []] >>> for element in mixed: ... if element: ... print element ... cat ['dog'] That is, we don’t need to say if len(element) > 0: in the condition. What’s the difference between using if...elif as opposed to using a couple of if statements in a row? Well, consider the following situation: >>> animals = ['cat', 'dog'] >>> if 'cat' in animals: ... print 1 ... elif 'dog' in animals: ... print 2 ... 1 Since the if clause of the statement is satisfied, Python never tries to evaluate the elif clause, so we never get to print out 2. By contrast, if we replaced the elif by an if, then we would print out both 1 and 2. So an elif clause potentially gives us more information than a bare if clause; when it evaluates to true, it tells us not only that the condition is satisfied, but also that the condition of the main if clause was not satisfied. The functions all() and any() can be applied to a list (or other sequence) to check whether all or any items meet some condition: >>> sent = ['No', 'good', 'fish', 'goes', 'anywhere', 'without', 'a', 'porpoise', '.'] >>> all(len(w) > 4 for w in sent) False >>> any(len(w) > 4 for w in sent) True 4.2 Sequences So far, we have seen two kinds of sequence object: strings and lists. Another kind of sequence is called a tuple. Tuples are formed with the comma operator , and typically enclosed using parentheses. We’ve actually seen them in the previous chapters, and sometimes referred to them as “pairs,” since there were always two members. However, tuples can have any number of members. Like lists and strings, tuples can be indexed and sliced , and have a length . >>> t = 'walk', 'fem', 3 >>> t ('walk', 'fem', 3) 4.2 Sequences | 133 >>> t[0] 'walk' >>> t[1:] ('fem', 3) >>> len(t) Caution! Tuples are constructed using the comma operator. Parentheses are a more general feature of Python syntax, designed for grouping. A tuple containing the single element 'snark' is defined by adding a trailing comma, like this: 'snark',. The empty tuple is a special case, and is defined using empty parentheses (). Let’s compare strings, lists, and tuples directly, and do the indexing, slice, and length operation on each type: >>> raw = 'I turned off the spectroroute' >>> text = ['I', 'turned', 'off', 'the', 'spectroroute'] >>> pair = (6, 'turned') >>> raw[2], text[3], pair[1] ('t', 'the', 'turned') >>> raw[-3:], text[-3:], pair[-3:] ('ute', ['off', 'the', 'spectroroute'], (6, 'turned')) >>> len(raw), len(text), len(pair) (29, 5, 2) Notice in this code sample that we computed multiple values on a single line, separated by commas. These comma-separated expressions are actually just tuples—Python al- lows us to omit the parentheses around tuples if there is no ambiguity. When we print a tuple, the parentheses are always displayed. By using tuples in this way, we are im- plicitly aggregating items together. Your Turn: Define a set, e.g., using set(text), and see what happens when you convert it to a list or iterate over its members. Operating on Sequence Types We can iterate over the items in a sequence s in a variety of useful ways, as shown in Table 4-1. Table 4-1. Various ways to iterate over sequences Python expression Comment for item in s Iterate over the items of s for item in sorted(s) Iterate over the items of s in order for item in set(s) Iterate over unique elements of s 134 | Chapter 4: Writing Structured Programs Python expression Comment for item in reversed(s) Iterate over elements of s in reverse for item in set(s).difference(t) Iterate over elements of s not in t for item in random.shuffle(s) Iterate over elements of s in random order The sequence functions illustrated in Table 4-1 can be combined in various ways; for example, to get unique elements of s sorted in reverse, use reversed(sorted(set(s))). We can convert between these sequence types. For example, tuple(s) converts any kind of sequence into a tuple, and list(s) converts any kind of sequence into a list. We can convert a list of strings to a single string using the join() function, e.g., ':'.join(words). Some other objects, such as a FreqDist, can be converted into a sequence (using list()) and support iteration: >>> raw = 'Red lorry, yellow lorry, red lorry, yellow lorry.' >>> text = nltk.word_tokenize(raw) >>> fdist = nltk.FreqDist(text) >>> list(fdist) ['lorry', ',', 'yellow', '.', 'Red', 'red'] >>> for key in fdist: ... print fdist[key], ... 4 3 2 1 1 1 In the next example, we use tuples to re-arrange the contents of our list. (We can omit the parentheses because the comma has higher precedence than assignment.) >>> words = ['I', 'turned', 'off', 'the', 'spectroroute'] >>> words[2], words[3], words[4] = words[3], words[4], words[2] >>> words ['I', 'turned', 'the', 'spectroroute', 'off'] This is an idiomatic and readable way to move items inside a list. It is equivalent to the following traditional way of doing such tasks that does not use tuples (notice that this method needs a temporary variable tmp). >>> tmp = words[2] >>> words[2] = words[3] >>> words[3] = words[4] >>> words[4] = tmp As we have seen, Python has sequence functions such as sorted() and reversed() that rearrange the items of a sequence. There are also functions that modify the structure of a sequence, which can be handy for language processing. Thus, zip() takes the items of two or more sequences and “zips” them together into a single list of pairs. Given a sequence s, enumerate(s) returns pairs consisting of an index and the item at that index. >>> words = ['I', 'turned', 'off', 'the', 'spectroroute'] >>> tags = ['noun', 'verb', 'prep', 'det', 'noun'] >>> zip(words, tags) 4.2 Sequences | 135 [('I', 'noun'), ('turned', 'verb'), ('off', 'prep'), ('the', 'det'), ('spectroroute', 'noun')] >>> list(enumerate(words)) [(0, 'I'), (1, 'turned'), (2, 'off'), (3, 'the'), (4, 'spectroroute')] For some NLP tasks it is necessary to cut up a sequence into two or more parts. For instance, we might want to “train” a system on 90% of the data and test it on the remaining 10%. To do this we decide the location where we want to cut the data , then cut the sequence at that location . >>> text = nltk.corpus.nps_chat.words() >>> cut = int(0.9 * len(text)) >>> training_data, test_data = text[:cut], text[cut:] >>> text == training_data + test_data True >>> len(training_data) / len(test_data) 9 We can verify that none of the original data is lost during this process, nor is it dupli- cated . We can also verify that the ratio of the sizes of the two pieces is what we intended . Combining Different Sequence Types Let’s combine our knowledge of these three sequence types, together with list com- prehensions, to perform the task of sorting the words in a string by their length. >>> words = 'I turned off the spectroroute'.split() >>> wordlens = [(len(word), word) for word in words] >>> wordlens.sort() >>> ' '.join(w for (_, w) in wordlens) 'I off the turned spectroroute' Each of the preceding lines of code contains a significant feature. A simple string is actually an object with methods defined on it, such as split() . We use a list com- prehension to build a list of tuples , where each tuple consists of a number (the word length) and the word, e.g., (3, 'the'). We use the sort() method to sort the list in place. Finally, we discard the length information and join the words back into a single string . (The underscore is just a regular Python variable, but we can use underscore by convention to indicate that we will not use its value.) We began by talking about the commonalities in these sequence types, but the previous code illustrates important differences in their roles. First, strings appear at the beginning and the end: this is typical in the context where our program is reading in some text and producing output for us to read. Lists and tuples are used in the middle, but for different purposes. A list is typically a sequence of objects all having the same type, of arbitrary length. We often use lists to hold sequences of words. In contrast, a tuple is typically a collection of objects of different types, of fixed length. We often use a tuple to hold a record, a collection of different fields relating to some entity. This distinction between the use of lists and tuples takes some getting used to, so here is another example: 136 | Chapter 4: Writing Structured Programs >>> lexicon = [ ... ('the', 'det', ['Di:', 'D@']), ... ('off', 'prep', ['Qf', 'O:f']) ... ] Here, a lexicon is represented as a list because it is a collection of objects of a single type—lexical entries—of no predetermined length. An individual entry is represented as a tuple because it is a collection of objects with different interpretations, such as the orthographic form, the part-of-speech, and the pronunciations (represented in the SAMPA computer-readable phonetic alphabet; see http://www.phon.ucl.ac.uk/home/ sampa/). Note that these pronunciations are stored using a list. (Why?) A good way to decide when to use tuples versus lists is to ask whether the interpretation of an item depends on its position. For example, a tagged token combines two strings having different interpretations, and we choose to interpret the first item as the token and the second item as the tag. Thus we use tuples like this: ('grail', 'noun'). A tuple of the form ('noun', 'grail') would be non-sensical since it would be a word noun tagged grail. In contrast, the elements of a text are all tokens, and position is not significant. Thus we use lists like this: ['venetian', 'blind']. A list of the form ['blind', 'venetian'] would be equally valid. The linguistic meaning of the words might be different, but the interpretation of list items as tokens is unchanged. The distinction between lists and tuples has been described in terms of usage. However, there is a more fundamental difference: in Python, lists are mutable, whereas tuples are immutable. In other words, lists can be modified, whereas tuples cannot. Here are some of the operations on lists that do in-place modification of the list: >>> lexicon.sort() >>> lexicon[1] = ('turned', 'VBD', ['t3:nd', 't3`nd']) >>> del lexicon[0] Your Turn: Convert lexicon to a tuple, using lexicon = tuple(lexicon), then try each of the operations, to confirm that none of them is permitted on tuples. Generator Expressions We’ve been making heavy use of list comprehensions, for compact and readable pro- cessing of texts. Here’s an example where we tokenize and normalize a text: >>> text = '''"When I use a word," Humpty Dumpty said in rather a scornful tone, ... "it means just what I choose it to mean - neither more nor less."''' >>> [w.lower() for w in nltk.word_tokenize(text)] ['"', 'when', 'i', 'use', 'a', 'word', ',', '"', 'humpty', 'dumpty', 'said', ...] 4.2 Sequences | 137 Suppose we now want to process these words further. We can do this by inserting the preceding expression inside a call to some other function , but Python allows us to omit the brackets . >>> max([w.lower() for w in nltk.word_tokenize(text)]) 'word' >>> max(w.lower() for w in nltk.word_tokenize(text)) 'word' The second line uses a generator expression. This is more than a notational conven- ience: in many language processing situations, generator expressions will be more ef- ficient. In , storage for the list object must be allocated before the value of max() is computed. If the text is very large, this could be slow. In , the data is streamed to the calling function. Since the calling function simply has to find the maximum value—the word that comes latest in lexicographic sort order—it can process the stream of data without having to store anything more than the maximum value seen so far. 4.3 Questions of Style Programming is as much an art as a science. The undisputed “bible” of programming, a 2,500 page multivolume work by Donald Knuth, is called The Art of Computer Pro- gramming. Many books have been written on Literate Programming, recognizing that humans, not just computers, must read and understand programs. Here we pick up on some issues of programming style that have important ramifications for the readability of your code, including code layout, procedural versus declarative style, and the use of loop variables. Python Coding Style When writing programs you make many subtle choices about names, spacing, com- ments, and so on. When you look at code written by other people, needless differences in style make it harder to interpret the code. Therefore, the designers of the Python language have published a style guide for Python code, available at http://www.python .org/dev/peps/pep-0008/. The underlying value presented in the style guide is consis- tency, for the purpose of maximizing the readability of code. We briefly review some of its key recommendations here, and refer readers to the full guide for detailed dis- cussion with examples. Code layout should use four spaces per indentation level. You should make sure that when you write Python code in a file, you avoid tabs for indentation, since these can be misinterpreted by different text editors and the indentation can be messed up. Lines should be less than 80 characters long; if necessary, you can break a line inside paren- theses, brackets, or braces, because Python is able to detect that the line continues over to the next line, as in the following examples: >>> cv_word_pairs = [(cv, w) for w in rotokas_words ... for cv in re.findall('[ptksvr][aeiou]', w)] 138 | Chapter 4: Writing Structured Programs >>> cfd = nltk.ConditionalFreqDist( ... (genre, word) ... for genre in brown.categories() ... for word in brown.words(categories=genre)) >>> ha_words = ['aaahhhh', 'ah', 'ahah', 'ahahah', 'ahh', 'ahhahahaha', ... 'ahhh', 'ahhhh', 'ahhhhhh', 'ahhhhhhhhhhhhhh', 'ha', ... 'haaa', 'hah', 'haha', 'hahaaa', 'hahah', 'hahaha'] If you need to break a line outside parentheses, brackets, or braces, you can often add extra parentheses, and you can always add a backslash at the end of the line that is broken: >>> if (len(syllables) > 4 and len(syllables[2]) == 3 and ... syllables[2][2] in [aeiou] and syllables[2][3] == syllables[1][3]): ... process(syllables) >>> if len(syllables) > 4 and len(syllables[2]) == 3 and \ ... syllables[2][2] in [aeiou] and syllables[2][3] == syllables[1][3]: ... process(syllables) Typing spaces instead of tabs soon becomes a chore. Many program- ming editors have built-in support for Python, and can automatically indent code and highlight any syntax errors (including indentation er- rors). For a list of Python-aware editors, please see http://wiki.python .org/moin/PythonEditors. Procedural Versus Declarative Style We have just seen how the same task can be performed in different ways, with impli- cations for efficiency. Another factor influencing program development is programming style. Consider the following program to compute the average length of words in the Brown Corpus: >>> tokens = nltk.corpus.brown.words(categories='news') >>> count = 0 >>> total = 0 >>> for token in tokens: ... count += 1 ... total += len(token) >>> print total / count 4.2765382469 In this program we use the variable count to keep track of the number of tokens seen, and total to store the combined length of all words. This is a low-level style, not far removed from machine code, the primitive operations performed by the computer’s CPU. The two variables are just like a CPU’s registers, accumulating values at many intermediate stages, values that are meaningless until the end. We say that this program is written in a procedural style, dictating the machine operations step by step. Now consider the following program that computes the same thing: 4.3 Questions of Style | 139 >>> total = sum(len(t) for t in tokens) >>> print total / len(tokens) 4.2765382469 The first line uses a generator expression to sum the token lengths, while the second line computes the average as before. Each line of code performs a complete, meaningful task, which can be understood in terms of high-level properties like: “total is the sum of the lengths of the tokens.” Implementation details are left to the Python interpreter. The second program uses a built-in function, and constitutes programming at a more abstract level; the resulting code is more declarative. Let’s look at an extreme example: >>> word_list = [] >>> len_word_list = len(word_list) >>> i = 0 >>> while i < len(tokens): ... j = 0 ... while j < len_word_list and word_list[j] < tokens[i]: ... j += 1 ... if j == 0 or tokens[i] != word_list[j]: ... word_list.insert(j, tokens[i]) ... len_word_list += 1 ... i += 1 The equivalent declarative version uses familiar built-in functions, and its purpose is instantly recognizable: >>> word_list = sorted(set(tokens)) Another case where a loop counter seems to be necessary is for printing a counter with each line of output. Instead, we can use enumerate(), which processes a sequence s and produces a tuple of the form (i, s[i]) for each item in s, starting with (0, s[0]). Here we enumerate the keys of the frequency distribution, and capture the integer-string pair in the variables rank and word. We print rank+1 so that the counting appears to start from 1, as required when producing a list of ranked items. >>> fd = nltk.FreqDist(nltk.corpus.brown.words()) >>> cumulative = 0.0 >>> for rank, word in enumerate(fd): ... cumulative += fd[word] * 100 / fd.N() ... print "%3d %6.2f%% %s" % (rank+1, cumulative, word) ... if cumulative > 25: ... break ... 1 5.40% the 2 10.42% , 3 14.67% . 4 17.78% of 5 20.19% and 6 22.40% to 7 24.29% a 8 25.97% in It’s sometimes tempting to use loop variables to store a maximum or minimum value seen so far. Let’s use this method to find the longest word in a text. 140 | Chapter 4: Writing Structured Programs >>> text = nltk.corpus.gutenberg.words('milton-paradise.txt') >>> longest = '' >>> for word in text: ... if len(word) > len(longest): ... longest = word >>> longest 'unextinguishable' However, a more transparent solution uses two list comprehensions, both having forms that should be familiar by now: >>> maxlen = max(len(word) for word in text) >>> [word for word in text if len(word) == maxlen] ['unextinguishable', 'transubstantiate', 'inextinguishable', 'incomprehensible'] Note that our first solution found the first word having the longest length, while the second solution found all of the longest words (which is usually what we would want). Although there’s a theoretical efficiency difference between the two solutions, the main overhead is reading the data into main memory; once it’s there, a second pass through the data is effectively instantaneous. We also need to balance our concerns about pro- gram efficiency with programmer efficiency. A fast but cryptic solution will be harder to understand and maintain. Some Legitimate Uses for Counters There are cases where we still want to use loop variables in a list comprehension. For example, we need to use a loop variable to extract successive overlapping n-grams from a list: >>> sent = ['The', 'dog', 'gave', 'John', 'the', 'newspaper'] >>> n = 3 >>> [sent[i:i+n] for i in range(len(sent)-n+1)] [['The', 'dog', 'gave'], ['dog', 'gave', 'John'], ['gave', 'John', 'the'], ['John', 'the', 'newspaper']] It is quite tricky to get the range of the loop variable right. Since this is a common operation in NLP, NLTK supports it with functions bigrams(text) and trigrams(text), and a general-purpose ngrams(text, n). Here’s an example of how we can use loop variables in building multidimensional structures. For example, to build an array with m rows and n columns, where each cell is a set, we could use a nested list comprehension: >>> m, n = 3, 7 >>> array = [[set() for i in range(n)] for j in range(m)] >>> array[2][5].add('Alice') >>> pprint.pprint(array) [[set([]), set([]), set([]), set([]), set([]), set([]), set([])], [set([]), set([]), set([]), set([]), set([]), set([]), set([])], [set([]), set([]), set([]), set([]), set([]), set(['Alice']), set([])]] 4.3 Questions of Style | 141 Observe that the loop variables i and j are not used anywhere in the resulting object; they are just needed for a syntactically correct for statement. As another example of this usage, observe that the expression ['very' for i in range(3)] produces a list containing three instances of 'very', with no integers in sight. Note that it would be incorrect to do this work using multiplication, for reasons con- cerning object copying that were discussed earlier in this section. >>> array = [[set()] * n] * m >>> array[2][5].add(7) >>> pprint.pprint(array) [[set([7]), set([7]), set([7]), set([7]), set([7]), set([7]), set([7])], [set([7]), set([7]), set([7]), set([7]), set([7]), set([7]), set([7])], [set([7]), set([7]), set([7]), set([7]), set([7]), set([7]), set([7])]] Iteration is an important programming device. It is tempting to adopt idioms from other languages. However, Python offers some elegant and highly readable alternatives, as we have seen. 4.4 Functions: The Foundation of Structured Programming Functions provide an effective way to package and reuse program code, as already explained in Section 2.3. For example, suppose we find that we often want to read text from an HTML file. This involves several steps: opening the file, reading it in, normal- izing whitespace, and stripping HTML markup. We can collect these steps into a func- tion, and give it a name such as get_text(), as shown in Example 4-1. Example 4-1. Read text from a file. import re def get_text(file): """Read text from a file, normalizing whitespace and stripping HTML markup.""" text = open(file).read() text = re.sub('\s+', ' ', text) text = re.sub(r'<.*?>', ' ', text) return text Now, any time we want to get cleaned-up text from an HTML file, we can just call get_text() with the name of the file as its only argument. It will return a string, and we can assign this to a variable, e.g., contents = get_text("test.html"). Each time we want to use this series of steps, we only have to call the function. Using functions has the benefit of saving space in our program. More importantly, our choice of name for the function helps make the program readable. In the case of the preceding example, whenever our program needs to read cleaned-up text from a file we don’t have to clutter the program with four lines of code; we simply need to call get_text(). This naming helps to provide some “semantic interpretation”—it helps a reader of our program to see what the program “means.” 142 | Chapter 4: Writing Structured Programs Notice that this example function definition contains a string. The first string inside a function definition is called a docstring. Not only does it document the purpose of the function to someone reading the code, it is accessible to a programmer who has loaded the code from a file: >>> help(get_text) Help on function get_text: get_text(file) Read text from a file, normalizing whitespace and stripping HTML markup. We have seen that functions help to make our work reusable and readable. They also help make it reliable. When we reuse code that has already been developed and tested, we can be more confident that it handles a variety of cases correctly. We also remove the risk of forgetting some important step or introducing a bug. The program that calls our function also has increased reliability. The author of that program is dealing with a shorter program, and its components behave transparently. To summarize, as its name suggests, a function captures functionality. It is a segment of code that can be given a meaningful name and which performs a well-defined task. Functions allow us to abstract away from the details, to see a bigger picture, and to program more effectively. The rest of this section takes a closer look at functions, exploring the mechanics and discussing ways to make your programs easier to read. Function Inputs and Outputs We pass information to functions using a function’s parameters, the parenthesized list of variables and constants following the function’s name in the function definition. Here’s a complete example: >>> def repeat(msg, num): ... return ' '.join([msg] * num) >>> monty = 'Monty Python' >>> repeat(monty, 3) 'Monty Python Monty Python Monty Python' We first define the function to take two parameters, msg and num . Then, we call the function and pass it two arguments, monty and 3 ; these arguments fill the “place- holders” provided by the parameters and provide values for the occurrences of msg and num in the function body. It is not necessary to have any parameters, as we see in the following example: >>> def monty(): ... return "Monty Python" >>> monty() 'Monty Python' 4.4 Functions: The Foundation of Structured Programming | 143 A function usually communicates its results back to the calling program via the return statement, as we have just seen. To the calling program, it looks as if the function call had been replaced with the function’s result: >>> repeat(monty(), 3) 'Monty Python Monty Python Monty Python' >>> repeat('Monty Python', 3) 'Monty Python Monty Python Monty Python' A Python function is not required to have a return statement. Some functions do their work as a side effect, printing a result, modifying a file, or updating the contents of a parameter to the function (such functions are called “procedures” in some other programming languages). Consider the following three sort functions. The third one is dangerous because a pro- grammer could use it without realizing that it had modified its input. In general, func- tions should modify the contents of a parameter (my_sort1()), or return a value (my_sort2()), but not both (my_sort3()). >>> def my_sort1(mylist): # good: modifies its argument, no return value ... mylist.sort() >>> def my_sort2(mylist): # good: doesn't touch its argument, returns value ... return sorted(mylist) >>> def my_sort3(mylist): # bad: modifies its argument and also returns it ... mylist.sort() ... return mylist Parameter Passing Back in Section 4.1, you saw that assignment works on values, but that the value of a structured object is a reference to that object. The same is true for functions. Python interprets function parameters as values (this is known as call-by-value). In the fol- lowing code, set_up() has two parameters, both of which are modified inside the func- tion. We begin by assigning an empty string to w and an empty dictionary to p. After calling the function, w is unchanged, while p is changed: >>> def set_up(word, properties): ... word = 'lolcat' ... properties.append('noun') ... properties = 5 ... >>> w = '' >>> p = [] >>> set_up(w, p) >>> w '' >>> p ['noun'] Notice that w was not changed by the function. When we called set_up(w, p), the value of w (an empty string) was assigned to a new variable word. Inside the function, the value 144 | Chapter 4: Writing Structured Programs of word was modified. However, that change did not propagate to w. This parameter passing is identical to the following sequence of assignments: >>> w = '' >>> word = w >>> word = 'lolcat' >>> w '' Let’s look at what happened with the list p. When we called set_up(w, p), the value of p (a reference to an empty list) was assigned to a new local variable properties, so both variables now reference the same memory location. The function modifies properties, and this change is also reflected in the value of p, as we saw. The function also assigned a new value to properties (the number 5); this did not modify the contents at that memory location, but created a new local variable. This behavior is just as if we had done the following sequence of assignments: >>> p = [] >>> properties = p >>> properties.append['noun'] >>> properties = 5 >>> p ['noun'] Thus, to understand Python’s call-by-value parameter passing, it is enough to under- stand how assignment works. Remember that you can use the id() function and is operator to check your understanding of object identity after each statement. Variable Scope Function definitions create a new local scope for variables. When you assign to a new variable inside the body of a function, the name is defined only within that function. The name is not visible outside the function, or in other functions. This behavior means you can choose variable names without being concerned about collisions with names used in your other function definitions. When you refer to an existing name from within the body of a function, the Python interpreter first tries to resolve the name with respect to the names that are local to the function. If nothing is found, the interpreter checks whether it is a global name within the module. Finally, if that does not succeed, the interpreter checks whether the name is a Python built-in. This is the so-called LGB rule of name resolution: local, then global, then built-in. Caution! A function can create a new global variable, using the global declaration. However, this practice should be avoided as much as possible. Defining global variables inside a function introduces dependencies on context and limits the portability (or reusability) of the function. In general you should use parameters for function inputs and return values for function outputs. 4.4 Functions: The Foundation of Structured Programming | 145 Checking Parameter Types Python does not force us to declare the type of a variable when we write a program, and this permits us to define functions that are flexible about the type of their argu- ments. For example, a tagger might expect a sequence of words, but it wouldn’t care whether this sequence is expressed as a list, a tuple, or an iterator (a new sequence type that we’ll discuss later). However, often we want to write programs for later use by others, and want to program in a defensive style, providing useful warnings when functions have not been invoked correctly. The author of the following tag() function assumed that its argument would always be a string. >>> def tag(word): ... if word in ['a', 'the', 'all']: ... return 'det' ... else: ... return 'noun' ... >>> tag('the') 'det' >>> tag('knight') 'noun' >>> tag(["'Tis", 'but', 'a', 'scratch']) 'noun' The function returns sensible values for the arguments 'the' and 'knight', but look what happens when it is passed a list —it fails to complain, even though the result which it returns is clearly incorrect. The author of this function could take some extra steps to ensure that the word parameter of the tag() function is a string. A naive ap- proach would be to check the type of the argument using if not type(word) is str, and if word is not a string, to simply return Python’s special empty value, None. This is a slight improvement, because the function is checking the type of the argument, and trying to return a “special” diagnostic value for the wrong input. However, it is also dangerous because the calling program may not detect that None is intended as a “spe- cial” value, and this diagnostic return value may then be propagated to other parts of the program with unpredictable consequences. This approach also fails if the word is a Unicode string, which has type unicode, not str. Here’s a better solution, using an assert statement together with Python’s basestring type that generalizes over both unicode and str. >>> def tag(word): ... assert isinstance(word, basestring), "argument to tag() must be a string" ... if word in ['a', 'the', 'all']: ... return 'det' ... else: ... return 'noun' If the assert statement fails, it will produce an error that cannot be ignored, since it halts program execution. Additionally, the error message is easy to interpret. Adding 146 | Chapter 4: Writing Structured Programs assertions to a program helps you find logical errors, and is a kind of defensive pro- gramming. A more fundamental approach is to document the parameters to each function using docstrings, as described later in this section. Functional Decomposition Well-structured programs usually make extensive use of functions. When a block of program code grows longer than 10–20 lines, it is a great help to readability if the code is broken up into one or more functions, each one having a clear purpose. This is analogous to the way a good essay is divided into paragraphs, each expressing one main idea. Functions provide an important kind of abstraction. They allow us to group multiple actions into a single, complex action, and associate a name with it. (Compare this with the way we combine the actions of go and bring back into a single more complex action fetch.) When we use functions, the main program can be written at a higher level of abstraction, making its structure transparent, as in the following: >>> data = load_corpus() >>> results = analyze(data) >>> present(results) Appropriate use of functions makes programs more readable and maintainable. Addi- tionally, it becomes possible to reimplement a function—replacing the function’s body with more efficient code—without having to be concerned with the rest of the program. Consider the freq_words function in Example 4-2. It updates the contents of a frequency distribution that is passed in as a parameter, and it also prints a list of the n most frequent words. Example 4-2. Poorly designed function to compute frequent words. def freq_words(url, freqdist, n): text = nltk.clean_url(url) for word in nltk.word_tokenize(text): freqdist.inc(word.lower()) print freqdist.keys()[:n] >>> constitution = "http://www.archives.gov/national-archives-experience" \ ... "/charters/constitution_transcript.html" >>> fd = nltk.FreqDist() >>> freq_words(constitution, fd, 20) ['the', 'of', 'charters', 'bill', 'constitution', 'rights', ',', 'declaration', 'impact', 'freedom', '-', 'making', 'independence'] This function has a number of problems. The function has two side effects: it modifies the contents of its second parameter, and it prints a selection of the results it has com- puted. The function would be easier to understand and to reuse elsewhere if we initialize the FreqDist() object inside the function (in the same place it is populated), and if we moved the selection and display of results to the calling program. In Example 4-3 we refactor this function, and simplify its interface by providing a single url parameter. 4.4 Functions: The Foundation of Structured Programming | 147 Example 4-3. Well-designed function to compute frequent words. def freq_words(url): freqdist = nltk.FreqDist() text = nltk.clean_url(url) for word in nltk.word_tokenize(text): freqdist.inc(word.lower()) return freqdist >>> fd = freq_words(constitution) >>> print fd.keys()[:20] ['the', 'of', 'charters', 'bill', 'constitution', 'rights', ',', 'declaration', 'impact', 'freedom', '-', 'making', 'independence'] Note that we have now simplified the work of freq_words to the point that we can do its work with three lines of code: >>> words = nltk.word_tokenize(nltk.clean_url(constitution)) >>> fd = nltk.FreqDist(word.lower() for word in words) >>> fd.keys()[:20] ['the', 'of', 'charters', 'bill', 'constitution', 'rights', ',', 'declaration', 'impact', 'freedom', '-', 'making', 'independence'] Documenting Functions If we have done a good job at decomposing our program into functions, then it should be easy to describe the purpose of each function in plain language, and provide this in the docstring at the top of the function definition. This statement should not explain how the functionality is implemented; in fact, it should be possible to reimplement the function using a different method without changing this statement. For the simplest functions, a one-line docstring is usually adequate (see Example 4-1). You should provide a triple-quoted string containing a complete sentence on a single line. For non-trivial functions, you should still provide a one-sentence summary on the first line, since many docstring processing tools index this string. This should be fol- lowed by a blank line, then a more detailed description of the functionality (see http:// www.python.org/dev/peps/pep-0257/ for more information on docstring conventions). Docstrings can include a doctest block, illustrating the use of the function and the expected output. These can be tested automatically using Python’s docutils module. Docstrings should document the type of each parameter to the function, and the return type. At a minimum, that can be done in plain text. However, note that NLTK uses the “epytext” markup language to document parameters. This format can be automatically converted into richly structured API documentation (see http://www.nltk.org/), and in- cludes special handling of certain “fields,” such as @param, which allow the inputs and outputs of functions to be clearly documented. Example 4-4 illustrates a complete docstring. 148 | Chapter 4: Writing Structured Programs Example 4-4. Illustration of a complete docstring, consisting of a one-line summary, a more detailed explanation, a doctest example, and epytext markup specifying the parameters, types, return type, and exceptions. def accuracy(reference, test): """ Calculate the fraction of test items that equal the corresponding reference items. Given a list of reference values and a corresponding list of test values, return the fraction of corresponding values that are equal. In particular, return the fraction of indexes {0>> accuracy(['ADJ', 'N', 'V', 'N'], ['N', 'N', 'V', 'ADJ']) 0.5 @param reference: An ordered list of reference values. @type reference: C{list} @param test: A list of values to compare against the corresponding reference values. @type test: C{list} @rtype: C{float} @raise ValueError: If C{reference} and C{length} do not have the same length. """ if len(reference) != len(test): raise ValueError("Lists must have the same length.") num_correct = 0 for x, y in izip(reference, test): if x == y: num_correct += 1 return float(num_correct) / len(reference) 4.5 Doing More with Functions This section discusses more advanced features, which you may prefer to skip on the first time through this chapter. Functions As Arguments So far the arguments we have passed into functions have been simple objects, such as strings, or structured objects, such as lists. Python also lets us pass a function as an argument to another function. Now we can abstract out the operation, and apply a different operation on the same data. As the following examples show, we can pass the built-in function len() or a user-defined function last_letter() as arguments to an- other function: >>> sent = ['Take', 'care', 'of', 'the', 'sense', ',', 'and', 'the', ... 'sounds', 'will', 'take', 'care', 'of', 'themselves', '.'] >>> def extract_property(prop): ... return [prop(word) for word in sent] ... 4.5 Doing More with Functions | 149 >>> extract_property(len) [4, 4, 2, 3, 5, 1, 3, 3, 6, 4, 4, 4, 2, 10, 1] >>> def last_letter(word): ... return word[-1] >>> extract_property(last_letter) ['e', 'e', 'f', 'e', 'e', ',', 'd', 'e', 's', 'l', 'e', 'e', 'f', 's', '.'] The objects len and last_letter can be passed around like lists and dictionaries. Notice that parentheses are used after a function name only if we are invoking the function; when we are simply treating the function as an object, these are omitted. Python provides us with one more way to define functions as arguments to other func- tions, so-called lambda expressions. Supposing there was no need to use the last_let ter() function in multiple places, and thus no need to give it a name. Let’s suppose we can equivalently write the following: >>> extract_property(lambda w: w[-1]) ['e', 'e', 'f', 'e', 'e', ',', 'd', 'e', 's', 'l', 'e', 'e', 'f', 's', '.'] Our next example illustrates passing a function to the sorted() function. When we call the latter with a single argument (the list to be sorted), it uses the built-in comparison function cmp(). However, we can supply our own sort function, e.g., to sort by de- creasing length. >>> sorted(sent) [',', '.', 'Take', 'and', 'care', 'care', 'of', 'of', 'sense', 'sounds', 'take', 'the', 'the', 'themselves', 'will'] >>> sorted(sent, cmp) [',', '.', 'Take', 'and', 'care', 'care', 'of', 'of', 'sense', 'sounds', 'take', 'the', 'the', 'themselves', 'will'] >>> sorted(sent, lambda x, y: cmp(len(y), len(x))) ['themselves', 'sounds', 'sense', 'Take', 'care', 'will', 'take', 'care', 'the', 'and', 'the', 'of', 'of', ',', '.'] Accumulative Functions These functions start by initializing some storage, and iterate over input to build it up, before returning some final object (a large structure or aggregated result). A standard way to do this is to initialize an empty list, accumulate the material, then return the list, as shown in function search1() in Example 4-5. Example 4-5. Accumulating output into a list. def search1(substring, words): result = [] for word in words: if substring in word: result.append(word) return result def search2(substring, words): for word in words: if substring in word: yield word 150 | Chapter 4: Writing Structured Programs print "search1:" for item in search1('zz', nltk.corpus.brown.words()): print item print "search2:" for item in search2('zz', nltk.corpus.brown.words()): print item The function search2() is a generator. The first time this function is called, it gets as far as the yield statement and pauses. The calling program gets the first word and does any necessary processing. Once the calling program is ready for another word, execu- tion of the function is continued from where it stopped, until the next time it encounters a yield statement. This approach is typically more efficient, as the function only gen- erates the data as it is required by the calling program, and does not need to allocate additional memory to store the output (see the earlier discussion of generator expres- sions). Here’s a more sophisticated example of a generator which produces all permutations of a list of words. In order to force the permutations() function to generate all its output, we wrap it with a call to list() . >>> def permutations(seq): ... if len(seq) <= 1: ... yield seq ... else: ... for perm in permutations(seq[1:]): ... for i in range(len(perm)+1): ... yield perm[:i] + seq[0:1] + perm[i:] ... >>> list(permutations(['police', 'fish', 'buffalo'])) [['police', 'fish', 'buffalo'], ['fish', 'police', 'buffalo'], ['fish', 'buffalo', 'police'], ['police', 'buffalo', 'fish'], ['buffalo', 'police', 'fish'], ['buffalo', 'fish', 'police']] The permutations function uses a technique called recursion, discussed later in Section 4.7. The ability to generate permutations of a set of words is useful for creating data to test a grammar (Chapter 8). Higher-Order Functions Python provides some higher-order functions that are standard features of functional programming languages such as Haskell. We illustrate them here, alongside the equiv- alent expression using list comprehensions. Let’s start by defining a function is_content_word() which checks whether a word is from the open class of content words. We use this function as the first parameter of filter(), which applies the function to each item in the sequence contained in its second parameter, and retains only the items for which the function returns True. 4.5 Doing More with Functions | 151 >>> def is_content_word(word): ... return word.lower() not in ['a', 'of', 'the', 'and', 'will', ',', '.'] >>> sent = ['Take', 'care', 'of', 'the', 'sense', ',', 'and', 'the', ... 'sounds', 'will', 'take', 'care', 'of', 'themselves', '.'] >>> filter(is_content_word, sent) ['Take', 'care', 'sense', 'sounds', 'take', 'care', 'themselves'] >>> [w for w in sent if is_content_word(w)] ['Take', 'care', 'sense', 'sounds', 'take', 'care', 'themselves'] Another higher-order function is map(), which applies a function to every item in a sequence. It is a general version of the extract_property() function we saw earlier in this section. Here is a simple way to find the average length of a sentence in the news section of the Brown Corpus, followed by an equivalent version with list comprehen- sion calculation: >>> lengths = map(len, nltk.corpus.brown.sents(categories='news')) >>> sum(lengths) / len(lengths) 21.7508111616 >>> lengths = [len(w) for w in nltk.corpus.brown.sents(categories='news'))] >>> sum(lengths) / len(lengths) 21.7508111616 In the previous examples, we specified a user-defined function is_content_word() and a built-in function len(). We can also provide a lambda expression. Here’s a pair of equivalent examples that count the number of vowels in each word. >>> map(lambda w: len(filter(lambda c: c.lower() in "aeiou", w)), sent) [2, 2, 1, 1, 2, 0, 1, 1, 2, 1, 2, 2, 1, 3, 0] >>> [len([c for c in w if c.lower() in "aeiou"]) for w in sent] [2, 2, 1, 1, 2, 0, 1, 1, 2, 1, 2, 2, 1, 3, 0] The solutions based on list comprehensions are usually more readable than the solu- tions based on higher-order functions, and we have favored the former approach throughout this book. Named Arguments When there are a lot of parameters it is easy to get confused about the correct order. Instead we can refer to parameters by name, and even assign them a default value just in case one was not provided by the calling program. Now the parameters can be speci- fied in any order, and can be omitted. >>> def repeat(msg='', num=1): ... return msg * num >>> repeat(num=3) '' >>> repeat(msg='Alice') 'Alice' >>> repeat(num=5, msg='Alice') 'AliceAliceAliceAliceAlice' These are called keyword arguments. If we mix these two kinds of parameters, then we must ensure that the unnamed parameters precede the named ones. It has to be this 152 | Chapter 4: Writing Structured Programs way, since unnamed parameters are defined by position. We can define a function that takes an arbitrary number of unnamed and named parameters, and access them via an in-place list of arguments *args and an in-place dictionary of keyword arguments **kwargs. >>> def generic(*args, **kwargs): ... print args ... print kwargs ... >>> generic(1, "African swallow", monty="python") (1, 'African swallow') {'monty': 'python'} When *args appears as a function parameter, it actually corresponds to all the unnamed parameters of the function. As another illustration of this aspect of Python syntax, consider the zip() function, which operates on a variable number of arguments. We’ll use the variable name *song to demonstrate that there’s nothing special about the name *args. >>> song = [['four', 'calling', 'birds'], ... ['three', 'French', 'hens'], ... ['two', 'turtle', 'doves']] >>> zip(song[0], song[1], song[2]) [('four', 'three', 'two'), ('calling', 'French', 'turtle'), ('birds', 'hens', 'doves')] >>> zip(*song) [('four', 'three', 'two'), ('calling', 'French', 'turtle'), ('birds', 'hens', 'doves')] It should be clear from this example that typing *song is just a convenient shorthand, and equivalent to typing out song[0], song[1], song[2]. Here’s another example of the use of keyword arguments in a function definition, along with three equivalent ways to call the function: >>> def freq_words(file, min=1, num=10): ... text = open(file).read() ... tokens = nltk.word_tokenize(text) ... freqdist = nltk.FreqDist(t for t in tokens if len(t) >= min) ... return freqdist.keys()[:num] >>> fw = freq_words('ch01.rst', 4, 10) >>> fw = freq_words('ch01.rst', min=4, num=10) >>> fw = freq_words('ch01.rst', num=10, min=4) A side effect of having named arguments is that they permit optionality. Thus we can leave out any arguments where we are happy with the default value: freq_words('ch01.rst', min=4), freq_words('ch01.rst', 4). Another common use of optional arguments is to permit a flag. Here’s a revised version of the same function that reports its progress if a verbose flag is set: >>> def freq_words(file, min=1, num=10, verbose=False): ... freqdist = FreqDist() ... if trace: print "Opening", file ... text = open(file).read() ... if trace: print "Read in %d characters" % len(file) ... for word in nltk.word_tokenize(text): 4.5 Doing More with Functions | 153 ... if len(word) >= min: ... freqdist.inc(word) ... if trace and freqdist.N() % 100 == 0: print "." ... if trace: print ... return freqdist.keys()[:num] Caution! Take care not to use a mutable object as the default value of a parameter. A series of calls to the function will use the same object, sometimes with bizarre results, as we will see in the discussion of debugging later. 4.6 Program Development Programming is a skill that is acquired over several years of experience with a variety of programming languages and tasks. Key high-level abilities are algorithm design and its manifestation in structured programming. Key low-level abilities include familiarity with the syntactic constructs of the language, and knowledge of a variety of diagnostic methods for trouble-shooting a program which does not exhibit the expected behavior. This section describes the internal structure of a program module and how to organize a multi-module program. Then it describes various kinds of error that arise during program development, what you can do to fix them and, better still, to avoid them in the first place. Structure of a Python Module The purpose of a program module is to bring logically related definitions and functions together in order to facilitate reuse and abstraction. Python modules are nothing more than individual .py files. For example, if you were working with a particular corpus format, the functions to read and write the format could be kept together. Constants used by both formats, such as field separators, or a EXTN = ".inf" filename extension, could be shared. If the format was updated, you would know that only one file needed to be changed. Similarly, a module could contain code for creating and manipulating a particular data structure such as syntax trees, or code for performing a particular processing task such as plotting corpus statistics. When you start writing Python modules, it helps to have some examples to emulate. You can locate the code for any NLTK module on your system using the __file__ variable: >>> nltk.metrics.distance.__file__ '/usr/lib/python2.5/site-packages/nltk/metrics/distance.pyc' This returns the location of the compiled .pyc file for the module, and you’ll probably see a different location on your machine. The file that you will need to open is the corresponding .py source file, and this will be in the same directory as the .pyc file. 154 | Chapter 4: Writing Structured Programs Alternatively, you can view the latest version of this module on the Web at http://code .google.com/p/nltk/source/browse/trunk/nltk/nltk/metrics/distance.py. Like every other NLTK module, distance.py begins with a group of comment lines giving a one-line title of the module and identifying the authors. (Since the code is distributed, it also includes the URL where the code is available, a copyright statement, and license information.) Next is the module-level docstring, a triple-quoted multiline string con- taining information about the module that will be printed when someone types help(nltk.metrics.distance). # Natural Language Toolkit: Distance Metrics # # Copyright (C) 2001-2009 NLTK Project # Author: Edward Loper # Steven Bird # Tom Lippincott # URL: # For license information, see LICENSE.TXT # """ Distance Metrics. Compute the distance between two items (usually strings). As metrics, they must satisfy the following three requirements: 1. d(a, a) = 0 2. d(a, b) >= 0 3. d(a, c) <= d(a, b) + d(b, c) """ After this comes all the import statements required for the module, then any global variables, followed by a series of function definitions that make up most of the module. Other modules define “classes,” the main building blocks of object-oriented program- ming, which falls outside the scope of this book. (Most NLTK modules also include a demo() function, which can be used to see examples of the module in use.) Some module variables and functions are only used within the module. These should have names beginning with an underscore, e.g., _helper(), since this will hide the name. If another module imports this one, using the idiom: from module import *, these names will not be imported. You can optionally list the externally accessible names of a module using a special built-in variable like this: __all__ = ['edit_dis tance', 'jaccard_distance']. Multimodule Programs Some programs bring together a diverse range of tasks, such as loading data from a corpus, performing some analysis tasks on the data, then visualizing it. We may already 4.6 Program Development | 155 have stable modules that take care of loading data and producing visualizations. Our work might involve coding up the analysis task, and just invoking functions from the existing modules. This scenario is depicted in Figure 4-2. Figure 4-2. Structure of a multimodule program: The main program my_program.py imports functions from two other modules; unique analysis tasks are localized to the main program, while common loading and visualization tasks are kept apart to facilitate reuse and abstraction. By dividing our work into several modules and using import statements to access func- tions defined elsewhere, we can keep the individual modules simple and easy to main- tain. This approach will also result in a growing collection of modules, and make it possible for us to build sophisticated systems involving a hierarchy of modules. De- signing such systems well is a complex software engineering task, and beyond the scope of this book. Sources of Error Mastery of programming depends on having a variety of problem-solving skills to draw upon when the program doesn’t work as expected. Something as trivial as a misplaced symbol might cause the program to behave very differently. We call these “bugs” be- cause they are tiny in comparison to the damage they can cause. They creep into our code unnoticed, and it’s only much later when we’re running the program on some new data that their presence is detected. Sometimes, fixing one bug only reveals an- other, and we get the distinct impression that the bug is on the move. The only reas- surance we have is that bugs are spontaneous and not the fault of the programmer. 156 | Chapter 4: Writing Structured Programs Flippancy aside, debugging code is hard because there are so many ways for it to be faulty. Our understanding of the input data, the algorithm, or even the programming language, may be at fault. Let’s look at examples of each of these. First, the input data may contain some unexpected characters. For example, WordNet synset names have the form tree.n.01, with three components separated using periods. The NLTK WordNet module initially decomposed these names using split('.'). However, this method broke when someone tried to look up the word PhD, which has the synset name ph.d..n.01, containing four periods instead of the expected two. The solution was to use rsplit('.', 2) to do at most two splits, using the rightmost in- stances of the period, and leaving the ph.d. string intact. Although several people had tested the module before it was released, it was some weeks before someone detected the problem (see http://code.google.com/p/nltk/issues/detail?id=297). Second, a supplied function might not behave as expected. For example, while testing NLTK’s interface to WordNet, one of the authors noticed that no synsets had any antonyms defined, even though the underlying database provided a large quantity of antonym information. What looked like a bug in the WordNet interface turned out to be a misunderstanding about WordNet itself: antonyms are defined for lemmas, not for synsets. The only “bug” was a misunderstanding of the interface (see http://code .google.com/p/nltk/issues/detail?id=98). Third, our understanding of Python’s semantics may be at fault. It is easy to make the wrong assumption about the relative scope of two operators. For example, "%s.%s. %02d" % "ph.d.", "n", 1 produces a runtime error TypeError: not enough arguments for format string. This is because the percent operator has higher precedence than the comma operator. The fix is to add parentheses in order to force the required scope. As another example, suppose we are defining a function to collect all tokens of a text having a given length. The function has parameters for the text and the word length, and an extra parameter that allows the initial value of the result to be given as a parameter: >>> def find_words(text, wordlength, result=[]): ... for word in text: ... if len(word) == wordlength: ... result.append(word) ... return result >>> find_words(['omg', 'teh', 'lolcat', 'sitted', 'on', 'teh', 'mat'], 3) ['omg', 'teh', 'teh', 'mat'] >>> find_words(['omg', 'teh', 'lolcat', 'sitted', 'on', 'teh', 'mat'], 2, ['ur']) ['ur', 'on'] >>> find_words(['omg', 'teh', 'lolcat', 'sitted', 'on', 'teh', 'mat'], 3) ['omg', 'teh', 'teh', 'mat', 'omg', 'teh', 'teh', 'mat'] The first time we call find_words() , we get all three-letter words as expected. The second time we specify an initial value for the result, a one-element list ['ur'], and as expected, the result has this word along with the other two-letter word in our text. Now, the next time we call find_words() we use the same parameters as in , but we get a different result! Each time we call find_words() with no third parameter, the 4.6 Program Development | 157 result will simply extend the result of the previous call, rather than start with the empty result list as specified in the function definition. The program’s behavior is not as ex- pected because we incorrectly assumed that the default value was created at the time the function was invoked. However, it is created just once, at the time the Python interpreter loads the function. This one list object is used whenever no explicit value is provided to the function. Debugging Techniques Since most code errors result from the programmer making incorrect assumptions, the first thing to do when you detect a bug is to check your assumptions. Localize the prob- lem by adding print statements to the program, showing the value of important vari- ables, and showing how far the program has progressed. If the program produced an “exception”—a runtime error—the interpreter will print a stack trace, pinpointing the location of program execution at the time of the error. If the program depends on input data, try to reduce this to the smallest size while still producing the error. Once you have localized the problem to a particular function or to a line of code, you need to work out what is going wrong. It is often helpful to recreate the situation using the interactive command line. Define some variables, and then copy-paste the offending line of code into the session and see what happens. Check your understanding of the code by reading some documentation and examining other code samples that purport to do the same thing that you are trying to do. Try explaining your code to someone else, in case she can see where things are going wrong. Python provides a debugger which allows you to monitor the execution of your pro- gram, specify line numbers where execution will stop (i.e., breakpoints), and step through sections of code and inspect the value of variables. You can invoke the debug- ger on your code as follows: >>> import pdb >>> import mymodule >>> pdb.run('mymodule.myfunction()') It will present you with a prompt (Pdb) where you can type instructions to the debugger. Type help to see the full list of commands. Typing step (or just s) will execute the current line and stop. If the current line calls a function, it will enter the function and stop at the first line. Typing next (or just n) is similar, but it stops execution at the next line in the current function. The break (or b) command can be used to create or list breakpoints. Type continue (or c) to continue execution as far as the next breakpoint. Type the name of any variable to inspect its value. We can use the Python debugger to locate the problem in our find_words() function. Remember that the problem arose the second time the function was called. We’ll start by calling the function without using the debugger , using the smallest possible input. The second time, we’ll call it with the debugger . 158 | Chapter 4: Writing Structured Programs >>> import pdb >>> find_words(['cat'], 3) ['cat'] >>> pdb.run("find_words(['dog'], 3)") > (1)() (Pdb) step --Call-- > (1)find_words() (Pdb) args text = ['dog'] wordlength = 3 result = ['cat'] Here we typed just two commands into the debugger: step took us inside the function, and args showed the values of its arguments (or parameters). We see immediately that result has an initial value of ['cat'], and not the empty list as expected. The debugger has helped us to localize the problem, prompting us to check our understanding of Python functions. Defensive Programming In order to avoid some of the pain of debugging, it helps to adopt some defensive programming habits. Instead of writing a 20-line program and then testing it, build the program bottom-up out of small pieces that are known to work. Each time you combine these pieces to make a larger unit, test it carefully to see that it works as expected. Consider adding assert statements to your code, specifying properties of a variable, e.g., assert(isinstance(text, list)). If the value of the text variable later becomes a string when your code is used in some larger context, this will raise an AssertionError and you will get immediate notification of the problem. Once you think you’ve found the bug, view your solution as a hypothesis. Try to predict the effect of your bugfix before re-running the program. If the bug isn’t fixed, don’t fall into the trap of blindly changing the code in the hope that it will magically start working again. Instead, for each change, try to articulate a hypothesis about what is wrong and why the change will fix the problem. Then undo the change if the problem was not resolved. As you develop your program, extend its functionality, and fix any bugs, it helps to maintain a suite of test cases. This is called regression testing, since it is meant to detect situations where the code “regresses”—where a change to the code has an un- intended side effect of breaking something that used to work. Python provides a simple regression-testing framework in the form of the doctest module. This module searches a file of code or documentation for blocks of text that look like an interactive Python session, of the form you have already seen many times in this book. It executes the Python commands it finds, and tests that their output matches the output supplied in the original file. Whenever there is a mismatch, it reports the expected and actual val- ues. For details, please consult the doctest documentation at 4.6 Program Development | 159 http://docs.python.org/library/doctest.html. Apart from its value for regression testing, the doctest module is useful for ensuring that your software documentation stays in sync with your code. Perhaps the most important defensive programming strategy is to set out your code clearly, choose meaningful variable and function names, and simplify the code wher- ever possible by decomposing it into functions and modules with well-documented interfaces. 4.7 Algorithm Design This section discusses more advanced concepts, which you may prefer to skip on the first time through this chapter. A major part of algorithmic problem solving is selecting or adapting an appropriate algorithm for the problem at hand. Sometimes there are several alternatives, and choos- ing the best one depends on knowledge about how each alternative performs as the size of the data grows. Whole books are written on this topic, and we only have space to introduce some key concepts and elaborate on the approaches that are most prevalent in natural language processing. The best-known strategy is known as divide-and-conquer. We attack a problem of size n by dividing it into two problems of size n/2, solve these problems, and combine their results into a solution of the original problem. For example, suppose that we had a pile of cards with a single word written on each card. We could sort this pile by splitting it in half and giving it to two other people to sort (they could do the same in turn). Then, when two sorted piles come back, it is an easy task to merge them into a single sorted pile. See Figure 4-3 for an illustration of this process. Another example is the process of looking up a word in a dictionary. We open the book somewhere around the middle and compare our word with the current page. If it’s earlier in the dictionary, we repeat the process on the first half; if it’s later, we use the second half. This search method is called binary search since it splits the problem in half at every step. In another approach to algorithm design, we attack a problem by transforming it into an instance of a problem we already know how to solve. For example, in order to detect duplicate entries in a list, we can pre-sort the list, then scan through it once to check whether any adjacent pairs of elements are identical. Recursion The earlier examples of sorting and searching have a striking property: to solve a prob- lem of size n, we have to break it in half and then work on one or more problems of size n/2. A common way to implement such methods uses recursion. We define a function f, which simplifies the problem, and calls itself to solve one or more easier 160 | Chapter 4: Writing Structured Programs instances of the same problem. It then combines the results into a solution for the original problem. For example, suppose we have a set of n words, and want to calculate how many dif- ferent ways they can be combined to make a sequence of words. If we have only one word (n=1), there is just one way to make it into a sequence. If we have a set of two words, there are two ways to put them into a sequence. For three words there are six possibilities. In general, for n words, there are n × n-1 × … × 2 × 1 ways (i.e., the factorial of n). We can code this up as follows: >>> def factorial1(n): ... result = 1 ... for i in range(n): ... result *= (i+1) ... return result However, there is also a recursive algorithm for solving this problem, based on the following observation. Suppose we have a way to construct all orderings for n-1 distinct words. Then for each such ordering, there are n places where we can insert a new word: at the start, the end, or any of the n-2 boundaries between the words. Thus we simply multiply the number of solutions found for n-1 by the value of n. We also need the base case, to say that if we have a single word, there’s just one ordering. We can code this up as follows: >>> def factorial2(n): ... if n == 1: ... return 1 ... else: ... return n * factorial2(n-1) Figure 4-3. Sorting by divide-and-conquer: To sort an array, we split it in half and sort each half (recursively); we merge each sorted half back into a whole list (again recursively); this algorithm is known as “Merge Sort.” 4.7 Algorithm Design | 161 These two algorithms solve the same problem. One uses iteration while the other uses recursion. We can use recursion to navigate a deeply nested object, such as the Word- Net hypernym hierarchy. Let’s count the size of the hypernym hierarchy rooted at a given synset s. We’ll do this by finding the size of each hyponym of s, then adding these together (we will also add 1 for the synset itself). The following function size1() does this work; notice that the body of the function includes a recursive call to size1(): >>> def size1(s): ... return 1 + sum(size1(child) for child in s.hyponyms()) We can also design an iterative solution to this problem which processes the hierarchy in layers. The first layer is the synset itself , then all the hyponyms of the synset, then all the hyponyms of the hyponyms. Each time through the loop it computes the next layer by finding the hyponyms of everything in the last layer . It also maintains a total of the number of synsets encountered so far . >>> def size2(s): ... layer = [s] ... total = 0 ... while layer: ... total += len(layer) ... layer = [h for c in layer for h in c.hyponyms()] ... return total Not only is the iterative solution much longer, it is harder to interpret. It forces us to think procedurally, and keep track of what is happening with the layer and total variables through time. Let’s satisfy ourselves that both solutions give the same result. We’ll use a new form of the import statement, allowing us to abbreviate the name wordnet to wn: >>> from nltk.corpus import wordnet as wn >>> dog = wn.synset('dog.n.01') >>> size1(dog) 190 >>> size2(dog) 190 As a final example of recursion, let’s use it to construct a deeply nested object. A letter trie is a data structure that can be used for indexing a lexicon, one letter at a time. (The name is based on the word retrieval.) For example, if trie contained a letter trie, then trie['c'] would be a smaller trie which held all words starting with c. Example 4-6 demonstrates the recursive process of building a trie, using Python dictionaries (Sec- tion 5.3). To insert the word chien (French for dog), we split off the c and recursively insert hien into the sub-trie trie['c']. The recursion continues until there are no letters remaining in the word, when we store the intended value (in this case, the word dog). 162 | Chapter 4: Writing Structured Programs Example 4-6. Building a letter trie: A recursive function that builds a nested dictionary structure; each level of nesting contains all words with a given prefix, and a sub-trie containing all possible continuations. def insert(trie, key, value): if key: first, rest = key[0], key[1:] if first not in trie: trie[first] = {} insert(trie[first], rest, value) else: trie['value'] = value >>> trie = nltk.defaultdict(dict) >>> insert(trie, 'chat', 'cat') >>> insert(trie, 'chien', 'dog') >>> insert(trie, 'chair', 'flesh') >>> insert(trie, 'chic', 'stylish') >>> trie = dict(trie) # for nicer printing >>> trie['c']['h']['a']['t']['value'] 'cat' >>> pprint.pprint(trie) {'c': {'h': {'a': {'t': {'value': 'cat'}}, {'i': {'r': {'value': 'flesh'}}}, 'i': {'e': {'n': {'value': 'dog'}}} {'c': {'value': 'stylish'}}}}} Caution! Despite the simplicity of recursive programming, it comes with a cost. Each time a function is called, some state information needs to be push- ed on a stack, so that once the function has completed, execution can continue from where it left off. For this reason, iterative solutions are often more efficient than recursive solutions. Space-Time Trade-offs We can sometimes significantly speed up the execution of a program by building an auxiliary data structure, such as an index. The listing in Example 4-7 implements a simple text retrieval system for the Movie Reviews Corpus. By indexing the document collection, it provides much faster lookup. Example 4-7. A simple text retrieval system. def raw(file): contents = open(file).read() contents = re.sub(r'<.*?>', ' ', contents) contents = re.sub('\s+', ' ', contents) return contents def snippet(doc, term): # buggy text = ' '*30 + raw(doc) + ' '*30 pos = text.index(term) return text[pos-30:pos+30] 4.7 Algorithm Design | 163 print "Building Index..." files = nltk.corpus.movie_reviews.abspaths() idx = nltk.Index((w, f) for f in files for w in raw(f).split()) query = '' while query != "quit": query = raw_input("query> ") if query in idx: for doc in idx[query]: print snippet(doc, query) else: print "Not found" A more subtle example of a space-time trade-off involves replacing the tokens of a corpus with integer identifiers. We create a vocabulary for the corpus, a list in which each word is stored once, then invert this list so that we can look up any word to find its identifier. Each document is preprocessed, so that a list of words becomes a list of integers. Any language models can now work with integers. See the listing in Exam- ple 4-8 for an example of how to do this for a tagged corpus. Example 4-8. Preprocess tagged corpus data, converting all words and tags to integers. def preprocess(tagged_corpus): words = set() tags = set() for sent in tagged_corpus: for word, tag in sent: words.add(word) tags.add(tag) wm = dict((w,i) for (i,w) in enumerate(words)) tm = dict((t,i) for (i,t) in enumerate(tags)) return [[(wm[w], tm[t]) for (w,t) in sent] for sent in tagged_corpus] Another example of a space-time trade-off is maintaining a vocabulary list. If you need to process an input text to check that all words are in an existing vocabulary, the vo- cabulary should be stored as a set, not a list. The elements of a set are automatically indexed, so testing membership of a large set will be much faster than testing mem- bership of the corresponding list. We can test this claim using the timeit module. The Timer class has two parameters: a statement that is executed multiple times, and setup code that is executed once at the beginning. We will simulate a vocabulary of 100,000 items using a list or set of integers. The test statement will generate a random item that has a 50% chance of being in the vocabulary . 164 | Chapter 4: Writing Structured Programs >>> from timeit import Timer >>> vocab_size = 100000 >>> setup_list = "import random; vocab = range(%d)" % vocab_size >>> setup_set = "import random; vocab = set(range(%d))" % vocab_size >>> statement = "random.randint(0, %d) in vocab" % vocab_size * 2 >>> print Timer(statement, setup_list).timeit(1000) 2.78092288971 >>> print Timer(statement, setup_set).timeit(1000) 0.0037260055542 Performing 1,000 list membership tests takes a total of 2.8 seconds, whereas the equiv- alent tests on a set take a mere 0.0037 seconds, or three orders of magnitude faster! Dynamic Programming Dynamic programming is a general technique for designing algorithms which is widely used in natural language processing. The term “programming” is used in a different sense to what you might expect, to mean planning or scheduling. Dynamic program- ming is used when a problem contains overlapping subproblems. Instead of computing solutions to these subproblems repeatedly, we simply store them in a lookup table. In the remainder of this section, we will introduce dynamic programming, but in a rather different context to syntactic parsing. Pingala was an Indian author who lived around the 5th century B.C., and wrote a treatise on Sanskrit prosody called the Chandas Shastra. Virahanka extended this work around the 6th century A.D., studying the number of ways of combining short and long syllables to create a meter of length n. Short syllables, marked S, take up one unit of length, while long syllables, marked L, take two. Pingala found, for example, that there are five ways to construct a meter of length 4: V4 = {LL, SSL, SLS, LSS, SSSS}. Observe that we can split V4 into two subsets, those starting with L and those starting with S, as shown in (1). (1) V4 = LL, LSS i.e. L prefixed to each item of V2 = {L, SS} SSL, SLS, SSSS i.e. S prefixed to each item of V3 = {SL, LS, SSS} With this observation, we can write a little recursive function called virahanka1() to compute these meters, shown in Example 4-9. Notice that, in order to compute V4 we first compute V3 and V2. But to compute V3, we need to first compute V2 and V1. This call structure is depicted in (2). 4.7 Algorithm Design | 165 Example 4-9. Four ways to compute Sanskrit meter: (i) iterative, (ii) bottom-up dynamic programming, (iii) top-down dynamic programming, and (iv) built-in memoization. def virahanka1(n): if n == 0: return [""] elif n == 1: return ["S"] else: s = ["S" + prosody for prosody in virahanka1(n-1)] l = ["L" + prosody for prosody in virahanka1(n-2)] return s + l def virahanka2(n): lookup = [[""], ["S"]] for i in range(n-1): s = ["S" + prosody for prosody in lookup[i+1]] l = ["L" + prosody for prosody in lookup[i]] lookup.append(s + l) return lookup[n] def virahanka3(n, lookup={0:[""], 1:["S"]}): if n not in lookup: s = ["S" + prosody for prosody in virahanka3(n-1)] l = ["L" + prosody for prosody in virahanka3(n-2)] lookup[n] = s + l return lookup[n] from nltk import memoize @memoize def virahanka4(n): if n == 0: return [""] elif n == 1: return ["S"] else: s = ["S" + prosody for prosody in virahanka4(n-1)] l = ["L" + prosody for prosody in virahanka4(n-2)] return s + l >>> virahanka1(4) ['SSSS', 'SSL', 'SLS', 'LSS', 'LL'] >>> virahanka2(4) ['SSSS', 'SSL', 'SLS', 'LSS', 'LL'] >>> virahanka3(4) ['SSSS', 'SSL', 'SLS', 'LSS', 'LL'] >>> virahanka4(4) ['SSSS', 'SSL', 'SLS', 'LSS', 'LL'] 166 | Chapter 4: Writing Structured Programs (2) As you can see, V2 is computed twice. This might not seem like a significant problem, but it turns out to be rather wasteful as n gets large: to compute V20 using this recursive technique, we would compute V2 4,181 times; and for V40 we would compute V2 63,245,986 times! A much better alternative is to store the value of V2 in a table and look it up whenever we need it. The same goes for other values, such as V3 and so on. Function virahanka2() implements a dynamic programming approach to the problem. It works by filling up a table (called lookup) with solutions to all smaller instances of the problem, stopping as soon as we reach the value we’re interested in. At this point we read off the value and return it. Crucially, each subproblem is only ever solved once. Notice that the approach taken in virahanka2() is to solve smaller problems on the way to solving larger problems. Accordingly, this is known as the bottom-up approach to dynamic programming. Unfortunately it turns out to be quite wasteful for some ap- plications, since it may compute solutions to sub-problems that are never required for solving the main problem. This wasted computation can be avoided using the top- down approach to dynamic programming, which is illustrated in the function vira hanka3() in Example 4-9. Unlike the bottom-up approach, this approach is recursive. It avoids the huge wastage of virahanka1() by checking whether it has previously stored the result. If not, it computes the result recursively and stores it in the table. The last step is to return the stored result. The final method, in virahanka4(), is to use a Python “decorator” called memoize, which takes care of the housekeeping work done by virahanka3() without cluttering up the program. This “memoization” process stores the result of each previous call to the function along with the parameters that were used. If the function is subsequently called with the same parameters, it returns the stored result instead of recalculating it. (This aspect of Python syntax is beyond the scope of this book.) This concludes our brief introduction to dynamic programming. We will encounter it again in Section 8.4. 4.8 A Sample of Python Libraries Python has hundreds of third-party libraries, specialized software packages that extend the functionality of Python. NLTK is one such library. To realize the full power of Python programming, you should become familiar with several other libraries. Most of these will need to be manually installed on your computer. 4.8 A Sample of Python Libraries | 167 Matplotlib Python has some libraries that are useful for visualizing language data. The Matplotlib package supports sophisticated plotting functions with a MATLAB-style interface, and is available from http://matplotlib.sourceforge.net/. So far we have focused on textual presentation and the use of formatted print statements to get output lined up in columns. It is often very useful to display numerical data in graphical form, since this often makes it easier to detect patterns. For example, in Example 3-5, we saw a table of numbers showing the frequency of particular modal verbs in the Brown Corpus, classified by genre. The program in Example 4-10 presents the same information in graphical format. The output is shown in Figure 4-4 (a color figure in the graphical display). Example 4-10. Frequency of modals in different sections of the Brown Corpus. colors = 'rgbcmyk' # red, green, blue, cyan, magenta, yellow, black def bar_chart(categories, words, counts): "Plot a bar chart showing counts for each word by category" import pylab ind = pylab.arange(len(words)) width = 1 / (len(categories) + 1) bar_groups = [] for c in range(len(categories)): bars = pylab.bar(ind+c*width, counts[categories[c]], width, color=colors[c % len(colors)]) bar_groups.append(bars) pylab.xticks(ind+width, words) pylab.legend([b[0] for b in bar_groups], categories, loc='upper left') pylab.ylabel('Frequency') pylab.title('Frequency of Six Modal Verbs by Genre') pylab.show() >>> genres = ['news', 'religion', 'hobbies', 'government', 'adventure'] >>> modals = ['can', 'could', 'may', 'might', 'must', 'will'] >>> cfdist = nltk.ConditionalFreqDist( ... (genre, word) ... for genre in genres ... for word in nltk.corpus.brown.words(categories=genre) ... if word in modals) ... >>> counts = {} >>> for genre in genres: ... counts[genre] = [cfdist[genre][word] for word in modals] >>> bar_chart(genres, modals, counts) From the bar chart it is immediately obvious that may and must have almost identical relative frequencies. The same goes for could and might. It is also possible to generate such data visualizations on the fly. For example, a web page with form input could permit visitors to specify search parameters, submit the form, and see a dynamically generated visualization. To do this we have to specify the 168 | Chapter 4: Writing Structured Programs Agg backend for matplotlib, which is a library for producing raster (pixel) images . Next, we use all the same PyLab methods as before, but instead of displaying the result on a graphical terminal using pylab.show(), we save it to a file using pylab.savefig() . We specify the filename and dpi, then print HTML markup that directs the web browser to load the file. >>> import matplotlib >>> matplotlib.use('Agg') >>> pylab.savefig('modals.png') >>> print 'Content-Type: text/html' >>> print >>> print '' >>> print '' >>> print '' Figure 4-4. Bar chart showing frequency of modals in different sections of Brown Corpus: This visualization was produced by the program in Example 4-10. NetworkX The NetworkX package is for defining and manipulating structures consisting of nodes and edges, known as graphs. It is available from https://networkx.lanl.gov/. NetworkX 4.8 A Sample of Python Libraries | 169 can be used in conjunction with Matplotlib to visualize networks, such as WordNet (the semantic network we introduced in Section 2.5). The program in Example 4-11 initializes an empty graph and then traverses the WordNet hypernym hierarchy adding edges to the graph . Notice that the traversal is recursive , applying the programming technique discussed in Section 4.7. The resulting display is shown in Figure 4-5. Example 4-11. Using the NetworkX and Matplotlib libraries. import networkx as nx import matplotlib from nltk.corpus import wordnet as wn def traverse(graph, start, node): graph.depth[node.name] = node.shortest_path_distance(start) for child in node.hyponyms(): graph.add_edge(node.name, child.name) traverse(graph, start, child) def hyponym_graph(start): G = nx.Graph() G.depth = {} traverse(G, start, start) return G def graph_draw(graph): nx.draw_graphviz(graph, node_size = [16 * graph.degree(n) for n in graph], node_color = [graph.depth[n] for n in graph], with_labels = False) matplotlib.pyplot.show() >>> dog = wn.synset('dog.n.01') >>> graph = hyponym_graph(dog) >>> graph_draw(graph) csv Language analysis work often involves data tabulations, containing information about lexical items, the participants in an empirical study, or the linguistic features extracted from a corpus. Here’s a fragment of a simple lexicon, in CSV format: sleep, sli:p, v.i, a condition of body and mind ... walk, wo:k, v.intr, progress by lifting and setting down each foot ... wake, weik, intrans, cease to sleep We can use Python’s CSV library to read and write files stored in this format. For example, we can open a CSV file called lexicon.csv and iterate over its rows : >>> import csv >>> input_file = open("lexicon.csv", "rb") >>> for row in csv.reader(input_file): ... print row ['sleep', 'sli:p', 'v.i', 'a condition of body and mind ...'] 170 | Chapter 4: Writing Structured Programs ['walk', 'wo:k', 'v.intr', 'progress by lifting and setting down each foot ...'] ['wake', 'weik', 'intrans', 'cease to sleep'] Each row is just a list of strings. If any fields contain numerical data, they will appear as strings, and will have to be converted using int() or float(). Figure 4-5. Visualization with NetworkX and Matplotlib: Part of the WordNet hypernym hierarchy is displayed, starting with dog.n.01 (the darkest node in the middle); node size is based on the number of children of the node, and color is based on the distance of the node from dog.n.01; this visualization was produced by the program in Example 4-11. NumPy The NumPy package provides substantial support for numerical processing in Python. NumPy has a multidimensional array object, which is easy to initialize and access: >>> from numpy import array >>> cube = array([ [[0,0,0], [1,1,1], [2,2,2]], ... [[3,3,3], [4,4,4], [5,5,5]], ... [[6,6,6], [7,7,7], [8,8,8]] ]) >>> cube[1,1,1] 4 >>> cube[2].transpose() array([[6, 7, 8], [6, 7, 8], [6, 7, 8]]) >>> cube[2,1:] array([[7, 7, 7], [8, 8, 8]]) NumPy includes linear algebra functions. Here we perform singular value decomposi- tion on a matrix, an operation used in latent semantic analysis to help identify implicit concepts in a document collection: 4.8 A Sample of Python Libraries | 171 >>> from numpy import linalg >>> a=array([[4,0], [3,-5]]) >>> u,s,vt = linalg.svd(a) >>> u array([[-0.4472136 , -0.89442719], [-0.89442719, 0.4472136 ]]) >>> s array([ 6.32455532, 3.16227766]) >>> vt array([[-0.70710678, 0.70710678], [-0.70710678, -0.70710678]]) NLTK’s clustering package nltk.cluster makes extensive use of NumPy arrays, and includes support for k-means clustering, Gaussian EM clustering, group average agglomerative clustering, and dendogram plots. For details, type help(nltk.cluster). Other Python Libraries There are many other Python libraries, and you can search for them with the help of the Python Package Index at http://pypi.python.org/. Many libraries provide an interface to external software, such as relational databases (e.g., mysql-python) and large docu- ment collections (e.g., PyLucene). Many other libraries give access to file formats such as PDF, MSWord, and XML (pypdf, pywin32, xml.etree), RSS feeds (e.g., feedparser), and electronic mail (e.g., imaplib, email). 4.9 Summary • Python’s assignment and parameter passing use object references; e.g., if a is a list and we assign b = a, then any operation on a will modify b, and vice versa. • The is operation tests whether two objects are identical internal objects, whereas == tests whether two objects are equivalent. This distinction parallels the type- token distinction. • Strings, lists, and tuples are different kinds of sequence object, supporting common operations such as indexing, slicing, len(), sorted(), and membership testing using in. • We can write text to a file by opening the file for writing ofile = open('output.txt', 'w' then adding content to the file ofile.write("Monty Python"), and finally closing the file ofile.close(). • A declarative programming style usually produces more compact, readable code; manually incremented loop variables are usually unnecessary. When a sequence must be enumerated, use enumerate(). • Functions are an essential programming abstraction: key concepts to understand are parameter passing, variable scope, and docstrings. 172 | Chapter 4: Writing Structured Programs • A function serves as a namespace: names defined inside a function are not visible outside that function, unless those names are declared to be global. • Modules permit logically related material to be localized in a file. A module serves as a namespace: names defined in a module—such as variables and functions— are not visible to other modules, unless those names are imported. • Dynamic programming is an algorithm design technique used widely in NLP that stores the results of previous computations in order to avoid unnecessary recomputation. 4.10 Further Reading This chapter has touched on many topics in programming, some specific to Python, and some quite general. We’ve just scratched the surface, and you may want to read more about these topics, starting with the further materials for this chapter available at http://www.nltk.org/. The Python website provides extensive documentation. It is important to understand the built-in functions and standard types, described at http://docs.python.org/library/ functions.html and http://docs.python.org/library/stdtypes.html. We have learned about generators and their importance for efficiency; for information about iterators, a closely related topic, see http://docs.python.org/library/itertools.html. Consult your favorite Py- thon book for more information on such topics. An excellent resource for using Python for multimedia processing, including working with sound files, is (Guzdial, 2005). When using the online Python documentation, be aware that your installed version might be different from the version of the documentation you are reading. You can easily check what version you have, with import sys; sys.version. Version-specific documentation is available at http://www.python.org/doc/versions/. Algorithm design is a rich field within computer science. Some good starting points are (Harel, 2004), (Levitin, 2004), and (Knuth, 2006). Useful guidance on the practice of software development is provided in (Hunt & Thomas, 2000) and (McConnell, 2004). 4.11 Exercises 1. ○ Find out more about sequence objects using Python’s help facility. In the inter- preter, type help(str), help(list), and help(tuple). This will give you a full list of the functions supported by each type. Some functions have special names flanked with underscores; as the help documentation shows, each such function corre- sponds to something more familiar. For example x.__getitem__(y) is just a long- winded way of saying x[y]. 2. ○ Identify three operations that can be performed on both tuples and lists. Identify three list operations that cannot be performed on tuples. Name a context where using a list instead of a tuple generates a Python error. 4.11 Exercises | 173 3. ○ Find out how to create a tuple consisting of a single item. There are at least two ways to do this. 4. ○ Create a list words = ['is', 'NLP', 'fun', '?']. Use a series of assignment statements (e.g., words[1] = words[2]) and a temporary variable tmp to transform this list into the list ['NLP', 'is', 'fun', '!']. Now do the same transformation using tuple assignment. 5. ○ Read about the built-in comparison function cmp, by typing help(cmp). How does it differ in behavior from the comparison operators? 6. ○ Does the method for creating a sliding window of n-grams behave correctly for the two limiting cases: n = 1 and n = len(sent)? 7. ○ We pointed out that when empty strings and empty lists occur in the condition part of an if clause, they evaluate to False. In this case, they are said to be occurring in a Boolean context. Experiment with different kinds of non-Boolean expressions in Boolean contexts, and see whether they evaluate as True or False. 8. ○ Use the inequality operators to compare strings, e.g., 'Monty' < 'Python'. What happens when you do 'Z' < 'a'? Try pairs of strings that have a common prefix, e.g., 'Monty' < 'Montague'. Read up on “lexicographical sort” in order to under- stand what is going on here. Try comparing structured objects, e.g., ('Monty', 1) < ('Monty', 2). Does this behave as expected? 9. ○ Write code that removes whitespace at the beginning and end of a string, and normalizes whitespace between words to be a single-space character. a. Do this task using split() and join(). b. Do this task using regular expression substitutions. 10. ○ Write a program to sort words by length. Define a helper function cmp_len which uses the cmp comparison function on word lengths. 11. ◑ Create a list of words and store it in a variable sent1. Now assign sent2 = sent1. Modify one of the items in sent1 and verify that sent2 has changed. a. Now try the same exercise, but instead assign sent2 = sent1[:]. Modify sent1 again and see what happens to sent2. Explain. b. Now define text1 to be a list of lists of strings (e.g., to represent a text consisting of multiple sentences). Now assign text2 = text1[:], assign a new value to one of the words, e.g., text1[1][1] = 'Monty'. Check what this did to text2. Explain. c. Load Python’s deepcopy() function (i.e., from copy import deepcopy), consult its documentation, and test that it makes a fresh copy of any object. 12. ◑ Initialize an n-by-m list of lists of empty strings using list multiplication, e.g., word_table = [[''] * n] * m. What happens when you set one of its values, e.g., word_table[1][2] = "hello"? Explain why this happens. Now write an expression using range() to construct a list of lists, and show that it does not have this problem. 174 | Chapter 4: Writing Structured Programs 13. ◑ Write code to initialize a two-dimensional array of sets called word_vowels and process a list of words, adding each word to word_vowels[l][v] where l is the length of the word and v is the number of vowels it contains. 14. ◑ Write a function novel10(text) that prints any word that appeared in the last 10% of a text that had not been encountered earlier. 15. ◑ Write a program that takes a sentence expressed as a single string, splits it, and counts up the words. Get it to print out each word and the word’s frequency, one per line, in alphabetical order. 16. ◑ Read up on Gematria, a method for assigning numbers to words, and for mapping between words having the same number to discover the hidden meaning of texts (http://en.wikipedia.org/wiki/Gematria, http://essenes.net/gemcal.htm). a. Write a function gematria() that sums the numerical values of the letters of a word, according to the letter values in letter_vals: >>> letter_vals = {'a':1, 'b':2, 'c':3, 'd':4, 'e':5, 'f':80, 'g':3, 'h':8, ... 'i':10, 'j':10, 'k':20, 'l':30, 'm':40, 'n':50, 'o':70, 'p':80, 'q':100, ... 'r':200, 's':300, 't':400, 'u':6, 'v':6, 'w':800, 'x':60, 'y':10, 'z':7} b. Process a corpus (e.g., nltk.corpus.state_union) and for each document, count how many of its words have the number 666. c. Write a function decode() to process a text, randomly replacing words with their Gematria equivalents, in order to discover the “hidden meaning” of the text. 17. ◑ Write a function shorten(text, n) to process a text, omitting the n most fre- quently occurring words of the text. How readable is it? 18. ◑ Write code to print out an index for a lexicon, allowing someone to look up words according to their meanings (or their pronunciations; whatever properties are contained in the lexical entries). 19. ◑ Write a list comprehension that sorts a list of WordNet synsets for proximity to a given synset. For example, given the synsets minke_whale.n.01, orca.n.01, novel.n.01, and tortoise.n.01, sort them according to their path_distance() from right_whale.n.01. 20. ◑ Write a function that takes a list of words (containing duplicates) and returns a list of words (with no duplicates) sorted by decreasing frequency. E.g., if the input list contained 10 instances of the word table and 9 instances of the word chair, then table would appear before chair in the output list. 21. ◑ Write a function that takes a text and a vocabulary as its arguments and returns the set of words that appear in the text but not in the vocabulary. Both arguments can be represented as lists of strings. Can you do this in a single line, using set.dif ference()? 22. ◑ Import the itemgetter() function from the operator module in Python’s standard library (i.e., from operator import itemgetter). Create a list words containing sev- 4.11 Exercises | 175 eral words. Now try calling: sorted(words, key=itemgetter(1)), and sor ted(words, key=itemgetter(-1)). Explain what itemgetter() is doing. 23. ◑ Write a recursive function lookup(trie, key) that looks up a key in a trie, and returns the value it finds. Extend the function to return a word when it is uniquely determined by its prefix (e.g., vanguard is the only word that starts with vang-, so lookup(trie, 'vang') should return the same thing as lookup(trie, 'vanguard')). 24. ◑ Read up on “keyword linkage” (Chapter 5 of (Scott & Tribble, 2006)). Extract keywords from NLTK’s Shakespeare Corpus and using the NetworkX package, plot keyword linkage networks. 25. ◑ Read about string edit distance and the Levenshtein Algorithm. Try the imple- mentation provided in nltk.edit_dist(). In what way is this using dynamic pro- gramming? Does it use the bottom-up or top-down approach? (See also http:// norvig.com/spell-correct.html.) 26. ◑ The Catalan numbers arise in many applications of combinatorial mathematics, including the counting of parse trees (Section 8.6). The series can be defined as follows: C0 = 1, and Cn+1 = Σ0..n (CiCn-i). a. Write a recursive function to compute nth Catalan number Cn. b. Now write another function that does this computation using dynamic pro- gramming. c. Use the timeit module to compare the performance of these functions as n increases. 27. ● Reproduce some of the results of (Zhao & Zobel, 2007) concerning authorship identification. 28. ● Study gender-specific lexical choice, and see if you can reproduce some of the results of http://www.clintoneast.com/articles/words.php. 29. ● Write a recursive function that pretty prints a trie in alphabetically sorted order, for example: chair: 'flesh' ---t: 'cat' --ic: 'stylish' ---en: 'dog' 30. ● With the help of the trie data structure, write a recursive function that processes text, locating the uniqueness point in each word, and discarding the remainder of each word. How much compression does this give? How readable is the resulting text? 31. ● Obtain some raw text, in the form of a single, long string. Use Python’s text wrap module to break it up into multiple lines. Now write code to add extra spaces between words, in order to justify the output. Each line must have the same width, and spaces must be approximately evenly distributed across each line. No line can begin or end with a space. 176 | Chapter 4: Writing Structured Programs 32. ● Develop a simple extractive summarization tool, that prints the sentences of a document which contain the highest total word frequency. Use FreqDist() to count word frequencies, and use sum to sum the frequencies of the words in each sentence. Rank the sentences according to their score. Finally, print the n highest-scoring sentences in document order. Carefully review the design of your program, especially your approach to this double sorting. Make sure the program is written as clearly as possible. 33. ● Develop your own NgramTagger class that inherits from NLTK’s class, and which encapsulates the method of collapsing the vocabulary of the tagged training and testing data that was described in Chapter 5. Make sure that the unigram and default backoff taggers have access to the full vocabulary. 34. ● Read the following article on semantic orientation of adjectives. Use the Net- workX package to visualize a network of adjectives with edges to indicate same versus different semantic orientation (see http://www.aclweb.org/anthology/P97 -1023). 35. ● Design an algorithm to find the “statistically improbable phrases” of a document collection (see http://www.amazon.com/gp/search-inside/sipshelp.html). 36. ● Write a program to implement a brute-force algorithm for discovering word squares, a kind of n × n: crossword in which the entry in the nth row is the same as the entry in the nth column. For discussion, see http://itre.cis.upenn.edu/~myl/ languagelog/archives/002679.html. 4.11 Exercises | 177 CHAPTER 5 Categorizing and Tagging Words Back in elementary school you learned the difference between nouns, verbs, adjectives, and adverbs. These “word classes” are not just the idle invention of grammarians, but are useful categories for many language processing tasks. As we will see, they arise from simple analysis of the distribution of words in text. The goal of this chapter is to answer the following questions: 1. What are lexical categories, and how are they used in natural language processing? 2. What is a good Python data structure for storing words and their categories? 3. How can we automatically tag each word of a text with its word class? Along the way, we’ll cover some fundamental techniques in NLP, including sequence labeling, n-gram models, backoff, and evaluation. These techniques are useful in many areas, and tagging gives us a simple context in which to present them. We will also see how tagging is the second step in the typical NLP pipeline, following tokenization. The process of classifying words into their parts-of-speech and labeling them accord- ingly is known as part-of-speech tagging, POS tagging, or simply tagging. Parts- of-speech are also known as word classes or lexical categories. The collection of tags used for a particular task is known as a tagset. Our emphasis in this chapter is on exploiting tags, and tagging text automatically. 5.1 Using a Tagger A part-of-speech tagger, or POS tagger, processes a sequence of words, and attaches a part of speech tag to each word (don’t forget to import nltk): >>> text = nltk.word_tokenize("And now for something completely different") >>> nltk.pos_tag(text) [('And', 'CC'), ('now', 'RB'), ('for', 'IN'), ('something', 'NN'), ('completely', 'RB'), ('different', 'JJ')] Here we see that and is CC, a coordinating conjunction; now and completely are RB, or adverbs; for is IN, a preposition; something is NN, a noun; and different is JJ, an adjective. 179 NLTK provides documentation for each tag, which can be queried using the tag, e.g., nltk.help.upenn_tagset('RB'), or a regular expression, e.g., nltk.help.upenn_brown_tagset('NN.*'). Some corpora have RE- ADME files with tagset documentation; see nltk.name.readme(), sub- stituting in the name of the corpus. Let’s look at another example, this time including some homonyms: >>> text = nltk.word_tokenize("They refuse to permit us to obtain the refuse permit") >>> nltk.pos_tag(text) [('They', 'PRP'), ('refuse', 'VBP'), ('to', 'TO'), ('permit', 'VB'), ('us', 'PRP'), ('to', 'TO'), ('obtain', 'VB'), ('the', 'DT'), ('refuse', 'NN'), ('permit', 'NN')] Notice that refuse and permit both appear as a present tense verb (VBP) and a noun (NN). E.g., refUSE is a verb meaning “deny,” while REFuse is a noun meaning “trash” (i.e., they are not homophones). Thus, we need to know which word is being used in order to pronounce the text correctly. (For this reason, text-to-speech systems usually perform POS tagging.) Your Turn: Many words, like ski and race, can be used as nouns or verbs with no difference in pronunciation. Can you think of others? Hint: think of a commonplace object and try to put the word to before it to see if it can also be a verb, or think of an action and try to put the before it to see if it can also be a noun. Now make up a sentence with both uses of this word, and run the POS tagger on this sentence. Lexical categories like “noun” and part-of-speech tags like NN seem to have their uses, but the details will be obscure to many readers. You might wonder what justification there is for introducing this extra level of information. Many of these categories arise from superficial analysis of the distribution of words in text. Consider the following analysis involving woman (a noun), bought (a verb), over (a preposition), and the (a determiner). The text.similar() method takes a word w, finds all contexts w1w w2, then finds all words w' that appear in the same context, i.e. w1w'w2. >>> text = nltk.Text(word.lower() for word in nltk.corpus.brown.words()) >>> text.similar('woman') Building word-context index... man time day year car moment world family house country child boy state job way war girl place room word >>> text.similar('bought') made said put done seen had found left given heard brought got been was set told took in felt that >>> text.similar('over') in on to of and for with from at by that into as up out down through is all about >>> text.similar('the') a his this their its her an that our any all one these my in your no some other and 180 | Chapter 5: Categorizing and Tagging Words Observe that searching for woman finds nouns; searching for bought mostly finds verbs; searching for over generally finds prepositions; searching for the finds several deter- miners. A tagger can correctly identify the tags on these words in the context of a sentence, e.g., The woman bought over $150,000 worth of clothes. A tagger can also model our knowledge of unknown words; for example, we can guess that scrobbling is probably a verb, with the root scrobble, and likely to occur in contexts like he was scrobbling. 5.2 Tagged Corpora Representing Tagged Tokens By convention in NLTK, a tagged token is represented using a tuple consisting of the token and the tag. We can create one of these special tuples from the standard string representation of a tagged token, using the function str2tuple(): >>> tagged_token = nltk.tag.str2tuple('fly/NN') >>> tagged_token ('fly', 'NN') >>> tagged_token[0] 'fly' >>> tagged_token[1] 'NN' We can construct a list of tagged tokens directly from a string. The first step is to tokenize the string to access the individual word/tag strings, and then to convert each of these into a tuple (using str2tuple()). >>> sent = ''' ... The/AT grand/JJ jury/NN commented/VBD on/IN a/AT number/NN of/IN ... other/AP topics/NNS ,/, AMONG/IN them/PPO the/AT Atlanta/NP and/CC ... Fulton/NP-tl County/NN-tl purchasing/VBG departments/NNS which/WDT it/PPS ... said/VBD ``/`` ARE/BER well/QL operated/VBN and/CC follow/VB generally/RB ... accepted/VBN practices/NNS which/WDT inure/VB to/IN the/AT best/JJT ... interest/NN of/IN both/ABX governments/NNS ''/'' ./. ... ''' >>> [nltk.tag.str2tuple(t) for t in sent.split()] [('The', 'AT'), ('grand', 'JJ'), ('jury', 'NN'), ('commented', 'VBD'), ('on', 'IN'), ('a', 'AT'), ('number', 'NN'), ... ('.', '.')] Reading Tagged Corpora Several of the corpora included with NLTK have been tagged for their part-of-speech. Here’s an example of what you might see if you opened a file from the Brown Corpus with a text editor: The/at Fulton/np-tl County/nn-tl Grand/jj-tl Jury/nn-tl said/vbd Friday/nr an/at inves- tigation/nn of/in Atlanta’s/np$ recent/jj primary/nn election/nn produced/vbd / no/at evidence/nn ''/'' that/cs any/dti irregularities/nns took/vbd place/nn ./. 5.2 Tagged Corpora | 181 Other corpora use a variety of formats for storing part-of-speech tags. NLTK’s corpus readers provide a uniform interface so that you don’t have to be concerned with the different file formats. In contrast with the file extract just shown, the corpus reader for the Brown Corpus represents the data as shown next. Note that part-of-speech tags have been converted to uppercase; this has become standard practice since the Brown Corpus was published. >>> nltk.corpus.brown.tagged_words() [('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'), ...] >>> nltk.corpus.brown.tagged_words(simplify_tags=True) [('The', 'DET'), ('Fulton', 'N'), ('County', 'N'), ...] Whenever a corpus contains tagged text, the NLTK corpus interface will have a tagged_words() method. Here are some more examples, again using the output format illustrated for the Brown Corpus: >>> print nltk.corpus.nps_chat.tagged_words() [('now', 'RB'), ('im', 'PRP'), ('left', 'VBD'), ...] >>> nltk.corpus.conll2000.tagged_words() [('Confidence', 'NN'), ('in', 'IN'), ('the', 'DT'), ...] >>> nltk.corpus.treebank.tagged_words() [('Pierre', 'NNP'), ('Vinken', 'NNP'), (',', ','), ...] Not all corpora employ the same set of tags; see the tagset help functionality and the readme() methods mentioned earlier for documentation. Initially we want to avoid the complications of these tagsets, so we use a built-in mapping to a simplified tagset: >>> nltk.corpus.brown.tagged_words(simplify_tags=True) [('The', 'DET'), ('Fulton', 'NP'), ('County', 'N'), ...] >>> nltk.corpus.treebank.tagged_words(simplify_tags=True) [('Pierre', 'NP'), ('Vinken', 'NP'), (',', ','), ...] Tagged corpora for several other languages are distributed with NLTK, including Chi- nese, Hindi, Portuguese, Spanish, Dutch, and Catalan. These usually contain non- ASCII text, and Python always displays this in hexadecimal when printing a larger structure such as a list. >>> nltk.corpus.sinica_treebank.tagged_words() [('\xe4\xb8\x80', 'Neu'), ('\xe5\x8f\x8b\xe6\x83\x85', 'Nad'), ...] >>> nltk.corpus.indian.tagged_words() [('\xe0\xa6\xae\xe0\xa6\xb9\xe0\xa6\xbf\xe0\xa6\xb7\xe0\xa7\x87\xe0\xa6\xb0', 'NN'), ('\xe0\xa6\xb8\xe0\xa6\xa8\xe0\xa7\x8d\xe0\xa6\xa4\xe0\xa6\xbe\xe0\xa6\xa8', 'NN'), ...] >>> nltk.corpus.mac_morpho.tagged_words() [('Jersei', 'N'), ('atinge', 'V'), ('m\xe9dia', 'N'), ...] >>> nltk.corpus.conll2002.tagged_words() [('Sao', 'NC'), ('Paulo', 'VMI'), ('(', 'Fpa'), ...] >>> nltk.corpus.cess_cat.tagged_words() [('El', 'da0ms0'), ('Tribunal_Suprem', 'np0000o'), ...] If your environment is set up correctly, with appropriate editors and fonts, you should be able to display individual strings in a human-readable way. For example, Fig- ure 5-1 shows data accessed using nltk.corpus.indian. 182 | Chapter 5: Categorizing and Tagging Words If the corpus is also segmented into sentences, it will have a tagged_sents() method that divides up the tagged words into sentences rather than presenting them as one big list. This will be useful when we come to developing automatic taggers, as they are trained and tested on lists of sentences, not words. A Simplified Part-of-Speech Tagset Tagged corpora use many different conventions for tagging words. To help us get star- ted, we will be looking at a simplified tagset (shown in Table 5-1). Table 5-1. Simplified part-of-speech tagset Tag Meaning Examples ADJ adjective new, good, high, special, big, local ADV adverb really, already, still, early, now CNJ conjunction and, or, but, if, while, although DET determiner the, a, some, most, every, no EX existential there, there’s FW foreign word dolce, ersatz, esprit, quo, maitre MOD modal verb will, can, would, may, must, should N noun year, home, costs, time, education NP proper noun Alison, Africa, April, Washington NUM number twenty-four, fourth, 1991, 14:24 PRO pronoun he, their, her, its, my, I, us P preposition on, of, at, with, by, into, under TO the word to to UH interjection ah, bang, ha, whee, hmpf, oops V verb is, has, get, do, make, see, run VD past tense said, took, told, made, asked VG present participle making, going, playing, working VN past participle given, taken, begun, sung WH wh determiner who, which, when, what, where, how 5.2 Tagged Corpora | 183 Figure 5-1. POS tagged data from four Indian languages: Bangla, Hindi, Marathi, and Telugu. Let’s see which of these tags are the most common in the news category of the Brown Corpus: >>> from nltk.corpus import brown >>> brown_news_tagged = brown.tagged_words(categories='news', simplify_tags=True) >>> tag_fd = nltk.FreqDist(tag for (word, tag) in brown_news_tagged) >>> tag_fd.keys() ['N', 'P', 'DET', 'NP', 'V', 'ADJ', ',', '.', 'CNJ', 'PRO', 'ADV', 'VD', ...] Your Turn: Plot the frequency distribution just shown using tag_fd.plot(cumulative=True). What percentage of words are tagged using the first five tags of the above list? We can use these tags to do powerful searches using a graphical POS-concordance tool nltk.app.concordance(). Use it to search for any combination of words and POS tags, e.g., N N N N, hit/VD, hit/VN, or the ADJ man. Nouns Nouns generally refer to people, places, things, or concepts, e.g., woman, Scotland, book, intelligence. Nouns can appear after determiners and adjectives, and can be the subject or object of the verb, as shown in Table 5-2. Table 5-2. Syntactic patterns involving some nouns Word After a determiner Subject of the verb woman the woman who I saw yesterday ... the woman sat down Scotland the Scotland I remember as a child ... Scotland has five million people book the book I bought yesterday ... this book recounts the colonization of Australia intelligence the intelligence displayed by the child ... Mary’s intelligence impressed her teachers The simplified noun tags are N for common nouns like book, and NP for proper nouns like Scotland. 184 | Chapter 5: Categorizing and Tagging Words Let’s inspect some tagged text to see what parts-of-speech occur before a noun, with the most frequent ones first. To begin with, we construct a list of bigrams whose mem- bers are themselves word-tag pairs, such as (('The', 'DET'), ('Fulton', 'NP')) and (('Fulton', 'NP'), ('County', 'N')). Then we construct a FreqDist from the tag parts of the bigrams. >>> word_tag_pairs = nltk.bigrams(brown_news_tagged) >>> list(nltk.FreqDist(a[1] for (a, b) in word_tag_pairs if b[1] == 'N')) ['DET', 'ADJ', 'N', 'P', 'NP', 'NUM', 'V', 'PRO', 'CNJ', '.', ',', 'VG', 'VN', ...] This confirms our assertion that nouns occur after determiners and adjectives, includ- ing numeral adjectives (tagged as NUM). Verbs Verbs are words that describe events and actions, e.g., fall and eat, as shown in Ta- ble 5-3. In the context of a sentence, verbs typically express a relation involving the referents of one or more noun phrases. Table 5-3. Syntactic patterns involving some verbs Word Simple With modifiers and adjuncts (italicized) fall Rome fell Dot com stocks suddenly fell like a stone eat Mice eat cheese John ate the pizza with gusto What are the most common verbs in news text? Let’s sort all the verbs by frequency: >>> wsj = nltk.corpus.treebank.tagged_words(simplify_tags=True) >>> word_tag_fd = nltk.FreqDist(wsj) >>> [word + "/" + tag for (word, tag) in word_tag_fd if tag.startswith('V')] ['is/V', 'said/VD', 'was/VD', 'are/V', 'be/V', 'has/V', 'have/V', 'says/V', 'were/VD', 'had/VD', 'been/VN', "'s/V", 'do/V', 'say/V', 'make/V', 'did/VD', 'rose/VD', 'does/V', 'expected/VN', 'buy/V', 'take/V', 'get/V', 'sell/V', 'help/V', 'added/VD', 'including/VG', 'according/VG', 'made/VN', 'pay/V', ...] Note that the items being counted in the frequency distribution are word-tag pairs. Since words and tags are paired, we can treat the word as a condition and the tag as an event, and initialize a conditional frequency distribution with a list of condition-event pairs. This lets us see a frequency-ordered list of tags given a word: >>> cfd1 = nltk.ConditionalFreqDist(wsj) >>> cfd1['yield'].keys() ['V', 'N'] >>> cfd1['cut'].keys() ['V', 'VD', 'N', 'VN'] We can reverse the order of the pairs, so that the tags are the conditions, and the words are the events. Now we can see likely words for a given tag: 5.2 Tagged Corpora | 185 >>> cfd2 = nltk.ConditionalFreqDist((tag, word) for (word, tag) in wsj) >>> cfd2['VN'].keys() ['been', 'expected', 'made', 'compared', 'based', 'priced', 'used', 'sold', 'named', 'designed', 'held', 'fined', 'taken', 'paid', 'traded', 'said', ...] To clarify the distinction between VD (past tense) and VN (past participle), let’s find words that can be both VD and VN, and see some surrounding text: >>> [w for w in cfd1.conditions() if 'VD' in cfd1[w] and 'VN' in cfd1[w]] ['Asked', 'accelerated', 'accepted', 'accused', 'acquired', 'added', 'adopted', ...] >>> idx1 = wsj.index(('kicked', 'VD')) >>> wsj[idx1-4:idx1+1] [('While', 'P'), ('program', 'N'), ('trades', 'N'), ('swiftly', 'ADV'), ('kicked', 'VD')] >>> idx2 = wsj.index(('kicked', 'VN')) >>> wsj[idx2-4:idx2+1] [('head', 'N'), ('of', 'P'), ('state', 'N'), ('has', 'V'), ('kicked', 'VN')] In this case, we see that the past participle of kicked is preceded by a form of the auxiliary verb have. Is this generally true? Your Turn: Given the list of past participles specified by cfd2['VN'].keys(), try to collect a list of all the word-tag pairs that im- mediately precede items in that list. Adjectives and Adverbs Two other important word classes are adjectives and adverbs. Adjectives describe nouns, and can be used as modifiers (e.g., large in the large pizza), or as predicates (e.g., the pizza is large). English adjectives can have internal structure (e.g., fall+ing in the falling stocks). Adverbs modify verbs to specify the time, manner, place, or direction of the event described by the verb (e.g., quickly in the stocks fell quickly). Adverbs may also modify adjectives (e.g., really in Mary’s teacher was really nice). English has several categories of closed class words in addition to prepositions, such as articles (also often called determiners) (e.g., the, a), modals (e.g., should, may), and personal pronouns (e.g., she, they). Each dictionary and grammar classifies these words differently. Your Turn: If you are uncertain about some of these parts-of-speech, study them using nltk.app.concordance(), or watch some of the School- house Rock! grammar videos available at YouTube, or consult Sec- tion 5.9. 186 | Chapter 5: Categorizing and Tagging Words Unsimplified Tags Let’s find the most frequent nouns of each noun part-of-speech type. The program in Example 5-1 finds all tags starting with NN, and provides a few example words for each one. You will see that there are many variants of NN; the most important contain $ for possessive nouns, S for plural nouns (since plural nouns typically end in s), and P for proper nouns. In addition, most of the tags have suffix modifiers: -NC for citations, -HL for words in headlines, and -TL for titles (a feature of Brown tags). Example 5-1. Program to find the most frequent noun tags. def findtags(tag_prefix, tagged_text): cfd = nltk.ConditionalFreqDist((tag, word) for (word, tag) in tagged_text if tag.startswith(tag_prefix)) return dict((tag, cfd[tag].keys()[:5]) for tag in cfd.conditions()) >>> tagdict = findtags('NN', nltk.corpus.brown.tagged_words(categories='news')) >>> for tag in sorted(tagdict): ... print tag, tagdict[tag] ... NN ['year', 'time', 'state', 'week', 'man'] NN$ ["year's", "world's", "state's", "nation's", "company's"] NN$-HL ["Golf's", "Navy's"] NN$-TL ["President's", "University's", "League's", "Gallery's", "Army's"] NN-HL ['cut', 'Salary', 'condition', 'Question', 'business'] NN-NC ['eva', 'ova', 'aya'] NN-TL ['President', 'House', 'State', 'University', 'City'] NN-TL-HL ['Fort', 'City', 'Commissioner', 'Grove', 'House'] NNS ['years', 'members', 'people', 'sales', 'men'] NNS$ ["children's", "women's", "men's", "janitors'", "taxpayers'"] NNS$-HL ["Dealers'", "Idols'"] NNS$-TL ["Women's", "States'", "Giants'", "Officers'", "Bombers'"] NNS-HL ['years', 'idols', 'Creations', 'thanks', 'centers'] NNS-TL ['States', 'Nations', 'Masters', 'Rules', 'Communists'] NNS-TL-HL ['Nations'] When we come to constructing part-of-speech taggers later in this chapter, we will use the unsimplified tags. Exploring Tagged Corpora Let’s briefly return to the kinds of exploration of corpora we saw in previous chapters, this time exploiting POS tags. Suppose we’re studying the word often and want to see how it is used in text. We could ask to see the words that follow often: >>> brown_learned_text = brown.words(categories='learned') >>> sorted(set(b for (a, b) in nltk.ibigrams(brown_learned_text) if a == 'often')) [',', '.', 'accomplished', 'analytically', 'appear', 'apt', 'associated', 'assuming', 'became', 'become', 'been', 'began', 'call', 'called', 'carefully', 'chose', ...] However, it’s probably more instructive use the tagged_words() method to look at the part-of-speech tag of the following words: 5.2 Tagged Corpora | 187 >>> brown_lrnd_tagged = brown.tagged_words(categories='learned', simplify_tags=True) >>> tags = [b[1] for (a, b) in nltk.ibigrams(brown_lrnd_tagged) if a[0] == 'often'] >>> fd = nltk.FreqDist(tags) >>> fd.tabulate() VN V VD DET ADJ ADV P CNJ , TO VG WH VBZ . 15 12 8 5 5 4 4 3 3 1 1 1 1 1 Notice that the most high-frequency parts-of-speech following often are verbs. Nouns never appear in this position (in this particular corpus). Next, let’s look at some larger context, and find words involving particular sequences of tags and words (in this case " to "). In Example 5-2, we consider each three-word window in the sentence , and check whether they meet our criterion . If the tags match, we print the corresponding words . Example 5-2. Searching for three-word phrases using POS tags. from nltk.corpus import brown def process(sentence): for (w1,t1), (w2,t2), (w3,t3) in nltk.trigrams(sentence): if (t1.startswith('V') and t2 == 'TO' and t3.startswith('V')): print w1, w2, w3 >>> for tagged_sent in brown.tagged_sents(): ... process(tagged_sent) ... combined to achieve continue to place serve to protect wanted to wait allowed to place expected to become ... Finally, let’s look for words that are highly ambiguous as to their part-of-speech tag. Understanding why such words are tagged as they are in each context can help us clarify the distinctions between the tags. >>> brown_news_tagged = brown.tagged_words(categories='news', simplify_tags=True) >>> data = nltk.ConditionalFreqDist((word.lower(), tag) ... for (word, tag) in brown_news_tagged) >>> for word in data.conditions(): ... if len(data[word]) > 3: ... tags = data[word].keys() ... print word, ' '.join(tags) ... best ADJ ADV NP V better ADJ ADV V DET close ADV ADJ V N cut V N VN VD even ADV DET ADJ V grant NP N V - hit V VD VN N lay ADJ V NP VD left VD ADJ N VN 188 | Chapter 5: Categorizing and Tagging Words like CNJ V ADJ P - near P ADV ADJ DET open ADJ V N ADV past N ADJ DET P present ADJ ADV V N read V VN VD NP right ADJ N DET ADV second NUM ADV DET N set VN V VD N - that CNJ V WH DET Your Turn: Open the POS concordance tool nltk.app.concordance() and load the complete Brown Corpus (simplified tagset). Now pick some of the words listed at the end of the previous code example and see how the tag of the word correlates with the context of the word. E.g., search for near to see all forms mixed together, near/ADJ to see it used as an adjective, near N to see just those cases where a noun follows, and so forth. 5.3 Mapping Words to Properties Using Python Dictionaries As we have seen, a tagged word of the form (word, tag) is an association between a word and a part-of-speech tag. Once we start doing part-of-speech tagging, we will be creating programs that assign a tag to a word, the tag which is most likely in a given context. We can think of this process as mapping from words to tags. The most natural way to store mappings in Python uses the so-called dictionary data type (also known as an associative array or hash array in other programming languages). In this sec- tion, we look at dictionaries and see how they can represent a variety of language in- formation, including parts-of-speech. Indexing Lists Versus Dictionaries A text, as we have seen, is treated in Python as a list of words. An important property of lists is that we can “look up” a particular item by giving its index, e.g., text1[100]. Notice how we specify a number and get back a word. We can think of a list as a simple kind of table, as shown in Figure 5-2. Figure 5-2. List lookup: We access the contents of a Python list with the help of an integer index. 5.3 Mapping Words to Properties Using Python Dictionaries | 189 Contrast this situation with frequency distributions (Section 1.3), where we specify a word and get back a number, e.g., fdist['monstrous'], which tells us the number of times a given word has occurred in a text. Lookup using words is familiar to anyone who has used a dictionary. Some more examples are shown in Figure 5-3. Figure 5-3. Dictionary lookup: we access the entry of a dictionary using a key such as someone’s name, a web domain, or an English word; other names for dictionary are map, hashmap, hash, and associative array. In the case of a phonebook, we look up an entry using a name and get back a number. When we type a domain name in a web browser, the computer looks this up to get back an IP address. A word frequency table allows us to look up a word and find its frequency in a text collection. In all these cases, we are mapping from names to num- bers, rather than the other way around as with a list. In general, we would like to be able to map between arbitrary types of information. Table 5-4 lists a variety of linguistic objects, along with what they map. Table 5-4. Linguistic objects as mappings from keys to values Linguistic object Maps from Maps to Document Index Word List of pages (where word is found) Thesaurus Word sense List of synonyms Dictionary Headword Entry (part-of-speech, sense definitions, etymology) Comparative Wordlist Gloss term Cognates (list of words, one per language) Morph Analyzer Surface form Morphological analysis (list of component morphemes) Most often, we are mapping from a “word” to some structured object. For example, a document index maps from a word (which we can represent as a string) to a list of pages (represented as a list of integers). In this section, we will see how to represent such mappings in Python. Dictionaries in Python Python provides a dictionary data type that can be used for mapping between arbitrary types. It is like a conventional dictionary, in that it gives you an efficient way to look things up. However, as we see from Table 5-4, it has a much wider range of uses. 190 | Chapter 5: Categorizing and Tagging Words To illustrate, we define pos to be an empty dictionary and then add four entries to it, specifying the part-of-speech of some words. We add entries to a dictionary using the familiar square bracket notation: >>> pos = {} >>> pos {} >>> pos['colorless'] = 'ADJ' >>> pos {'colorless': 'ADJ'} >>> pos['ideas'] = 'N' >>> pos['sleep'] = 'V' >>> pos['furiously'] = 'ADV' >>> pos {'furiously': 'ADV', 'ideas': 'N', 'colorless': 'ADJ', 'sleep': 'V'} So, for example, says that the part-of-speech of colorless is adjective, or more spe- cifically, that the key 'colorless' is assigned the value 'ADJ' in dictionary pos. When we inspect the value of pos we see a set of key-value pairs. Once we have populated the dictionary in this way, we can employ the keys to retrieve values: >>> pos['ideas'] 'N' >>> pos['colorless'] 'ADJ' Of course, we might accidentally use a key that hasn’t been assigned a value. >>> pos['green'] Traceback (most recent call last): File "", line 1, in ? KeyError: 'green' This raises an important question. Unlike lists and strings, where we can use len() to work out which integers will be legal indexes, how do we work out the legal keys for a dictionary? If the dictionary is not too big, we can simply inspect its contents by eval- uating the variable pos. As we saw earlier in line , this gives us the key-value pairs. Notice that they are not in the same order they were originally entered; this is because dictionaries are not sequences but mappings (see Figure 5-3), and the keys are not inherently ordered. Alternatively, to just find the keys, we can either convert the dictionary to a list or use the dictionary in a context where a list is expected, as the parameter of sorted() or in a for loop . >>> list(pos) ['ideas', 'furiously', 'colorless', 'sleep'] >>> sorted(pos) ['colorless', 'furiously', 'ideas', 'sleep'] >>> [w for w in pos if w.endswith('s')] ['colorless', 'ideas'] 5.3 Mapping Words to Properties Using Python Dictionaries | 191 When you type list(pos), you might see a different order to the one shown here. If you want to see the keys in order, just sort them. As well as iterating over all keys in the dictionary with a for loop, we can use the for loop as we did for printing lists: >>> for word in sorted(pos): ... print word + ":", pos[word] ... colorless: ADJ furiously: ADV sleep: V ideas: N Finally, the dictionary methods keys(), values(), and items() allow us to access the keys, values, and key-value pairs as separate lists. We can even sort tuples , which orders them according to their first element (and if the first elements are the same, it uses their second elements). >>> pos.keys() ['colorless', 'furiously', 'sleep', 'ideas'] >>> pos.values() ['ADJ', 'ADV', 'V', 'N'] >>> pos.items() [('colorless', 'ADJ'), ('furiously', 'ADV'), ('sleep', 'V'), ('ideas', 'N')] >>> for key, val in sorted(pos.items()): ... print key + ":", val ... colorless: ADJ furiously: ADV ideas: N sleep: V We want to be sure that when we look something up in a dictionary, we get only one value for each key. Now suppose we try to use a dictionary to store the fact that the word sleep can be used as both a verb and a noun: >>> pos['sleep'] = 'V' >>> pos['sleep'] 'V' >>> pos['sleep'] = 'N' >>> pos['sleep'] 'N' Initially, pos['sleep'] is given the value 'V'. But this is immediately overwritten with the new value, 'N'. In other words, there can be only one entry in the dictionary for 'sleep'. However, there is a way of storing multiple values in that entry: we use a list value, e.g., pos['sleep'] = ['N', 'V']. In fact, this is what we saw in Section 2.4 for the CMU Pronouncing Dictionary, which stores multiple pronunciations for a single word. 192 | Chapter 5: Categorizing and Tagging Words Defining Dictionaries We can use the same key-value pair format to create a dictionary. There are a couple of ways to do this, and we will normally use the first: >>> pos = {'colorless': 'ADJ', 'ideas': 'N', 'sleep': 'V', 'furiously': 'ADV'} >>> pos = dict(colorless='ADJ', ideas='N', sleep='V', furiously='ADV') Note that dictionary keys must be immutable types, such as strings and tuples. If we try to define a dictionary using a mutable key, we get a TypeError: >>> pos = {['ideas', 'blogs', 'adventures']: 'N'} Traceback (most recent call last): File "", line 1, in TypeError: list objects are unhashable Default Dictionaries If we try to access a key that is not in a dictionary, we get an error. However, it’s often useful if a dictionary can automatically create an entry for this new key and give it a default value, such as zero or the empty list. Since Python 2.5, a special kind of dic- tionary called a defaultdict has been available. (It is provided as nltk.defaultdict for the benefit of readers who are using Python 2.4.) In order to use it, we have to supply a parameter which can be used to create the default value, e.g., int, float, str, list, dict, tuple. >>> frequency = nltk.defaultdict(int) >>> frequency['colorless'] = 4 >>> frequency['ideas'] 0 >>> pos = nltk.defaultdict(list) >>> pos['sleep'] = ['N', 'V'] >>> pos['ideas'] [] These default values are actually functions that convert other objects to the specified type (e.g., int("2"), list("2")). When they are called with no parameter—say, int(), list()—they return 0 and [] respectively. The preceding examples specified the default value of a dictionary entry to be the default value of a particular data type. However, we can specify any default value we like, simply by providing the name of a function that can be called with no arguments to create the required value. Let’s return to our part-of-speech example, and create a dictionary whose default value for any entry is 'N' . When we access a non-existent entry , it is automatically added to the dictionary . >>> pos = nltk.defaultdict(lambda: 'N') >>> pos['colorless'] = 'ADJ' >>> pos['blog'] 'N' 5.3 Mapping Words to Properties Using Python Dictionaries | 193 >>> pos.items() [('blog', 'N'), ('colorless', 'ADJ')] This example used a lambda expression, introduced in Section 4.4. This lambda expression specifies no parameters, so we call it using paren- theses with no arguments. Thus, the following definitions of f and g are equivalent: >>> f = lambda: 'N' >>> f() 'N' >>> def g(): ... return 'N' >>> g() 'N' Let’s see how default dictionaries could be used in a more substantial language pro- cessing task. Many language processing tasks—including tagging—struggle to cor- rectly process the hapaxes of a text. They can perform better with a fixed vocabulary and a guarantee that no new words will appear. We can preprocess a text to replace low-frequency words with a special “out of vocabulary” token, UNK, with the help of a default dictionary. (Can you work out how to do this without reading on?) We need to create a default dictionary that maps each word to its replacement. The most frequent n words will be mapped to themselves. Everything else will be mapped to UNK. >>> alice = nltk.corpus.gutenberg.words('carroll-alice.txt') >>> vocab = nltk.FreqDist(alice) >>> v1000 = list(vocab)[:1000] >>> mapping = nltk.defaultdict(lambda: 'UNK') >>> for v in v1000: ... mapping[v] = v ... >>> alice2 = [mapping[v] for v in alice] >>> alice2[:100] ['UNK', 'Alice', "'", 's', 'Adventures', 'in', 'Wonderland', 'by', 'UNK', 'UNK', 'UNK', 'UNK', 'CHAPTER', 'I', '.', 'UNK', 'the', 'Rabbit', '-', 'UNK', 'Alice', 'was', 'beginning', 'to', 'get', 'very', 'tired', 'of', 'sitting', 'by', 'her', 'sister', 'on', 'the', 'bank', ',', 'and', 'of', 'having', 'nothing', 'to', 'do', ':', 'once', 'or', 'twice', 'she', 'had', 'UNK', 'into', 'the', 'book', 'her', 'sister', 'was', 'UNK', ',', 'but', 'it', 'had', 'no', 'pictures', 'or', 'UNK', 'in', 'it', ',', "'", 'and', 'what', 'is', 'the', 'use', 'of', 'a', 'book', ",'", 'thought', 'Alice', "'", 'without', 'pictures', 'or', 'conversation', "?'", ...] >>> len(set(alice2)) 1001 Incrementally Updating a Dictionary We can employ dictionaries to count occurrences, emulating the method for tallying words shown in Figure 1-3. We begin by initializing an empty defaultdict, then process each part-of-speech tag in the text. If the tag hasn’t been seen before, it will have a zero 194 | Chapter 5: Categorizing and Tagging Words count by default. Each time we encounter a tag, we increment its count using the += operator (see Example 5-3). Example 5-3. Incrementally updating a dictionary, and sorting by value. >>> counts = nltk.defaultdict(int) >>> from nltk.corpus import brown >>> for (word, tag) in brown.tagged_words(categories='news'): ... counts[tag] += 1 ... >>> counts['N'] 22226 >>> list(counts) ['FW', 'DET', 'WH', "''", 'VBZ', 'VB+PPO', "'", ')', 'ADJ', 'PRO', '*', '-', ...] >>> from operator import itemgetter >>> sorted(counts.items(), key=itemgetter(1), reverse=True) [('N', 22226), ('P', 10845), ('DET', 10648), ('NP', 8336), ('V', 7313), ...] >>> [t for t, c in sorted(counts.items(), key=itemgetter(1), reverse=True)] ['N', 'P', 'DET', 'NP', 'V', 'ADJ', ',', '.', 'CNJ', 'PRO', 'ADV', 'VD', ...] The listing in Example 5-3 illustrates an important idiom for sorting a dictionary by its values, to show words in decreasing order of frequency. The first parameter of sorted() is the items to sort, which is a list of tuples consisting of a POS tag and a frequency. The second parameter specifies the sort key using a function itemget ter(). In general, itemgetter(n) returns a function that can be called on some other sequence object to obtain the nth element: >>> pair = ('NP', 8336) >>> pair[1] 8336 >>> itemgetter(1)(pair) 8336 The last parameter of sorted() specifies that the items should be returned in reverse order, i.e., decreasing values of frequency. There’s a second useful programming idiom at the beginning of Example 5-3, where we initialize a defaultdict and then use a for loop to update its values. Here’s a sche- matic version: >>> my_dictionary = nltk.defaultdict(function to create default value) >>> for item in sequence: ... my_dictionary[item_key] is updated with information about item Here’s another instance of this pattern, where we index words according to their last two letters: >>> last_letters = nltk.defaultdict(list) >>> words = nltk.corpus.words.words('en') >>> for word in words: ... key = word[-2:] ... last_letters[key].append(word) ... 5.3 Mapping Words to Properties Using Python Dictionaries | 195 >>> last_letters['ly'] ['abactinally', 'abandonedly', 'abasedly', 'abashedly', 'abashlessly', 'abbreviately', 'abdominally', 'abhorrently', 'abidingly', 'abiogenetically', 'abiologically', ...] >>> last_letters['zy'] ['blazy', 'bleezy', 'blowzy', 'boozy', 'breezy', 'bronzy', 'buzzy', 'Chazy', ...] The following example uses the same pattern to create an anagram dictionary. (You might experiment with the third line to get an idea of why this program works.) >>> anagrams = nltk.defaultdict(list) >>> for word in words: ... key = ''.join(sorted(word)) ... anagrams[key].append(word) ... >>> anagrams['aeilnrt'] ['entrail', 'latrine', 'ratline', 'reliant', 'retinal', 'trenail'] Since accumulating words like this is such a common task, NLTK provides a more convenient way of creating a defaultdict(list), in the form of nltk.Index(): >>> anagrams = nltk.Index((''.join(sorted(w)), w) for w in words) >>> anagrams['aeilnrt'] ['entrail', 'latrine', 'ratline', 'reliant', 'retinal', 'trenail'] nltk.Index is a defaultdict(list) with extra support for initialization. Similarly, nltk.FreqDist is essentially a defaultdict(int) with extra support for initialization (along with sorting and plotting methods). Complex Keys and Values We can use default dictionaries with complex keys and values. Let’s study the range of possible tags for a word, given the word itself and the tag of the previous word. We will see how this information can be used by a POS tagger. >>> pos = nltk.defaultdict(lambda: nltk.defaultdict(int)) >>> brown_news_tagged = brown.tagged_words(categories='news', simplify_tags=True) >>> for ((w1, t1), (w2, t2)) in nltk.ibigrams(brown_news_tagged): ... pos[(t1, w2)][t2] += 1 ... >>> pos[('DET', 'right')] defaultdict(, {'ADV': 3, 'ADJ': 9, 'N': 3}) This example uses a dictionary whose default value for an entry is a dictionary (whose default value is int(), i.e., zero). Notice how we iterated over the bigrams of the tagged corpus, processing a pair of word-tag pairs for each iteration . Each time through the loop we updated our pos dictionary’s entry for (t1, w2), a tag and its following word . When we look up an item in pos we must specify a compound key , and we get back a dictionary object. A POS tagger could use such information to decide that the word right, when preceded by a determiner, should be tagged as ADJ. 196 | Chapter 5: Categorizing and Tagging Words Inverting a Dictionary Dictionaries support efficient lookup, so long as you want to get the value for any key. If d is a dictionary and k is a key, we type d[k] and immediately obtain the value. Finding a key given a value is slower and more cumbersome: >>> counts = nltk.defaultdict(int) >>> for word in nltk.corpus.gutenberg.words('milton-paradise.txt'): ... counts[word] += 1 ... >>> [key for (key, value) in counts.items() if value == 32] ['brought', 'Him', 'virtue', 'Against', 'There', 'thine', 'King', 'mortal', 'every', 'been'] If we expect to do this kind of “reverse lookup” often, it helps to construct a dictionary that maps values to keys. In the case that no two keys have the same value, this is an easy thing to do. We just get all the key-value pairs in the dictionary, and create a new dictionary of value-key pairs. The next example also illustrates another way of initial- izing a dictionary pos with key-value pairs. >>> pos = {'colorless': 'ADJ', 'ideas': 'N', 'sleep': 'V', 'furiously': 'ADV'} >>> pos2 = dict((value, key) for (key, value) in pos.items()) >>> pos2['N'] 'ideas' Let’s first make our part-of-speech dictionary a bit more realistic and add some more words to pos using the dictionary update() method, to create the situation where mul- tiple keys have the same value. Then the technique just shown for reverse lookup will no longer work (why not?). Instead, we have to use append() to accumulate the words for each part-of-speech, as follows: >>> pos.update({'cats': 'N', 'scratch': 'V', 'peacefully': 'ADV', 'old': 'ADJ'}) >>> pos2 = nltk.defaultdict(list) >>> for key, value in pos.items(): ... pos2[value].append(key) ... >>> pos2['ADV'] ['peacefully', 'furiously'] Now we have inverted the pos dictionary, and can look up any part-of-speech and find all words having that part-of-speech. We can do the same thing even more simply using NLTK’s support for indexing, as follows: >>> pos2 = nltk.Index((value, key) for (key, value) in pos.items()) >>> pos2['ADV'] ['peacefully', 'furiously'] A summary of Python’s dictionary methods is given in Table 5-5. 5.3 Mapping Words to Properties Using Python Dictionaries | 197 Table 5-5. Python’s dictionary methods: A summary of commonly used methods and idioms involving dictionaries Example Description d = {} Create an empty dictionary and assign it to d d[key] = value Assign a value to a given dictionary key d.keys() The list of keys of the dictionary list(d) The list of keys of the dictionary sorted(d) The keys of the dictionary, sorted key in d Test whether a particular key is in the dictionary for key in d Iterate over the keys of the dictionary d.values() The list of values in the dictionary dict([(k1,v1), (k2,v2), ...]) Create a dictionary from a list of key-value pairs d1.update(d2) Add all items from d2 to d1 defaultdict(int) A dictionary whose default value is zero 5.4 Automatic Tagging In the rest of this chapter we will explore various ways to automatically add part-of- speech tags to text. We will see that the tag of a word depends on the word and its context within a sentence. For this reason, we will be working with data at the level of (tagged) sentences rather than words. We’ll begin by loading the data we will be using. >>> from nltk.corpus import brown >>> brown_tagged_sents = brown.tagged_sents(categories='news') >>> brown_sents = brown.sents(categories='news') The Default Tagger The simplest possible tagger assigns the same tag to each token. This may seem to be a rather banal step, but it establishes an important baseline for tagger performance. In order to get the best result, we tag each word with the most likely tag. Let’s find out which tag is most likely (now using the unsimplified tagset): >>> tags = [tag for (word, tag) in brown.tagged_words(categories='news')] >>> nltk.FreqDist(tags).max() 'NN' Now we can create a tagger that tags everything as NN. >>> raw = 'I do not like green eggs and ham, I do not like them Sam I am!' >>> tokens = nltk.word_tokenize(raw) >>> default_tagger = nltk.DefaultTagger('NN') >>> default_tagger.tag(tokens) [('I', 'NN'), ('do', 'NN'), ('not', 'NN'), ('like', 'NN'), ('green', 'NN'), ('eggs', 'NN'), ('and', 'NN'), ('ham', 'NN'), (',', 'NN'), ('I', 'NN'), 198 | Chapter 5: Categorizing and Tagging Words ('do', 'NN'), ('not', 'NN'), ('like', 'NN'), ('them', 'NN'), ('Sam', 'NN'), ('I', 'NN'), ('am', 'NN'), ('!', 'NN')] Unsurprisingly, this method performs rather poorly. On a typical corpus, it will tag only about an eighth of the tokens correctly, as we see here: >>> default_tagger.evaluate(brown_tagged_sents) 0.13089484257215028 Default taggers assign their tag to every single word, even words that have never been encountered before. As it happens, once we have processed several thousand words of English text, most new words will be nouns. As we will see, this means that default taggers can help to improve the robustness of a language processing system. We will return to them shortly. The Regular Expression Tagger The regular expression tagger assigns tags to tokens on the basis of matching patterns. For instance, we might guess that any word ending in ed is the past participle of a verb, and any word ending with ’s is a possessive noun. We can express these as a list of regular expressions: >>> patterns = [ ... (r'.*ing$', 'VBG'), # gerunds ... (r'.*ed$', 'VBD'), # simple past ... (r'.*es$', 'VBZ'), # 3rd singular present ... (r'.*ould$', 'MD'), # modals ... (r'.*\'s$', 'NN$'), # possessive nouns ... (r'.*s$', 'NNS'), # plural nouns ... (r'^-?[0-9]+(.[0-9]+)?$', 'CD'), # cardinal numbers ... (r'.*', 'NN') # nouns (default) ... ] Note that these are processed in order, and the first one that matches is applied. Now we can set up a tagger and use it to tag a sentence. After this step, it is correct about a fifth of the time. >>> regexp_tagger = nltk.RegexpTagger(patterns) >>> regexp_tagger.tag(brown_sents[3]) [('``', 'NN'), ('Only', 'NN'), ('a', 'NN'), ('relative', 'NN'), ('handful', 'NN'), ('of', 'NN'), ('such', 'NN'), ('reports', 'NNS'), ('was', 'NNS'), ('received', 'VBD'), ("''", 'NN'), (',', 'NN'), ('the', 'NN'), ('jury', 'NN'), ('said', 'NN'), (',', 'NN'), ('``', 'NN'), ('considering', 'VBG'), ('the', 'NN'), ('widespread', 'NN'), ...] >>> regexp_tagger.evaluate(brown_tagged_sents) 0.20326391789486245 The final regular expression «.*» is a catch-all that tags everything as a noun. This is equivalent to the default tagger (only much less efficient). Instead of respecifying this as part of the regular expression tagger, is there a way to combine this tagger with the default tagger? We will see how to do this shortly. 5.4 Automatic Tagging | 199 Your Turn: See if you can come up with patterns to improve the per- formance of the regular expression tagger just shown. (Note that Sec- tion 6.1 describes a way to partially automate such work.) The Lookup Tagger A lot of high-frequency words do not have the NN tag. Let’s find the hundred most frequent words and store their most likely tag. We can then use this information as the model for a “lookup tagger” (an NLTK UnigramTagger): >>> fd = nltk.FreqDist(brown.words(categories='news')) >>> cfd = nltk.ConditionalFreqDist(brown.tagged_words(categories='news')) >>> most_freq_words = fd.keys()[:100] >>> likely_tags = dict((word, cfd[word].max()) for word in most_freq_words) >>> baseline_tagger = nltk.UnigramTagger(model=likely_tags) >>> baseline_tagger.evaluate(brown_tagged_sents) 0.45578495136941344 It should come as no surprise by now that simply knowing the tags for the 100 most frequent words enables us to tag a large fraction of tokens correctly (nearly half, in fact). Let’s see what it does on some untagged input text: >>> sent = brown.sents(categories='news')[3] >>> baseline_tagger.tag(sent) [('``', '``'), ('Only', None), ('a', 'AT'), ('relative', None), ('handful', None), ('of', 'IN'), ('such', None), ('reports', None), ('was', 'BEDZ'), ('received', None), ("''", "''"), (',', ','), ('the', 'AT'), ('jury', None), ('said', 'VBD'), (',', ','), ('``', '``'), ('considering', None), ('the', 'AT'), ('widespread', None), ('interest', None), ('in', 'IN'), ('the', 'AT'), ('election', None), (',', ','), ('the', 'AT'), ('number', None), ('of', 'IN'), ('voters', None), ('and', 'CC'), ('the', 'AT'), ('size', None), ('of', 'IN'), ('this', 'DT'), ('city', None), ("''", "''"), ('.', '.')] Many words have been assigned a tag of None, because they were not among the 100 most frequent words. In these cases we would like to assign the default tag of NN. In other words, we want to use the lookup table first, and if it is unable to assign a tag, then use the default tagger, a process known as backoff (Section 5.5). We do this by specifying one tagger as a parameter to the other, as shown next. Now the lookup tagger will only store word-tag pairs for words other than nouns, and whenever it cannot assign a tag to a word, it will invoke the default tagger. >>> baseline_tagger = nltk.UnigramTagger(model=likely_tags, ... backoff=nltk.DefaultTagger('NN')) Let’s put all this together and write a program to create and evaluate lookup taggers having a range of sizes (Example 5-4). 200 | Chapter 5: Categorizing and Tagging Words Example 5-4. Lookup tagger performance with varying model size. def performance(cfd, wordlist): lt = dict((word, cfd[word].max()) for word in wordlist) baseline_tagger = nltk.UnigramTagger(model=lt, backoff=nltk.DefaultTagger('NN')) return baseline_tagger.evaluate(brown.tagged_sents(categories='news')) def display(): import pylab words_by_freq = list(nltk.FreqDist(brown.words(categories='news'))) cfd = nltk.ConditionalFreqDist(brown.tagged_words(categories='news')) sizes = 2 ** pylab.arange(15) perfs = [performance(cfd, words_by_freq[:size]) for size in sizes] pylab.plot(sizes, perfs, '-bo') pylab.title('Lookup Tagger Performance with Varying Model Size') pylab.xlabel('Model Size') pylab.ylabel('Performance') pylab.show() >>> display() Observe in Figure 5-4 that performance initially increases rapidly as the model size grows, eventually reaching a plateau, when large increases in model size yield little improvement in performance. (This example used the pylab plotting package, dis- cussed in Section 4.8.) Evaluation In the previous examples, you will have noticed an emphasis on accuracy scores. In fact, evaluating the performance of such tools is a central theme in NLP. Recall the processing pipeline in Figure 1-5; any errors in the output of one module are greatly multiplied in the downstream modules. We evaluate the performance of a tagger relative to the tags a human expert would assign. Since we usually don’t have access to an expert and impartial human judge, we make do instead with gold standard test data. This is a corpus which has been man- ually annotated and accepted as a standard against which the guesses of an automatic system are assessed. The tagger is regarded as being correct if the tag it guesses for a given word is the same as the gold standard tag. Of course, the humans who designed and carried out the original gold standard anno- tation were only human. Further analysis might show mistakes in the gold standard, or may eventually lead to a revised tagset and more elaborate guidelines. Nevertheless, the gold standard is by definition “correct” as far as the evaluation of an automatic tagger is concerned. 5.4 Automatic Tagging | 201 Developing an annotated corpus is a major undertaking. Apart from the data, it generates sophisticated tools, documentation, and practices for ensuring high-quality annotation. The tagsets and other coding schemes inevitably depend on some theoretical position that is not shared by all. However, corpus creators often go to great lengths to make their work as theory-neutral as possible in order to maximize the usefulness of their work. We will discuss the challenges of creating a corpus in Chapter 11. 5.5 N-Gram Tagging Unigram Tagging Unigram taggers are based on a simple statistical algorithm: for each token, assign the tag that is most likely for that particular token. For example, it will assign the tag JJ to any occurrence of the word frequent, since frequent is used as an adjective (e.g., a fre- quent word) more often than it is used as a verb (e.g., I frequent this cafe). A unigram tagger behaves just like a lookup tagger (Section 5.4), except there is a more convenient Figure 5-4. Lookup tagger 202 | Chapter 5: Categorizing and Tagging Words technique for setting it up, called training. In the following code sample, we train a unigram tagger, use it to tag a sentence, and then evaluate: >>> from nltk.corpus import brown >>> brown_tagged_sents = brown.tagged_sents(categories='news') >>> brown_sents = brown.sents(categories='news') >>> unigram_tagger = nltk.UnigramTagger(brown_tagged_sents) >>> unigram_tagger.tag(brown_sents[2007]) [('Various', 'JJ'), ('of', 'IN'), ('the', 'AT'), ('apartments', 'NNS'), ('are', 'BER'), ('of', 'IN'), ('the', 'AT'), ('terrace', 'NN'), ('type', 'NN'), (',', ','), ('being', 'BEG'), ('on', 'IN'), ('the', 'AT'), ('ground', 'NN'), ('floor', 'NN'), ('so', 'QL'), ('that', 'CS'), ('entrance', 'NN'), ('is', 'BEZ'), ('direct', 'JJ'), ('.', '.')] >>> unigram_tagger.evaluate(brown_tagged_sents) 0.9349006503968017 We train a UnigramTagger by specifying tagged sentence data as a parameter when we initialize the tagger. The training process involves inspecting the tag of each word and storing the most likely tag for any word in a dictionary that is stored inside the tagger. Separating the Training and Testing Data Now that we are training a tagger on some data, we must be careful not to test it on the same data, as we did in the previous example. A tagger that simply memorized its training data and made no attempt to construct a general model would get a perfect score, but would be useless for tagging new text. Instead, we should split the data, training on 90% and testing on the remaining 10%: >>> size = int(len(brown_tagged_sents) * 0.9) >>> size 4160 >>> train_sents = brown_tagged_sents[:size] >>> test_sents = brown_tagged_sents[size:] >>> unigram_tagger = nltk.UnigramTagger(train_sents) >>> unigram_tagger.evaluate(test_sents) 0.81202033290142528 Although the score is worse, we now have a better picture of the usefulness of this tagger, i.e., its performance on previously unseen text. General N-Gram Tagging When we perform a language processing task based on unigrams, we are using one item of context. In the case of tagging, we consider only the current token, in isolation from any larger context. Given such a model, the best we can do is tag each word with its a priori most likely tag. This means we would tag a word such as wind with the same tag, regardless of whether it appears in the context the wind or to wind. An n-gram tagger is a generalization of a unigram tagger whose context is the current word together with the part-of-speech tags of the n-1 preceding tokens, as shown in Figure 5-5. The tag to be chosen, tn, is circled, and the context is shaded in grey. In the example of an n-gram tagger shown in Figure 5-5, we have n=3; that is, we consider 5.5 N-Gram Tagging | 203 the tags of the two preceding words in addition to the current word. An n-gram tagger picks the tag that is most likely in the given context. A 1-gram tagger is another term for a unigram tagger: i.e., the context used to tag a token is just the text of the token itself. 2-gram taggers are also called bigram taggers, and 3-gram taggers are called trigram taggers. The NgramTagger class uses a tagged training corpus to determine which part-of-speech tag is most likely for each context. Here we see a special case of an n-gram tagger, namely a bigram tagger. First we train it, then use it to tag untagged sentences: >>> bigram_tagger = nltk.BigramTagger(train_sents) >>> bigram_tagger.tag(brown_sents[2007]) [('Various', 'JJ'), ('of', 'IN'), ('the', 'AT'), ('apartments', 'NNS'), ('are', 'BER'), ('of', 'IN'), ('the', 'AT'), ('terrace', 'NN'), ('type', 'NN'), (',', ','), ('being', 'BEG'), ('on', 'IN'), ('the', 'AT'), ('ground', 'NN'), ('floor', 'NN'), ('so', 'CS'), ('that', 'CS'), ('entrance', 'NN'), ('is', 'BEZ'), ('direct', 'JJ'), ('.', '.')] >>> unseen_sent = brown_sents[4203] >>> bigram_tagger.tag(unseen_sent) [('The', 'AT'), ('population', 'NN'), ('of', 'IN'), ('the', 'AT'), ('Congo', 'NP'), ('is', 'BEZ'), ('13.5', None), ('million', None), (',', None), ('divided', None), ('into', None), ('at', None), ('least', None), ('seven', None), ('major', None), ('``', None), ('culture', None), ('clusters', None), ("''", None), ('and', None), ('innumerable', None), ('tribes', None), ('speaking', None), ('400', None), ('separate', None), ('dialects', None), ('.', None)] Notice that the bigram tagger manages to tag every word in a sentence it saw during training, but does badly on an unseen sentence. As soon as it encounters a new word (i.e., 13.5), it is unable to assign a tag. It cannot tag the following word (i.e., million), even if it was seen during training, simply because it never saw it during training with a None tag on the previous word. Consequently, the tagger fails to tag the rest of the sentence. Its overall accuracy score is very low: >>> bigram_tagger.evaluate(test_sents) 0.10276088906608193 Figure 5-5. Tagger context. 204 | Chapter 5: Categorizing and Tagging Words As n gets larger, the specificity of the contexts increases, as does the chance that the data we wish to tag contains contexts that were not present in the training data. This is known as the sparse data problem, and is quite pervasive in NLP. As a consequence, there is a trade-off between the accuracy and the coverage of our results (and this is related to the precision/recall trade-off in information retrieval). Caution! N-gram taggers should not consider context that crosses a sentence boundary. Accordingly, NLTK taggers are designed to work with lists of sentences, where each sentence is a list of words. At the start of a sentence, tn-1 and preceding tags are set to None. Combining Taggers One way to address the trade-off between accuracy and coverage is to use the more accurate algorithms when we can, but to fall back on algorithms with wider coverage when necessary. For example, we could combine the results of a bigram tagger, a unigram tagger, and a default tagger, as follows: 1. Try tagging the token with the bigram tagger. 2. If the bigram tagger is unable to find a tag for the token, try the unigram tagger. 3. If the unigram tagger is also unable to find a tag, use a default tagger. Most NLTK taggers permit a backoff tagger to be specified. The backoff tagger may itself have a backoff tagger: >>> t0 = nltk.DefaultTagger('NN') >>> t1 = nltk.UnigramTagger(train_sents, backoff=t0) >>> t2 = nltk.BigramTagger(train_sents, backoff=t1) >>> t2.evaluate(test_sents) 0.84491179108940495 Your Turn: Extend the preceding example by defining a TrigramTag ger called t3, which backs off to t2. Note that we specify the backoff tagger when the tagger is initialized so that training can take advantage of the backoff tagger. Thus, if the bigram tagger would assign the same tag as its unigram backoff tagger in a certain context, the bigram tagger discards the training instance. This keeps the bigram tagger model as small as possible. We can further specify that a tagger needs to see more than one instance of a context in order to retain it. For example, nltk.BigramTagger(sents, cutoff=2, backoff=t1) will dis- card contexts that have only been seen once or twice. 5.5 N-Gram Tagging | 205 Tagging Unknown Words Our approach to tagging unknown words still uses backoff to a regular expression tagger or a default tagger. These are unable to make use of context. Thus, if our tagger encountered the word blog, not seen during training, it would assign it the same tag, regardless of whether this word appeared in the context the blog or to blog. How can we do better with these unknown words, or out-of-vocabulary items? A useful method to tag unknown words based on context is to limit the vocabulary of a tagger to the most frequent n words, and to replace every other word with a special word UNK using the method shown in Section 5.3. During training, a unigram tagger will probably learn that UNK is usually a noun. However, the n-gram taggers will detect contexts in which it has some other tag. For example, if the preceding word is to (tagged TO), then UNK will probably be tagged as a verb. Storing Taggers Training a tagger on a large corpus may take a significant time. Instead of training a tagger every time we need one, it is convenient to save a trained tagger in a file for later reuse. Let’s save our tagger t2 to a file t2.pkl: >>> from cPickle import dump >>> output = open('t2.pkl', 'wb') >>> dump(t2, output, -1) >>> output.close() Now, in a separate Python process, we can load our saved tagger: >>> from cPickle import load >>> input = open('t2.pkl', 'rb') >>> tagger = load(input) >>> input.close() Now let’s check that it can be used for tagging: >>> text = """The board's action shows what free enterprise ... is up against in our complex maze of regulatory laws .""" >>> tokens = text.split() >>> tagger.tag(tokens) [('The', 'AT'), ("board's", 'NN$'), ('action', 'NN'), ('shows', 'NNS'), ('what', 'WDT'), ('free', 'JJ'), ('enterprise', 'NN'), ('is', 'BEZ'), ('up', 'RP'), ('against', 'IN'), ('in', 'IN'), ('our', 'PP$'), ('complex', 'JJ'), ('maze', 'NN'), ('of', 'IN'), ('regulatory', 'NN'), ('laws', 'NNS'), ('.', '.')] Performance Limitations What is the upper limit to the performance of an n-gram tagger? Consider the case of a trigram tagger. How many cases of part-of-speech ambiguity does it encounter? We can determine the answer to this question empirically: 206 | Chapter 5: Categorizing and Tagging Words >>> cfd = nltk.ConditionalFreqDist( ... ((x[1], y[1], z[0]), z[1]) ... for sent in brown_tagged_sents ... for x, y, z in nltk.trigrams(sent)) >>> ambiguous_contexts = [c for c in cfd.conditions() if len(cfd[c]) > 1] >>> sum(cfd[c].N() for c in ambiguous_contexts) / cfd.N() 0.049297702068029296 Thus, 1 out of 20 trigrams is ambiguous. Given the current word and the previous two tags, in 5% of cases there is more than one tag that could be legitimately assigned to the current word according to the training data. Assuming we always pick the most likely tag in such ambiguous contexts, we can derive a lower bound on the performance of a trigram tagger. Another way to investigate the performance of a tagger is to study its mistakes. Some tags may be harder than others to assign, and it might be possible to treat them specially by pre- or post-processing the data. A convenient way to look at tagging errors is the confusion matrix. It charts expected tags (the gold standard) against actual tags gen- erated by a tagger: >>> test_tags = [tag for sent in brown.sents(categories='editorial') ... for (word, tag) in t2.tag(sent)] >>> gold_tags = [tag for (word, tag) in brown.tagged_words(categories='editorial')] >>> print nltk.ConfusionMatrix(gold, test) Based on such analysis we may decide to modify the tagset. Perhaps a distinction be- tween tags that is difficult to make can be dropped, since it is not important in the context of some larger processing task. Another way to analyze the performance bound on a tagger comes from the less than 100% agreement between human annotators. In general, observe that the tagging process collapses distinctions: e.g., lexical identity is usually lost when all personal pronouns are tagged PRP. At the same time, the tagging process introduces new distinctions and removes ambiguities: e.g., deal tagged as VB or NN. This characteristic of collapsing certain distinctions and introducing new distinc- tions is an important feature of tagging which facilitates classification and prediction. When we introduce finer distinctions in a tagset, an n-gram tagger gets more detailed information about the left-context when it is deciding what tag to assign to a particular word. However, the tagger simultaneously has to do more work to classify the current token, simply because there are more tags to choose from. Conversely, with fewer dis- tinctions (as with the simplified tagset), the tagger has less information about context, and it has a smaller range of choices in classifying the current token. We have seen that ambiguity in the training data leads to an upper limit in tagger performance. Sometimes more context will resolve the ambiguity. In other cases, how- ever, as noted by (Abney, 1996), the ambiguity can be resolved only with reference to syntax or to world knowledge. Despite these imperfections, part-of-speech tagging has played a central role in the rise of statistical approaches to natural language processing. In the early 1990s, the surprising accuracy of statistical taggers was a striking 5.5 N-Gram Tagging | 207 demonstration that it was possible to solve one small part of the language understand- ing problem, namely part-of-speech disambiguation, without reference to deeper sour- ces of linguistic knowledge. Can this idea be pushed further? In Chapter 7, we will see that it can. Tagging Across Sentence Boundaries An n-gram tagger uses recent tags to guide the choice of tag for the current word. When tagging the first word of a sentence, a trigram tagger will be using the part-of-speech tag of the previous two tokens, which will normally be the last word of the previous sentence and the sentence-ending punctuation. However, the lexical category that closed the previous sentence has no bearing on the one that begins the next sentence. To deal with this situation, we can train, run, and evaluate taggers using lists of tagged sentences, as shown in Example 5-5. Example 5-5. N-gram tagging at the sentence level. brown_tagged_sents = brown.tagged_sents(categories='news') brown_sents = brown.sents(categories='news') size = int(len(brown_tagged_sents) * 0.9) train_sents = brown_tagged_sents[:size] test_sents = brown_tagged_sents[size:] t0 = nltk.DefaultTagger('NN') t1 = nltk.UnigramTagger(train_sents, backoff=t0) t2 = nltk.BigramTagger(train_sents, backoff=t1) >>> t2.evaluate(test_sents) 0.84491179108940495 5.6 Transformation-Based Tagging A potential issue with n-gram taggers is the size of their n-gram table (or language model). If tagging is to be employed in a variety of language technologies deployed on mobile computing devices, it is important to strike a balance between model size and tagger performance. An n-gram tagger with backoff may store trigram and bigram ta- bles, which are large, sparse arrays that may have hundreds of millions of entries. A second issue concerns context. The only information an n-gram tagger considers from prior context is tags, even though words themselves might be a useful source of information. It is simply impractical for n-gram models to be conditioned on the iden- tities of words in the context. In this section, we examine Brill tagging, an inductive tagging method which performs very well using models that are only a tiny fraction of the size of n-gram taggers. Brill tagging is a kind of transformation-based learning, named after its inventor. The general idea is very simple: guess the tag of each word, then go back and fix the mistakes. 208 | Chapter 5: Categorizing and Tagging Words In this way, a Brill tagger successively transforms a bad tagging of a text into a better one. As with n-gram tagging, this is a supervised learning method, since we need an- notated training data to figure out whether the tagger’s guess is a mistake or not. How- ever, unlike n-gram tagging, it does not count observations but compiles a list of trans- formational correction rules. The process of Brill tagging is usually explained by analogy with painting. Suppose we were painting a tree, with all its details of boughs, branches, twigs, and leaves, against a uniform sky-blue background. Instead of painting the tree first and then trying to paint blue in the gaps, it is simpler to paint the whole canvas blue, then “correct” the tree section by over-painting the blue background. In the same fashion, we might paint the trunk a uniform brown before going back to over-paint further details with even finer brushes. Brill tagging uses the same idea: begin with broad brush strokes, and then fix up the details, with successively finer changes. Let’s look at an example in- volving the following sentence: (1) The President said he will ask Congress to increase grants to states for voca- tional rehabilitation. We will examine the operation of two rules: (a) replace NN with VB when the previous word is TO; (b) replace TO with IN when the next tag is NNS. Table 5-6 illustrates this process, first tagging with the unigram tagger, then applying the rules to fix the errors. Table 5-6. Steps in Brill tagging Phrase to increase grants to states for vocational rehabilitation Unigram TO NN NNS TO NNS IN JJ NN Rule 1 VB Rule 2 IN Output TO VB NNS IN NNS IN JJ NN Gold TO VB NNS IN NNS IN JJ NN In this table, we see two rules. All such rules are generated from a template of the following form: “replace T1 with T2 in the context C.” Typical contexts are the identity or the tag of the preceding or following word, or the appearance of a specific tag within two to three words of the current word. During its training phase, the tagger guesses values for T1, T2, and C, to create thousands of candidate rules. Each rule is scored according to its net benefit: the number of incorrect tags that it corrects, less the number of correct tags it incorrectly modifies. Brill taggers have another interesting property: the rules are linguistically interpretable. Compare this with the n-gram taggers, which employ a potentially massive table of n- grams. We cannot learn much from direct inspection of such a table, in comparison to the rules learned by the Brill tagger. Example 5-6 demonstrates NLTK’s Brill tagger. 5.6 Transformation-Based Tagging | 209 Example 5-6. Brill tagger demonstration: The tagger has a collection of templates of the form X → Y if the preceding word is Z; the variables in these templates are instantiated to particular words and tags to create “rules”; the score for a rule is the number of broken examples it corrects minus the number of correct cases it breaks; apart from training a tagger, the demonstration displays residual errors. >>> nltk.tag.brill.demo() Training Brill tagger on 80 sentences... Finding initial useful rules... Found 6555 useful rules. B | S F r O | Score = Fixed - Broken c i o t | R Fixed = num tags changed incorrect -> correct o x k h | u Broken = num tags changed correct -> incorrect r e e e | l Other = num tags changed incorrect -> incorrect e d n r | e ------------------+------------------------------------------------------- 12 13 1 4 | NN -> VB if the tag of the preceding word is 'TO' 8 9 1 23 | NN -> VBD if the tag of the following word is 'DT' 8 8 0 9 | NN -> VBD if the tag of the preceding word is 'NNS' 6 9 3 16 | NN -> NNP if the tag of words i-2...i-1 is '-NONE-' 5 8 3 6 | NN -> NNP if the tag of the following word is 'NNP' 5 6 1 0 | NN -> NNP if the text of words i-2...i-1 is 'like' 5 5 0 3 | NN -> VBN if the text of the following word is '*-1' ... >>> print(open("errors.out").read()) left context | word/test->gold | right context --------------------------+------------------------+-------------------------- | Then/NN->RB | ,/, in/IN the/DT guests/N , in/IN the/DT guests/NNS | '/VBD->POS | honor/NN ,/, the/DT speed '/POS honor/NN ,/, the/DT | speedway/JJ->NN | hauled/VBD out/RP four/CD NN ,/, the/DT speedway/NN | hauled/NN->VBD | out/RP four/CD drivers/NN DT speedway/NN hauled/VBD | out/NNP->RP | four/CD drivers/NNS ,/, c dway/NN hauled/VBD out/RP | four/NNP->CD | drivers/NNS ,/, crews/NNS hauled/VBD out/RP four/CD | drivers/NNP->NNS | ,/, crews/NNS and/CC even P four/CD drivers/NNS ,/, | crews/NN->NNS | and/CC even/RB the/DT off NNS and/CC even/RB the/DT | official/NNP->JJ | Indianapolis/NNP 500/CD a | After/VBD->IN | the/DT race/NN ,/, Fortun ter/IN the/DT race/NN ,/, | Fortune/IN->NNP | 500/CD executives/NNS dro s/NNS drooled/VBD like/IN | schoolboys/NNP->NNS | over/IN the/DT cars/NNS a olboys/NNS over/IN the/DT | cars/NN->NNS | and/CC drivers/NNS ./. 5.7 How to Determine the Category of a Word Now that we have examined word classes in detail, we turn to a more basic question: how do we decide what category a word belongs to in the first place? In general, linguists use morphological, syntactic, and semantic clues to determine the category of a word. 210 | Chapter 5: Categorizing and Tagging Words Morphological Clues The internal structure of a word may give useful clues as to the word’s category. For example, -ness is a suffix that combines with an adjective to produce a noun, e.g., happy → happiness, ill → illness. So if we encounter a word that ends in -ness, this is very likely to be a noun. Similarly, -ment is a suffix that combines with some verbs to produce a noun, e.g., govern → government and establish → establishment. English verbs can also be morphologically complex. For instance, the present par- ticiple of a verb ends in -ing, and expresses the idea of ongoing, incomplete action (e.g., falling, eating). The -ing suffix also appears on nouns derived from verbs, e.g., the falling of the leaves (this is known as the gerund). Syntactic Clues Another source of information is the typical contexts in which a word can occur. For example, assume that we have already determined the category of nouns. Then we might say that a syntactic criterion for an adjective in English is that it can occur im- mediately before a noun, or immediately following the words be or very. According to these tests, near should be categorized as an adjective: (2) a. the near window b. The end is (very) near. Semantic Clues Finally, the meaning of a word is a useful clue as to its lexical category. For example, the best-known definition of a noun is semantic: “the name of a person, place, or thing.” Within modern linguistics, semantic criteria for word classes are treated with suspicion, mainly because they are hard to formalize. Nevertheless, semantic criteria underpin many of our intuitions about word classes, and enable us to make a good guess about the categorization of words in languages with which we are unfamiliar. For example, if all we know about the Dutch word verjaardag is that it means the same as the English word birthday, then we can guess that verjaardag is a noun in Dutch. However, some care is needed: although we might translate zij is vandaag jarig as it’s her birthday to- day, the word jarig is in fact an adjective in Dutch, and has no exact equivalent in English. New Words All languages acquire new lexical items. A list of words recently added to the Oxford Dictionary of English includes cyberslacker, fatoush, blamestorm, SARS, cantopop, bupkis, noughties, muggle, and robata. Notice that all these new words are nouns, and this is reflected in calling nouns an open class. By contrast, prepositions are regarded as a closed class. That is, there is a limited set of words belonging to the class (e.g., above, along, at, below, beside, between, during, for, from, in, near, on, outside, over, 5.7 How to Determine the Category of a Word | 211 past, through, towards, under, up, with), and membership of the set only changes very gradually over time. Morphology in Part-of-Speech Tagsets Common tagsets often capture some morphosyntactic information, that is, informa- tion about the kind of morphological markings that words receive by virtue of their syntactic role. Consider, for example, the selection of distinct grammatical forms of the word go illustrated in the following sentences: (3) a. Go away! b. He sometimes goes to the cafe. c. All the cakes have gone. d. We went on the excursion. Each of these forms—go, goes, gone, and went—is morphologically distinct from the others. Consider the form goes. This occurs in a restricted set of grammatical contexts, and requires a third person singular subject. Thus, the following sentences are ungrammatical. (4) a. *They sometimes goes to the cafe. b. *I sometimes goes to the cafe. By contrast, gone is the past participle form; it is required after have (and cannot be replaced in this context by goes), and cannot occur as the main verb of a clause. (5) a. *All the cakes have goes. b. *He sometimes gone to the cafe. We can easily imagine a tagset in which the four distinct grammatical forms just dis- cussed were all tagged as VB. Although this would be adequate for some purposes, a more fine-grained tagset provides useful information about these forms that can help other processors that try to detect patterns in tag sequences. The Brown tagset captures these distinctions, as summarized in Table 5-7. Table 5-7. Some morphosyntactic distinctions in the Brown tagset Form Category Tag go base VB goes third singular present VBZ gone past participle VBN going gerund VBG went simple past VBD 212 | Chapter 5: Categorizing and Tagging Words In addition to this set of verb tags, the various forms of the verb to be have special tags: be/BE, being/BEG, am/BEM, are/BER, is/BEZ, been/BEN, were/BED, and was/BEDZ (plus extra tags for negative forms of the verb). All told, this fine-grained tagging of verbs means that an automatic tagger that uses this tagset is effectively carrying out a limited amount of morphological analysis. Most part-of-speech tagsets make use of the same basic categories, such as noun, verb, adjective, and preposition. However, tagsets differ both in how finely they divide words into categories, and in how they define their categories. For example, is might be tagged simply as a verb in one tagset, but as a distinct form of the lexeme be in another tagset (as in the Brown Corpus). This variation in tagsets is unavoidable, since part-of-speech tags are used in different ways for different tasks. In other words, there is no one “right way” to assign tags, only more or less useful ways depending on one’s goals. 5.8 Summary • Words can be grouped into classes, such as nouns, verbs, adjectives, and adverbs. These classes are known as lexical categories or parts-of-speech. Parts-of-speech are assigned short labels, or tags, such as NN and VB. • The process of automatically assigning parts-of-speech to words in text is called part-of-speech tagging, POS tagging, or just tagging. • Automatic tagging is an important step in the NLP pipeline, and is useful in a variety of situations, including predicting the behavior of previously unseen words, ana- lyzing word usage in corpora, and text-to-speech systems. • Some linguistic corpora, such as the Brown Corpus, have been POS tagged. • A variety of tagging methods are possible, e.g., default tagger, regular expression tagger, unigram tagger, and n-gram taggers. These can be combined using a tech- nique known as backoff. • Taggers can be trained and evaluated using tagged corpora. • Backoff is a method for combining models: when a more specialized model (such as a bigram tagger) cannot assign a tag in a given context, we back off to a more general model (such as a unigram tagger). • Part-of-speech tagging is an important, early example of a sequence classification task in NLP: a classification decision at any one point in the sequence makes use of words and tags in the local context. • A dictionary is used to map between arbitrary types of information, such as a string and a number: freq['cat'] = 12. We create dictionaries using the brace notation: pos = {}, pos = {'furiously': 'adv', 'ideas': 'n', 'colorless': 'adj'}. • N-gram taggers can be defined for large values of n, but once n is larger than 3, we usually encounter the sparse data problem; even with a large quantity of training data, we see only a tiny fraction of possible contexts. 5.8 Summary | 213 • Transformation-based tagging involves learning a series of repair rules of the form “change tag s to tag t in context c,” where each rule fixes mistakes and possibly introduces a (smaller) number of errors. 5.9 Further Reading Extra materials for this chapter are posted at http://www.nltk.org/, including links to freely available resources on the Web. For more examples of tagging with NLTK, please see the Tagging HOWTO at http://www.nltk.org/howto. Chapters 4 and 5 of (Jurafsky & Martin, 2008) contain more advanced material on n-grams and part-of-speech tag- ging. Other approaches to tagging involve machine learning methods (Chapter 6). In Chapter 7, we will see a generalization of tagging called chunking in which a contiguous sequence of words is assigned a single tag. For tagset documentation, see nltk.help.upenn_tagset() and nltk.help.brown_tag set(). Lexical categories are introduced in linguistics textbooks, including those listed in Chapter 1 of this book. There are many other kinds of tagging. Words can be tagged with directives to a speech synthesizer, indicating which words should be emphasized. Words can be tagged with sense numbers, indicating which sense of the word was used. Words can also be tagged with morphological features. Examples of each of these kinds of tags are shown in the following list. For space reasons, we only show the tag for a single word. Note also that the first two examples use XML-style tags, where elements in angle brackets enclose the word that is tagged. Speech Synthesis Markup Language (W3C SSML) That is a big car! SemCor: Brown Corpus tagged with WordNet senses Space in any form is completely meas ured by the three dimensions. (Wordnet form/nn sense 4: “shape, form, config- uration, contour, conformation”) Morphological tagging, from the Turin University Italian Treebank E' italiano , come progetto e realizzazione , il primo (PRIMO ADJ ORDIN M SING) porto turistico dell' Albania . Note that tagging is also performed at higher levels. Here is an example of dialogue act tagging, from the NPS Chat Corpus (Forsyth & Martell, 2007) included with NLTK. Each turn of the dialogue is categorized as to its communicative function: Statement User117 Dude..., I wanted some of that ynQuestion User120 m I missing something? Bye User117 I'm gonna go fix food, I'll be back later. System User122 JOIN System User2 slaps User122 around a bit with a large trout. Statement User121 18/m pm me if u tryin to chat 214 | Chapter 5: Categorizing and Tagging Words 5.10 Exercises 1. ○ Search the Web for “spoof newspaper headlines,” to find such gems as: British Left Waffles on Falkland Islands, and Juvenile Court to Try Shooting Defendant. Manually tag these headlines to see whether knowledge of the part-of-speech tags removes the ambiguity. 2. ○ Working with someone else, take turns picking a word that can be either a noun or a verb (e.g., contest); the opponent has to predict which one is likely to be the most frequent in the Brown Corpus. Check the opponent’s prediction, and tally the score over several turns. 3. ○ Tokenize and tag the following sentence: They wind back the clock, while we chase after the wind. What different pronunciations and parts-of-speech are involved? 4. ○ Review the mappings in Table 5-4. Discuss any other examples of mappings you can think of. What type of information do they map from and to? 5. ○ Using the Python interpreter in interactive mode, experiment with the dictionary examples in this chapter. Create a dictionary d, and add some entries. What hap- pens whether you try to access a non-existent entry, e.g., d['xyz']? 6. ○ Try deleting an element from a dictionary d, using the syntax del d['abc']. Check that the item was deleted. 7. ○ Create two dictionaries, d1 and d2, and add some entries to each. Now issue the command d1.update(d2). What did this do? What might it be useful for? 8. ○ Create a dictionary e, to represent a single lexical entry for some word of your choice. Define keys such as headword, part-of-speech, sense, and example, and as- sign them suitable values. 9. ○ Satisfy yourself that there are restrictions on the distribution of go and went, in the sense that they cannot be freely interchanged in the kinds of contexts illustrated in (3), Section 5.7. 10. ○ Train a unigram tagger and run it on some new text. Observe that some words are not assigned a tag. Why not? 11. ○ Learn about the affix tagger (type help(nltk.AffixTagger)). Train an affix tagger and run it on some new text. Experiment with different settings for the affix length and the minimum word length. Discuss your findings. 12. ○ Train a bigram tagger with no backoff tagger, and run it on some of the training data. Next, run it on some new data. What happens to the performance of the tagger? Why? 13. ○ We can use a dictionary to specify the values to be substituted into a formatting string. Read Python’s library documentation for formatting strings (http://docs.py 5.10 Exercises | 215 thon.org/lib/typesseq-strings.html) and use this method to display today’s date in two different formats. 14. ◑ Use sorted() and set() to get a sorted list of tags used in the Brown Corpus, removing duplicates. 15. ◑ Write programs to process the Brown Corpus and find answers to the following questions: a. Which nouns are more common in their plural form, rather than their singular form? (Only consider regular plurals, formed with the -s suffix.) b. Which word has the greatest number of distinct tags? What are they, and what do they represent? c. List tags in order of decreasing frequency. What do the 20 most frequent tags represent? d. Which tags are nouns most commonly found after? What do these tags represent? 16. ◑ Explore the following issues that arise in connection with the lookup tagger: a. What happens to the tagger performance for the various model sizes when a backoff tagger is omitted? b. Consider the curve in Figure 5-4; suggest a good size for a lookup tagger that balances memory and performance. Can you come up with scenarios where it would be preferable to minimize memory usage, or to maximize performance with no regard for memory usage? 17. ◑ What is the upper limit of performance for a lookup tagger, assuming no limit to the size of its table? (Hint: write a program to work out what percentage of tokens of a word are assigned the most likely tag for that word, on average.) 18. ◑ Generate some statistics for tagged data to answer the following questions: a. What proportion of word types are always assigned the same part-of-speech tag? b. How many words are ambiguous, in the sense that they appear with at least two tags? c. What percentage of word tokens in the Brown Corpus involve these ambiguous words? 19. ◑ The evaluate() method works out how accurately the tagger performs on this text. For example, if the supplied tagged text was [('the', 'DT'), ('dog', 'NN')] and the tagger produced the output [('the', 'NN'), ('dog', 'NN')], then the score would be 0.5. Let’s try to figure out how the evaluation method works: a. A tagger t takes a list of words as input, and produces a list of tagged words as output. However, t.evaluate() is given correctly tagged text as its only parameter. What must it do with this input before performing the tagging? 216 | Chapter 5: Categorizing and Tagging Words b. Once the tagger has created newly tagged text, how might the evaluate() method go about comparing it with the original tagged text and computing the accuracy score? c. Now examine the source code to see how the method is implemented. Inspect nltk.tag.api.__file__ to discover the location of the source code, and open this file using an editor (be sure to use the api.py file and not the compiled api.pyc binary file). 20. ◑ Write code to search the Brown Corpus for particular words and phrases ac- cording to tags, to answer the following questions: a. Produce an alphabetically sorted list of the distinct words tagged as MD. b. Identify words that can be plural nouns or third person singular verbs (e.g., deals, flies). c. Identify three-word prepositional phrases of the form IN + DET + NN (e.g., in the lab). d. What is the ratio of masculine to feminine pronouns? 21. ◑ In Table 3-1, we saw a table involving frequency counts for the verbs adore, love, like, and prefer, and preceding qualifiers such as really. Investigate the full range of qualifiers (Brown tag QL) that appear before these four verbs. 22. ◑ We defined the regexp_tagger that can be used as a fall-back tagger for unknown words. This tagger only checks for cardinal numbers. By testing for particular prefix or suffix strings, it should be possible to guess other tags. For example, we could tag any word that ends with -s as a plural noun. Define a regular expression tagger (using RegexpTagger()) that tests for at least five other patterns in the spelling of words. (Use inline documentation to explain the rules.) 23. ◑ Consider the regular expression tagger developed in the exercises in the previous section. Evaluate the tagger using its accuracy() method, and try to come up with ways to improve its performance. Discuss your findings. How does objective eval- uation help in the development process? 24. ◑ How serious is the sparse data problem? Investigate the performance of n-gram taggers as n increases from 1 to 6. Tabulate the accuracy score. Estimate the training data required for these taggers, assuming a vocabulary size of 105 and a tagset size of 102. 25. ◑ Obtain some tagged data for another language, and train and evaluate a variety of taggers on it. If the language is morphologically complex, or if there are any orthographic clues (e.g., capitalization) to word classes, consider developing a reg- ular expression tagger for it (ordered after the unigram tagger, and before the de- fault tagger). How does the accuracy of your tagger(s) compare with the same taggers run on English data? Discuss any issues you encounter in applying these methods to the language. 5.10 Exercises | 217 26. ◑ Example 5-4 plotted a curve showing change in the performance of a lookup tagger as the model size was increased. Plot the performance curve for a unigram tagger, as the amount of training data is varied. 27. ◑ Inspect the confusion matrix for the bigram tagger t2 defined in Section 5.5, and identify one or more sets of tags to collapse. Define a dictionary to do the mapping, and evaluate the tagger on the simplified data. 28. ◑ Experiment with taggers using the simplified tagset (or make one of your own by discarding all but the first character of each tag name). Such a tagger has fewer distinctions to make, but much less information on which to base its work. Discuss your findings. 29. ◑ Recall the example of a bigram tagger which encountered a word it hadn’t seen during training, and tagged the rest of the sentence as None. It is possible for a bigram tagger to fail partway through a sentence even if it contains no unseen words (even if the sentence was used during training). In what circumstance can this happen? Can you write a program to find some examples of this? 30. ◑ Preprocess the Brown News data by replacing low-frequency words with UNK, but leaving the tags untouched. Now train and evaluate a bigram tagger on this data. How much does this help? What is the contribution of the unigram tagger and default tagger now? 31. ◑ Modify the program in Example 5-4 to use a logarithmic scale on the x-axis, by replacing pylab.plot() with pylab.semilogx(). What do you notice about the shape of the resulting plot? Does the gradient tell you anything? 32. ◑ Consult the documentation for the Brill tagger demo function, using help(nltk.tag.brill.demo). Experiment with the tagger by setting different values for the parameters. Is there any trade-off between training time (corpus size) and performance? 33. ◑ Write code that builds a dictionary of dictionaries of sets. Use it to store the set of POS tags that can follow a given word having a given POS tag, i.e., wordi → tagi → tagi+1. 34. ● There are 264 distinct words in the Brown Corpus having exactly three possible tags. a. Print a table with the integers 1..10 in one column, and the number of distinct words in the corpus having 1..10 distinct tags in the other column. b. For the word with the greatest number of distinct tags, print out sentences from the corpus containing the word, one for each possible tag. 35. ● Write a program to classify contexts involving the word must according to the tag of the following word. Can this be used to discriminate between the epistemic and deontic uses of must? 36. ● Create a regular expression tagger and various unigram and n-gram taggers, incorporating backoff, and train them on part of the Brown Corpus. 218 | Chapter 5: Categorizing and Tagging Words a. Create three different combinations of the taggers. Test the accuracy of each combined tagger. Which combination works best? b. Try varying the size of the training corpus. How does it affect your results? 37. ● Our approach for tagging an unknown word has been to consider the letters of the word (using RegexpTagger()), or to ignore the word altogether and tag it as a noun (using nltk.DefaultTagger()). These methods will not do well for texts hav- ing new words that are not nouns. Consider the sentence I like to blog on Kim’s blog. If blog is a new word, then looking at the previous tag (TO versus NP$) would probably be helpful, i.e., we need a default tagger that is sensitive to the preceding tag. a. Create a new kind of unigram tagger that looks at the tag of the previous word, and ignores the current word. (The best way to do this is to modify the source code for UnigramTagger(), which presumes knowledge of object-oriented pro- gramming in Python.) b. Add this tagger to the sequence of backoff taggers (including ordinary trigram and bigram taggers that look at words), right before the usual default tagger. c. Evaluate the contribution of this new unigram tagger. 38. ● Consider the code in Section 5.5, which determines the upper bound for accuracy of a trigram tagger. Review Abney’s discussion concerning the impossibility of exact tagging (Abney, 2006). Explain why correct tagging of these examples re- quires access to other kinds of information than just words and tags. How might you estimate the scale of this problem? 39. ● Use some of the estimation techniques in nltk.probability, such as Lidstone or Laplace estimation, to develop a statistical tagger that does a better job than n- gram backoff taggers in cases where contexts encountered during testing were not seen during training. 40. ● Inspect the diagnostic files created by the Brill tagger rules.out and errors.out. Obtain the demonstration code by accessing the source code (at http: //www.nltk.org/code) and create your own version of the Brill tagger. Delete some of the rule templates, based on what you learned from inspecting rules.out. Add some new rule templates which employ contexts that might help to correct the errors you saw in errors.out. 41. ● Develop an n-gram backoff tagger that permits “anti-n-grams” such as ["the", "the"] to be specified when a tagger is initialized. An anti-n-gram is assigned a count of zero and is used to prevent backoff for this n-gram (e.g., to avoid esti- mating P(the | the) as just P(the)). 42. ● Investigate three different ways to define the split between training and testing data when developing a tagger using the Brown Corpus: genre (category), source (fileid), and sentence. Compare their relative performance and discuss which method is the most legitimate. (You might use n-fold cross validation, discussed in Section 6.3, to improve the accuracy of the evaluations.) 5.10 Exercises | 219 CHAPTER 6 Learning to Classify Text Detecting patterns is a central part of Natural Language Processing. Words ending in -ed tend to be past tense verbs (Chapter 5). Frequent use of will is indicative of news text (Chapter 3). These observable patterns—word structure and word frequency— happen to correlate with particular aspects of meaning, such as tense and topic. But how did we know where to start looking, which aspects of form to associate with which aspects of meaning? The goal of this chapter is to answer the following questions: 1. How can we identify particular features of language data that are salient for clas- sifying it? 2. How can we construct models of language that can be used to perform language processing tasks automatically? 3. What can we learn about language from these models? Along the way we will study some important machine learning techniques, including decision trees, naive Bayes classifiers, and maximum entropy classifiers. We will gloss over the mathematical and statistical underpinnings of these techniques, focusing in- stead on how and when to use them (see Section 6.9 for more technical background). Before looking at these methods, we first need to appreciate the broad scope of this topic. 6.1 Supervised Classification Classification is the task of choosing the correct class label for a given input. In basic classification tasks, each input is considered in isolation from all other inputs, and the set of labels is defined in advance. Some examples of classification tasks are: 221 • Deciding whether an email is spam or not. • Deciding what the topic of a news article is, from a fixed list of topic areas such as “sports,” “technology,” and “politics.” • Deciding whether a given occurrence of the word bank is used to refer to a river bank, a financial institution, the act of tilting to the side, or the act of depositing something in a financial institution. The basic classification task has a number of interesting variants. For example, in multi- class classification, each instance may be assigned multiple labels; in open-class clas- sification, the set of labels is not defined in advance; and in sequence classification, a list of inputs are jointly classified. A classifier is called supervised if it is built based on training corpora containing the correct label for each input. The framework used by supervised classification is shown in Figure 6-1. Figure 6-1. Supervised classification. (a) During training, a feature extractor is used to convert each input value to a feature set. These feature sets, which capture the basic information about each input that should be used to classify it, are discussed in the next section. Pairs of feature sets and labels are fed into the machine learning algorithm to generate a model. (b) During prediction, the same feature extractor is used to convert unseen inputs to feature sets. These feature sets are then fed into the model, which generates predicted labels. In the rest of this section, we will look at how classifiers can be employed to solve a wide variety of tasks. Our discussion is not intended to be comprehensive, but to give a representative sample of tasks that can be performed with the help of text classifiers. Gender Identification In Section 2.4, we saw that male and female names have some distinctive characteristics. Names ending in a, e, and i are likely to be female, while names ending in k, o, r, s, and t are likely to be male. Let’s build a classifier to model these differences more precisely. 222 | Chapter 6: Learning to Classify Text The first step in creating a classifier is deciding what features of the input are relevant, and how to encode those features. For this example, we’ll start by just looking at the final letter of a given name. The following feature extractor function builds a dic- tionary containing relevant information about a given name: >>> def gender_features(word): ... return {'last_letter': word[-1]} >>> gender_features('Shrek') {'last_letter': 'k'} The dictionary that is returned by this function is called a feature set and maps from features’ names to their values. Feature names are case-sensitive strings that typically provide a short human-readable description of the feature. Feature values are values with simple types, such as Booleans, numbers, and strings. Most classification methods require that features be encoded using sim- ple value types, such as Booleans, numbers, and strings. But note that just because a feature has a simple type, this does not necessarily mean that the feature’s value is simple to express or compute; indeed, it is even possible to use very complex and informative values, such as the output of a second supervised classifier, as features. Now that we’ve defined a feature extractor, we need to prepare a list of examples and corresponding class labels: >>> from nltk.corpus import names >>> import random >>> names = ([(name, 'male') for name in names.words('male.txt')] + ... [(name, 'female') for name in names.words('female.txt')]) >>> random.shuffle(names) Next, we use the feature extractor to process the names data, and divide the resulting list of feature sets into a training set and a test set. The training set is used to train a new “naive Bayes” classifier. >>> featuresets = [(gender_features(n), g) for (n,g) in names] >>> train_set, test_set = featuresets[500:], featuresets[:500] >>> classifier = nltk.NaiveBayesClassifier.train(train_set) We will learn more about the naive Bayes classifier later in the chapter. For now, let’s just test it out on some names that did not appear in its training data: >>> classifier.classify(gender_features('Neo')) 'male' >>> classifier.classify(gender_features('Trinity')) 'female' Observe that these character names from The Matrix are correctly classified. Although this science fiction movie is set in 2199, it still conforms with our expectations about names and genders. We can systematically evaluate the classifier on a much larger quantity of unseen data: 6.1 Supervised Classification | 223 >>> print nltk.classify.accuracy(classifier, test_set) 0.758 Finally, we can examine the classifier to determine which features it found most effec- tive for distinguishing the names’ genders: >>> classifier.show_most_informative_features(5) Most Informative Features last_letter = 'a' female : male = 38.3 : 1.0 last_letter = 'k' male : female = 31.4 : 1.0 last_letter = 'f' male : female = 15.3 : 1.0 last_letter = 'p' male : female = 10.6 : 1.0 last_letter = 'w' male : female = 10.6 : 1.0 This listing shows that the names in the training set that end in a are female 38 times more often than they are male, but names that end in k are male 31 times more often than they are female. These ratios are known as likelihood ratios, and can be useful for comparing different feature-outcome relationships. Your Turn: Modify the gender_features() function to provide the clas- sifier with features encoding the length of the name, its first letter, and any other features that seem like they might be informative. Retrain the classifier with these new features, and test its accuracy. When working with large corpora, constructing a single list that contains the features of every instance can use up a large amount of memory. In these cases, use the function nltk.classify.apply_features, which returns an object that acts like a list but does not store all the feature sets in memory: >>> from nltk.classify import apply_features >>> train_set = apply_features(gender_features, names[500:]) >>> test_set = apply_features(gender_features, names[:500]) Choosing the Right Features Selecting relevant features and deciding how to encode them for a learning method can have an enormous impact on the learning method’s ability to extract a good model. Much of the interesting work in building a classifier is deciding what features might be relevant, and how we can represent them. Although it’s often possible to get decent performance by using a fairly simple and obvious set of features, there are usually sig- nificant gains to be had by using carefully constructed features based on a thorough understanding of the task at hand. Typically, feature extractors are built through a process of trial-and-error, guided by intuitions about what information is relevant to the problem. It’s common to start with a “kitchen sink” approach, including all the features that you can think of, and then checking to see which features actually are helpful. We take this approach for name gender features in Example 6-1. 224 | Chapter 6: Learning to Classify Text Example 6-1. A feature extractor that overfits gender features. The featuresets returned by this feature extractor contain a large number of specific features, leading to overfitting for the relatively small Names Corpus. def gender_features2(name): features = {} features["firstletter"] = name[0].lower() features["lastletter"] = name[–1].lower() for letter in 'abcdefghijklmnopqrstuvwxyz': features["count(%s)" % letter] = name.lower().count(letter) features["has(%s)" % letter] = (letter in name.lower()) return features >>> gender_features2('John') {'count(j)': 1, 'has(d)': False, 'count(b)': 0, ...} However, there are usually limits to the number of features that you should use with a given learning algorithm—if you provide too many features, then the algorithm will have a higher chance of relying on idiosyncrasies of your training data that don’t gen- eralize well to new examples. This problem is known as overfitting, and can be espe- cially problematic when working with small training sets. For example, if we train a naive Bayes classifier using the feature extractor shown in Example 6-1, it will overfit the relatively small training set, resulting in a system whose accuracy is about 1% lower than the accuracy of a classifier that only pays attention to the final letter of each name: >>> featuresets = [(gender_features2(n), g) for (n,g) in names] >>> train_set, test_set = featuresets[500:], featuresets[:500] >>> classifier = nltk.NaiveBayesClassifier.train(train_set) >>> print nltk.classify.accuracy(classifier, test_set) 0.748 Once an initial set of features has been chosen, a very productive method for refining the feature set is error analysis. First, we select a development set, containing the corpus data for creating the model. This development set is then subdivided into the training set and the dev-test set. >>> train_names = names[1500:] >>> devtest_names = names[500:1500] >>> test_names = names[:500] The training set is used to train the model, and the dev-test set is used to perform error analysis. The test set serves in our final evaluation of the system. For reasons discussed later, it is important that we employ a separate dev-test set for error analysis, rather than just using the test set. The division of the corpus data into different subsets is shown in Figure 6-2. Having divided the corpus into appropriate datasets, we train a model using the training set , and then run it on the dev-test set . >>> train_set = [(gender_features(n), g) for (n,g) in train_names] >>> devtest_set = [(gender_features(n), g) for (n,g) in devtest_names] >>> test_set = [(gender_features(n), g) for (n,g) in test_names] >>> classifier = nltk.NaiveBayesClassifier.train(train_set) 6.1 Supervised Classification | 225 >>> print nltk.classify.accuracy(classifier, devtest_set) 0.765 Figure 6-2. Organization of corpus data for training supervised classifiers. The corpus data is divided into two sets: the development set and the test set. The development set is often further subdivided into a training set and a dev-test set. Using the dev-test set, we can generate a list of the errors that the classifier makes when predicting name genders: >>> errors = [] >>> for (name, tag) in devtest_names: ... guess = classifier.classify(gender_features(name)) ... if guess != tag: ... errors.append( (tag, guess, name) ) We can then examine individual error cases where the model predicted the wrong label, and try to determine what additional pieces of information would allow it to make the right decision (or which existing pieces of information are tricking it into making the wrong decision). The feature set can then be adjusted accordingly. The names classifier that we have built generates about 100 errors on the dev-test corpus: >>> for (tag, guess, name) in sorted(errors): # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE ... print 'correct=%-8s guess=%-8s name=%-30s' % (tag, guess, name) ... correct=female guess=male name=Cindelyn ... correct=female guess=male name=Katheryn correct=female guess=male name=Kathryn ... correct=male guess=female name=Aldrich ... correct=male guess=female name=Mitch ... correct=male guess=female name=Rich ... 226 | Chapter 6: Learning to Classify Text Looking through this list of errors makes it clear that some suffixes that are more than one letter can be indicative of name genders. For example, names ending in yn appear to be predominantly female, despite the fact that names ending in n tend to be male; and names ending in ch are usually male, even though names that end in h tend to be female. We therefore adjust our feature extractor to include features for two-letter suffixes: >>> def gender_features(word): ... return {'suffix1': word[-1:], ... 'suffix2': word[-2:]} Rebuilding the classifier with the new feature extractor, we see that the performance on the dev-test dataset improves by almost three percentage points (from 76.5% to 78.2%): >>> train_set = [(gender_features(n), g) for (n,g) in train_names] >>> devtest_set = [(gender_features(n), g) for (n,g) in devtest_names] >>> classifier = nltk.NaiveBayesClassifier.train(train_set) >>> print nltk.classify.accuracy(classifier, devtest_set) 0.782 This error analysis procedure can then be repeated, checking for patterns in the errors that are made by the newly improved classifier. Each time the error analysis procedure is repeated, we should select a different dev-test/training split, to ensure that the clas- sifier does not start to reflect idiosyncrasies in the dev-test set. But once we’ve used the dev-test set to help us develop the model, we can no longer trust that it will give us an accurate idea of how well the model would perform on new data. It is therefore important to keep the test set separate, and unused, until our model development is complete. At that point, we can use the test set to evaluate how well our model will perform on new input values. Document Classification In Section 2.1, we saw several examples of corpora where documents have been labeled with categories. Using these corpora, we can build classifiers that will automatically tag new documents with appropriate category labels. First, we construct a list of docu- ments, labeled with the appropriate categories. For this example, we’ve chosen the Movie Reviews Corpus, which categorizes each review as positive or negative. >>> from nltk.corpus import movie_reviews >>> documents = [(list(movie_reviews.words(fileid)), category) ... for category in movie_reviews.categories() ... for fileid in movie_reviews.fileids(category)] >>> random.shuffle(documents) Next, we define a feature extractor for documents, so the classifier will know which aspects of the data it should pay attention to (see Example 6-2). For document topic identification, we can define a feature for each word, indicating whether the document contains that word. To limit the number of features that the classifier needs to process, we begin by constructing a list of the 2,000 most frequent words in the overall 6.1 Supervised Classification | 227 corpus . We can then define a feature extractor that simply checks whether each of these words is present in a given document. Example 6-2. A feature extractor for document classification, whose features indicate whether or not individual words are present in a given document. all_words = nltk.FreqDist(w.lower() for w in movie_reviews.words()) word_features = all_words.keys()[:2000] def document_features(document): document_words = set(document) features = {} for word in word_features: features['contains(%s)' % word] = (word in document_words) return features >>> print document_features(movie_reviews.words('pos/cv957_8737.txt')) {'contains(waste)': False, 'contains(lot)': False, ...} We compute the set of all words in a document in , rather than just checking if word in document, because checking whether a word occurs in a set is much faster than checking whether it occurs in a list (see Section 4.7). Now that we’ve defined our feature extractor, we can use it to train a classifier to label new movie reviews (Example 6-3). To check how reliable the resulting classifier is, we compute its accuracy on the test set . And once again, we can use show_most_infor mative_features() to find out which features the classifier found to be most informative . Example 6-3. Training and testing a classifier for document classification. featuresets = [(document_features(d), c) for (d,c) in documents] train_set, test_set = featuresets[100:], featuresets[:100] classifier = nltk.NaiveBayesClassifier.train(train_set) >>> print nltk.classify.accuracy(classifier, test_set) 0.81 >>> classifier.show_most_informative_features(5) Most Informative Features contains(outstanding) = True pos : neg = 11.1 : 1.0 contains(seagal) = True neg : pos = 7.7 : 1.0 contains(wonderfully) = True pos : neg = 6.8 : 1.0 contains(damon) = True pos : neg = 5.9 : 1.0 contains(wasted) = True neg : pos = 5.8 : 1.0 Apparently in this corpus, a review that mentions Seagal is almost 8 times more likely to be negative than positive, while a review that mentions Damon is about 6 times more likely to be positive. 228 | Chapter 6: Learning to Classify Text Part-of-Speech Tagging In Chapter 5, we built a regular expression tagger that chooses a part-of-speech tag for a word by looking at the internal makeup of the word. However, this regular expression tagger had to be handcrafted. Instead, we can train a classifier to work out which suf- fixes are most informative. Let’s begin by finding the most common suffixes: >>> from nltk.corpus import brown >>> suffix_fdist = nltk.FreqDist() >>> for word in brown.words(): ... word = word.lower() ... suffix_fdist.inc(word[-1:]) ... suffix_fdist.inc(word[-2:]) ... suffix_fdist.inc(word[-3:]) >>> common_suffixes = suffix_fdist.keys()[:100] >>> print common_suffixes ['e', ',', '.', 's', 'd', 't', 'he', 'n', 'a', 'of', 'the', 'y', 'r', 'to', 'in', 'f', 'o', 'ed', 'nd', 'is', 'on', 'l', 'g', 'and', 'ng', 'er', 'as', 'ing', 'h', 'at', 'es', 'or', 're', 'it', '``', 'an', "''", 'm', ';', 'i', 'ly', 'ion', ...] Next, we’ll define a feature extractor function that checks a given word for these suffixes: >>> def pos_features(word): ... features = {} ... for suffix in common_suffixes: ... features['endswith(%s)' % suffix] = word.lower().endswith(suffix) ... return features Feature extraction functions behave like tinted glasses, highlighting some of the prop- erties (colors) in our data and making it impossible to see other properties. The classifier will rely exclusively on these highlighted properties when determining how to label inputs. In this case, the classifier will make its decisions based only on information about which of the common suffixes (if any) a given word has. Now that we’ve defined our feature extractor, we can use it to train a new “decision tree” classifier (to be discussed in Section 6.4): >>> tagged_words = brown.tagged_words(categories='news') >>> featuresets = [(pos_features(n), g) for (n,g) in tagged_words] >>> size = int(len(featuresets) * 0.1) >>> train_set, test_set = featuresets[size:], featuresets[:size] >>> classifier = nltk.DecisionTreeClassifier.train(train_set) >>> nltk.classify.accuracy(classifier, test_set) 0.62705121829935351 >>> classifier.classify(pos_features('cats')) 'NNS' One nice feature of decision tree models is that they are often fairly easy to interpret. We can even instruct NLTK to print them out as pseudocode: 6.1 Supervised Classification | 229 >>> print classifier.pseudocode(depth=4) if endswith(,) == True: return ',' if endswith(,) == False: if endswith(the) == True: return 'AT' if endswith(the) == False: if endswith(s) == True: if endswith(is) == True: return 'BEZ' if endswith(is) == False: return 'VBZ' if endswith(s) == False: if endswith(.) == True: return '.' if endswith(.) == False: return 'NN' Here, we can see that the classifier begins by checking whether a word ends with a comma—if so, then it will receive the special tag ",". Next, the classifier checks whether the word ends in "the", in which case it’s almost certainly a determiner. This “suffix” gets used early by the decision tree because the word the is so common. Continuing on, the classifier checks if the word ends in s. If so, then it’s most likely to receive the verb tag VBZ (unless it’s the word is, which has the special tag BEZ), and if not, then it’s most likely a noun (unless it’s the punctuation mark “.”). The actual classifier contains further nested if-then statements below the ones shown here, but the depth=4 argument just displays the top portion of the decision tree. Exploiting Context By augmenting the feature extraction function, we could modify this part-of-speech tagger to leverage a variety of other word-internal features, such as the length of the word, the number of syllables it contains, or its prefix. However, as long as the feature extractor just looks at the target word, we have no way to add features that depend on the context in which the word appears. But contextual features often provide powerful clues about the correct tag—for example, when tagging the word fly, knowing that the previous word is a will allow us to determine that it is functioning as a noun, not a verb. In order to accommodate features that depend on a word’s context, we must revise the pattern that we used to define our feature extractor. Instead of just passing in the word to be tagged, we will pass in a complete (untagged) sentence, along with the index of the target word. This approach is demonstrated in Example 6-4, which employs a con- text-dependent feature extractor to define a part-of-speech tag classifier. 230 | Chapter 6: Learning to Classify Text Example 6-4. A part-of-speech classifier whose feature detector examines the context in which a word appears in order to determine which part-of-speech tag should be assigned. In particular, the identity of the previous word is included as a feature. def pos_features(sentence, i): features = {"suffix(1)": sentence[i][-1:], "suffix(2)": sentence[i][-2:], "suffix(3)": sentence[i][-3:]} if i == 0: features["prev-word"] = "" else: features["prev-word"] = sentence[i-1] return features >>> pos_features(brown.sents()[0], 8) {'suffix(3)': 'ion', 'prev-word': 'an', 'suffix(2)': 'on', 'suffix(1)': 'n'} >>> tagged_sents = brown.tagged_sents(categories='news') >>> featuresets = [] >>> for tagged_sent in tagged_sents: ... untagged_sent = nltk.tag.untag(tagged_sent) ... for i, (word, tag) in enumerate(tagged_sent): ... featuresets.append( (pos_features(untagged_sent, i), tag) ) >>> size = int(len(featuresets) * 0.1) >>> train_set, test_set = featuresets[size:], featuresets[:size] >>> classifier = nltk.NaiveBayesClassifier.train(train_set) >>> nltk.classify.accuracy(classifier, test_set) 0.78915962207856782 It’s clear that exploiting contextual features improves the performance of our part-of- speech tagger. For example, the classifier learns that a word is likely to be a noun if it comes immediately after the word large or the word gubernatorial. However, it is unable to learn the generalization that a word is probably a noun if it follows an adjective, because it doesn’t have access to the previous word’s part-of-speech tag. In general, simple classifiers always treat each input as independent from all other inputs. In many contexts, this makes perfect sense. For example, decisions about whether names tend to be male or female can be made on a case-by-case basis. However, there are often cases, such as part-of-speech tagging, where we are interested in solving classification problems that are closely related to one another. Sequence Classification In order to capture the dependencies between related classification tasks, we can use joint classifier models, which choose an appropriate labeling for a collection of related inputs. In the case of part-of-speech tagging, a variety of different sequence classifier models can be used to jointly choose part-of-speech tags for all the words in a given sentence. 6.1 Supervised Classification | 231 One sequence classification strategy, known as consecutive classification or greedy sequence classification, is to find the most likely class label for the first input, then to use that answer to help find the best label for the next input. The process can then be repeated until all of the inputs have been labeled. This is the approach that was taken by the bigram tagger from Section 5.5, which began by choosing a part-of-speech tag for the first word in the sentence, and then chose the tag for each subsequent word based on the word itself and the predicted tag for the previous word. This strategy is demonstrated in Example 6-5. First, we must augment our feature extractor function to take a history argument, which provides a list of the tags that we’ve predicted for the sentence so far . Each tag in history corresponds with a word in sentence. But note that history will only contain tags for words we’ve already clas- sified, that is, words to the left of the target word. Thus, although it is possible to look at some features of words to the right of the target word, it is not possible to look at the tags for those words (since we haven’t generated them yet). Having defined a feature extractor, we can proceed to build our sequence classifier . During training, we use the annotated tags to provide the appropriate history to the feature extractor, but when tagging new sentences, we generate the his- tory list based on the output of the tagger itself. Example 6-5. Part-of-speech tagging with a consecutive classifier. def pos_features(sentence, i, history): features = {"suffix(1)": sentence[i][-1:], "suffix(2)": sentence[i][-2:], "suffix(3)": sentence[i][-3:]} if i == 0: features["prev-word"] = "" features["prev-tag"] = "" else: features["prev-word"] = sentence[i-1] features["prev-tag"] = history[i-1] return features class ConsecutivePosTagger(nltk.TaggerI): def __init__(self, train_sents): train_set = [] for tagged_sent in train_sents: untagged_sent = nltk.tag.untag(tagged_sent) history = [] for i, (word, tag) in enumerate(tagged_sent): featureset = pos_features(untagged_sent, i, history) train_set.append( (featureset, tag) ) history.append(tag) self.classifier = nltk.NaiveBayesClassifier.train(train_set) def tag(self, sentence): history = [] for i, word in enumerate(sentence): featureset = pos_features(sentence, i, history) 232 | Chapter 6: Learning to Classify Text tag = self.classifier.classify(featureset) history.append(tag) return zip(sentence, history) >>> tagged_sents = brown.tagged_sents(categories='news') >>> size = int(len(tagged_sents) * 0.1) >>> train_sents, test_sents = tagged_sents[size:], tagged_sents[:size] >>> tagger = ConsecutivePosTagger(train_sents) >>> print tagger.evaluate(test_sents) 0.79796012981 Other Methods for Sequence Classification One shortcoming of this approach is that we commit to every decision that we make. For example, if we decide to label a word as a noun, but later find evidence that it should have been a verb, there’s no way to go back and fix our mistake. One solution to this problem is to adopt a transformational strategy instead. Transformational joint classi- fiers work by creating an initial assignment of labels for the inputs, and then iteratively refining that assignment in an attempt to repair inconsistencies between related inputs. The Brill tagger, described in Section 5.6, is a good example of this strategy. Another solution is to assign scores to all of the possible sequences of part-of-speech tags, and to choose the sequence whose overall score is highest. This is the approach taken by Hidden Markov Models. Hidden Markov Models are similar to consecutive classifiers in that they look at both the inputs and the history of predicted tags. How- ever, rather than simply finding the single best tag for a given word, they generate a probability distribution over tags. These probabilities are then combined to calculate probability scores for tag sequences, and the tag sequence with the highest probability is chosen. Unfortunately, the number of possible tag sequences is quite large. Given a tag set with 30 tags, there are about 600 trillion (3010) ways to label a 10-word sentence. In order to avoid considering all these possible sequences separately, Hidden Markov Models require that the feature extractor only look at the most recent tag (or the most recent n tags, where n is fairly small). Given that restriction, it is possible to use dynamic programming (Section 4.7) to efficiently find the most likely tag sequence. In particular, for each consecutive word index i, a score is computed for each possible current and previous tag. This same basic approach is taken by two more advanced models, called Maximum Entropy Markov Models and Linear-Chain Conditional Random Field Models; but different algorithms are used to find scores for tag sequences. 6.2 Further Examples of Supervised Classification Sentence Segmentation Sentence segmentation can be viewed as a classification task for punctuation: whenever we encounter a symbol that could possibly end a sentence, such as a period or a question mark, we have to decide whether it terminates the preceding sentence. 6.2 Further Examples of Supervised Classification | 233 The first step is to obtain some data that has already been segmented into sentences and convert it into a form that is suitable for extracting features: >>> sents = nltk.corpus.treebank_raw.sents() >>> tokens = [] >>> boundaries = set() >>> offset = 0 >>> for sent in nltk.corpus.treebank_raw.sents(): ... tokens.extend(sent) ... offset += len(sent) ... boundaries.add(offset-1) Here, tokens is a merged list of tokens from the individual sentences, and boundaries is a set containing the indexes of all sentence-boundary tokens. Next, we need to specify the features of the data that will be used in order to decide whether punctuation indi- cates a sentence boundary: >>> def punct_features(tokens, i): ... return {'next-word-capitalized': tokens[i+1][0].isupper(), ... 'prevword': tokens[i-1].lower(), ... 'punct': tokens[i], ... 'prev-word-is-one-char': len(tokens[i-1]) == 1} Based on this feature extractor, we can create a list of labeled featuresets by selecting all the punctuation tokens, and tagging whether they are boundary tokens or not: >>> featuresets = [(punct_features(tokens, i), (i in boundaries)) ... for i in range(1, len(tokens)-1) ... if tokens[i] in '.?!'] Using these featuresets, we can train and evaluate a punctuation classifier: >>> size = int(len(featuresets) * 0.1) >>> train_set, test_set = featuresets[size:], featuresets[:size] >>> classifier = nltk.NaiveBayesClassifier.train(train_set) >>> nltk.classify.accuracy(classifier, test_set) 0.97419354838709682 To use this classifier to perform sentence segmentation, we simply check each punc- tuation mark to see whether it’s labeled as a boundary, and divide the list of words at the boundary marks. The listing in Example 6-6 shows how this can be done. Example 6-6. Classification-based sentence segmenter. def segment_sentences(words): start = 0 sents = [] for i, word in words: if word in '.?!' and classifier.classify(words, i) == True: sents.append(words[start:i+1]) start = i+1 if start < len(words): sents.append(words[start:]) 234 | Chapter 6: Learning to Classify Text Identifying Dialogue Act Types When processing dialogue, it can be useful to think of utterances as a type of action performed by the speaker. This interpretation is most straightforward for performative statements such as I forgive you or I bet you can’t climb that hill. But greetings, questions, answers, assertions, and clarifications can all be thought of as types of speech-based actions. Recognizing the dialogue acts underlying the utterances in a dialogue can be an important first step in understanding the conversation. The NPS Chat Corpus, which was demonstrated in Section 2.1, consists of over 10,000 posts from instant messaging sessions. These posts have all been labeled with one of 15 dialogue act types, such as “Statement,” “Emotion,” “ynQuestion,” and “Contin- uer.” We can therefore use this data to build a classifier that can identify the dialogue act types for new instant messaging posts. The first step is to extract the basic messaging data. We will call xml_posts() to get a data structure representing the XML annotation for each post: >>> posts = nltk.corpus.nps_chat.xml_posts()[:10000] Next, we’ll define a simple feature extractor that checks what words the post contains: >>> def dialogue_act_features(post): ... features = {} ... for word in nltk.word_tokenize(post): ... features['contains(%s)' % word.lower()] = True ... return features Finally, we construct the training and testing data by applying the feature extractor to each post (using post.get('class') to get a post’s dialogue act type), and create a new classifier: >>> featuresets = [(dialogue_act_features(post.text), post.get('class')) ... for post in posts] >>> size = int(len(featuresets) * 0.1) >>> train_set, test_set = featuresets[size:], featuresets[:size] >>> classifier = nltk.NaiveBayesClassifier.train(train_set) >>> print nltk.classify.accuracy(classifier, test_set) 0.66 Recognizing Textual Entailment Recognizing textual entailment (RTE) is the task of determining whether a given piece of text T entails another text called the “hypothesis” (as already discussed in Sec- tion 1.5). To date, there have been four RTE Challenges, where shared development and test data is made available to competing teams. Here are a couple of examples of text/hypothesis pairs from the Challenge 3 development dataset. The label True indi- cates that the entailment holds, and False indicates that it fails to hold. 6.2 Further Examples of Supervised Classification | 235 Challenge 3, Pair 34 (True) T: Parviz Davudi was representing Iran at a meeting of the Shanghai Co-operation Organisation (SCO), the fledgling association that binds Russia, China and four former Soviet republics of central Asia together to fight terrorism. H: China is a member of SCO. Challenge 3, Pair 81 (False) T: According to NC Articles of Organization, the members of LLC company are H. Nelson Beavers, III, H. Chester Beavers and Jennie Beavers Stewart. H: Jennie Beavers Stewart is a share-holder of Carolina Analytical Laboratory. It should be emphasized that the relationship between text and hypothesis is not in- tended to be logical entailment, but rather whether a human would conclude that the text provides reasonable evidence for taking the hypothesis to be true. We can treat RTE as a classification task, in which we try to predict the True/False label for each pair. Although it seems likely that successful approaches to this task will in- volve a combination of parsing, semantics, and real-world knowledge, many early at- tempts at RTE achieved reasonably good results with shallow analysis, based on sim- ilarity between the text and hypothesis at the word level. In the ideal case, we would expect that if there is an entailment, then all the information expressed by the hypoth- esis should also be present in the text. Conversely, if there is information found in the hypothesis that is absent from the text, then there will be no entailment. In our RTE feature detector (Example 6-7), we let words (i.e., word types) serve as proxies for information, and our features count the degree of word overlap, and the degree to which there are words in the hypothesis but not in the text (captured by the method hyp_extra()). Not all words are equally important—named entity mentions, such as the names of people, organizations, and places, are likely to be more significant, which motivates us to extract distinct information for words and nes (named entities). In addition, some high-frequency function words are filtered out as “stopwords.” Example 6-7. “Recognizing Text Entailment” feature extractor: The RTEFeatureExtractor class builds a bag of words for both the text and the hypothesis after throwing away some stopwords, then calculates overlap and difference. def rte_features(rtepair): extractor = nltk.RTEFeatureExtractor(rtepair) features = {} features['word_overlap'] = len(extractor.overlap('word')) features['word_hyp_extra'] = len(extractor.hyp_extra('word')) features['ne_overlap'] = len(extractor.overlap('ne')) features['ne_hyp_extra'] = len(extractor.hyp_extra('ne')) return features To illustrate the content of these features, we examine some attributes of the text/ hypothesis Pair 34 shown earlier: 236 | Chapter 6: Learning to Classify Text >>> rtepair = nltk.corpus.rte.pairs(['rte3_dev.xml'])[33] >>> extractor = nltk.RTEFeatureExtractor(rtepair) >>> print extractor.text_words set(['Russia', 'Organisation', 'Shanghai', 'Asia', 'four', 'at', 'operation', 'SCO', ...]) >>> print extractor.hyp_words set(['member', 'SCO', 'China']) >>> print extractor.overlap('word') set([]) >>> print extractor.overlap('ne') set(['SCO', 'China']) >>> print extractor.hyp_extra('word') set(['member']) These features indicate that all important words in the hypothesis are contained in the text, and thus there is some evidence for labeling this as True. The module nltk.classify.rte_classify reaches just over 58% accuracy on the com- bined RTE test data using methods like these. Although this figure is not very impressive, it requires significant effort, and more linguistic processing, to achieve much better results. Scaling Up to Large Datasets Python provides an excellent environment for performing basic text processing and feature extraction. However, it is not able to perform the numerically intensive calcu- lations required by machine learning methods nearly as quickly as lower-level languages such as C. Thus, if you attempt to use the pure-Python machine learning implemen- tations (such as nltk.NaiveBayesClassifier) on large datasets, you may find that the learning algorithm takes an unreasonable amount of time and memory to complete. If you plan to train classifiers with large amounts of training data or a large number of features, we recommend that you explore NLTK’s facilities for interfacing with external machine learning packages. Once these packages have been installed, NLTK can trans- parently invoke them (via system calls) to train classifier models significantly faster than the pure-Python classifier implementations. See the NLTK web page for a list of rec- ommended machine learning packages that are supported by NLTK. 6.3 Evaluation In order to decide whether a classification model is accurately capturing a pattern, we must evaluate that model. The result of this evaluation is important for deciding how trustworthy the model is, and for what purposes we can use it. Evaluation can also be an effective tool for guiding us in making future improvements to the model. The Test Set Most evaluation techniques calculate a score for a model by comparing the labels that it generates for the inputs in a test set (or evaluation set) with the correct labels for 6.3 Evaluation | 237 those inputs. This test set typically has the same format as the training set. However, it is very important that the test set be distinct from the training corpus: if we simply reused the training set as the test set, then a model that simply memorized its input, without learning how to generalize to new examples, would receive misleadingly high scores. When building the test set, there is often a trade-off between the amount of data avail- able for testing and the amount available for training. For classification tasks that have a small number of well-balanced labels and a diverse test set, a meaningful evaluation can be performed with as few as 100 evaluation instances. But if a classification task has a large number of labels or includes very infrequent labels, then the size of the test set should be chosen to ensure that the least frequent label occurs at least 50 times. Additionally, if the test set contains many closely related instances—such as instances drawn from a single document—then the size of the test set should be increased to ensure that this lack of diversity does not skew the evaluation results. When large amounts of annotated data are available, it is common to err on the side of safety by using 10% of the overall data for evaluation. Another consideration when choosing the test set is the degree of similarity between instances in the test set and those in the development set. The more similar these two datasets are, the less confident we can be that evaluation results will generalize to other datasets. For example, consider the part-of-speech tagging task. At one extreme, we could create the training set and test set by randomly assigning sentences from a data source that reflects a single genre, such as news: >>> import random >>> from nltk.corpus import brown >>> tagged_sents = list(brown.tagged_sents(categories='news')) >>> random.shuffle(tagged_sents) >>> size = int(len(tagged_sents) * 0.1) >>> train_set, test_set = tagged_sents[size:], tagged_sents[:size] In this case, our test set will be very similar to our training set. The training set and test set are taken from the same genre, and so we cannot be confident that evaluation results would generalize to other genres. What’s worse, because of the call to random.shuffle(), the test set contains sentences that are taken from the same docu- ments that were used for training. If there is any consistent pattern within a document (say, if a given word appears with a particular part-of-speech tag especially frequently), then that difference will be reflected in both the development set and the test set. A somewhat better approach is to ensure that the training set and test set are taken from different documents: >>> file_ids = brown.fileids(categories='news') >>> size = int(len(file_ids) * 0.1) >>> train_set = brown.tagged_sents(file_ids[size:]) >>> test_set = brown.tagged_sents(file_ids[:size]) If we want to perform a more stringent evaluation, we can draw the test set from docu- ments that are less closely related to those in the training set: 238 | Chapter 6: Learning to Classify Text >>> train_set = brown.tagged_sents(categories='news') >>> test_set = brown.tagged_sents(categories='fiction') If we build a classifier that performs well on this test set, then we can be confident that it has the power to generalize well beyond the data on which it was trained. Accuracy The simplest metric that can be used to evaluate a classifier, accuracy, measures the percentage of inputs in the test set that the classifier correctly labeled. For example, a name gender classifier that predicts the correct name 60 times in a test set containing 80 names would have an accuracy of 60/80 = 75%. The function nltk.classify.accu racy() will calculate the accuracy of a classifier model on a given test set: >>> classifier = nltk.NaiveBayesClassifier.train(train_set) >>> print 'Accuracy: %4.2f' % nltk.classify.accuracy(classifier, test_set) 0.75 When interpreting the accuracy score of a classifier, it is important to consider the frequencies of the individual class labels in the test set. For example, consider a classifier that determines the correct word sense for each occurrence of the word bank. If we evaluate this classifier on financial newswire text, then we may find that the financial- institution sense appears 19 times out of 20. In that case, an accuracy of 95% would hardly be impressive, since we could achieve that accuracy with a model that always returns the financial-institution sense. However, if we instead evaluate the classifier on a more balanced corpus, where the most frequent word sense has a frequency of 40%, then a 95% accuracy score would be a much more positive result. (A similar issue arises when measuring inter-annotator agreement in Section 11.2.) Precision and Recall Another instance where accuracy scores can be misleading is in “search” tasks, such as information retrieval, where we are attempting to find documents that are relevant to a particular task. Since the number of irrelevant documents far outweighs the number of relevant documents, the accuracy score for a model that labels every document as irrelevant would be very close to 100%. It is therefore conventional to employ a different set of measures for search tasks, based on the number of items in each of the four categories shown in Figure 6-3: • True positives are relevant items that we correctly identified as relevant. • True negatives are irrelevant items that we correctly identified as irrelevant. • False positives (or Type I errors) are irrelevant items that we incorrectly identi- fied as relevant. • False negatives (or Type II errors) are relevant items that we incorrectly identi- fied as irrelevant. 6.3 Evaluation | 239 Given these four numbers, we can define the following metrics: • Precision, which indicates how many of the items that we identified were relevant, is TP/(TP+FP). • Recall, which indicates how many of the relevant items that we identified, is TP/(TP+FN). • The F-Measure (or F-Score), which combines the precision and recall to give a single score, is defined to be the harmonic mean of the precision and recall (2 × Precision × Recall)/(Precision+Recall). Confusion Matrices When performing classification tasks with three or more labels, it can be informative to subdivide the errors made by the model based on which types of mistake it made. A confusion matrix is a table where each cell [i,j] indicates how often label j was pre- dicted when the correct label was i. Thus, the diagonal entries (i.e., cells [i,j]) indicate labels that were correctly predicted, and the off-diagonal entries indicate errors. In the following example, we generate a confusion matrix for the unigram tagger developed in Section 5.4: Figure 6-3. True and false positives and negatives. 240 | Chapter 6: Learning to Classify Text >>> def tag_list(tagged_sents): ... return [tag for sent in tagged_sents for (word, tag) in sent] >>> def apply_tagger(tagger, corpus): ... return [tagger.tag(nltk.tag.untag(sent)) for sent in corpus] >>> gold = tag_list(brown.tagged_sents(categories='editorial')) >>> test = tag_list(apply_tagger(t2, brown.tagged_sents(categories='editorial'))) >>> cm = nltk.ConfusionMatrix(gold, test) | N | | N I A J N V N | | N N T J . S , B P | ----+----------------------------------------------------------------+ NN | <11.8%> 0.0% . 0.2% . 0.0% . 0.3% 0.0% | IN | 0.0% <9.0%> . . . 0.0% . . . | AT | . . <8.6%> . . . . . . | JJ | 1.6% . . <4.0%> . . . 0.0% 0.0% | . | . . . . <4.8%> . . . . | NS | 1.5% . . . . <3.2%> . . 0.0% | , | . . . . . . <4.4%> . . | B | 0.9% . . 0.0% . . . <2.4%> . | NP | 1.0% . . 0.0% . . . . <1.9%>| ----+----------------------------------------------------------------+ (row = reference; col = test) The confusion matrix indicates that common errors include a substitution of NN for JJ (for 1.6% of words), and of NN for NNS (for 1.5% of words). Note that periods (.) indicate cells whose value is 0, and that the diagonal entries—which correspond to correct classifications—are marked with angle brackets. Cross-Validation In order to evaluate our models, we must reserve a portion of the annotated data for the test set. As we already mentioned, if the test set is too small, our evaluation may not be accurate. However, making the test set larger usually means making the training set smaller, which can have a significant impact on performance if a limited amount of annotated data is available. One solution to this problem is to perform multiple evaluations on different test sets, then to combine the scores from those evaluations, a technique known as cross- validation. In particular, we subdivide the original corpus into N subsets called folds. For each of these folds, we train a model using all of the data except the data in that fold, and then test that model on the fold. Even though the individual folds might be too small to give accurate evaluation scores on their own, the combined evaluation score is based on a large amount of data and is therefore quite reliable. A second, and equally important, advantage of using cross-validation is that it allows us to examine how widely the performance varies across different training sets. If we get very similar scores for all N training sets, then we can be fairly confident that the score is accurate. On the other hand, if scores vary widely across the N training sets, then we should probably be skeptical about the accuracy of the evaluation score. 6.3 Evaluation | 241 6.4 Decision Trees In the next three sections, we’ll take a closer look at three machine learning methods that can be used to automatically build classification models: decision trees, naive Bayes classifiers, and Maximum Entropy classifiers. As we’ve seen, it’s possible to treat these learning methods as black boxes, simply training models and using them for prediction without understanding how they work. But there’s a lot to be learned from taking a closer look at how these learning methods select models based on the data in a training set. An understanding of these methods can help guide our selection of appropriate features, and especially our decisions about how those features should be encoded. And an understanding of the generated models can allow us to extract information about which features are most informative, and how those features relate to one an- other. A decision tree is a simple flowchart that selects labels for input values. This flowchart consists of decision nodes, which check feature values, and leaf nodes, which assign labels. To choose the label for an input value, we begin at the flowchart’s initial decision node, known as its root node. This node contains a condition that checks one of the input value’s features, and selects a branch based on that feature’s value. Following the branch that describes our input value, we arrive at a new decision node, with a new condition on the input value’s features. We continue following the branch selected by each node’s condition, until we arrive at a leaf node which provides a label for the input value. Figure 6-4 shows an example decision tree model for the name gender task. Once we have a decision tree, it is straightforward to use it to assign labels to new input values. What’s less straightforward is how we can build a decision tree that models a given training set. But before we look at the learning algorithm for building decision trees, we’ll consider a simpler task: picking the best “decision stump” for a corpus. A Figure 6-4. Decision Tree model for the name gender task. Note that tree diagrams are conventionally drawn “upside down,” with the root at the top, and the leaves at the bottom. 242 | Chapter 6: Learning to Classify Text decision stump is a decision tree with a single node that decides how to classify inputs based on a single feature. It contains one leaf for each possible feature value, specifying the class label that should be assigned to inputs whose features have that value. In order to build a decision stump, we must first decide which feature should be used. The simplest method is to just build a decision stump for each possible feature, and see which one achieves the highest accuracy on the training data, although there are other alternatives that we will discuss later. Once we’ve picked a feature, we can build the decision stump by assigning a label to each leaf based on the most frequent label for the selected examples in the training set (i.e., the examples where the selected feature has that value). Given the algorithm for choosing decision stumps, the algorithm for growing larger decision trees is straightforward. We begin by selecting the overall best decision stump for the classification task. We then check the accuracy of each of the leaves on the training set. Leaves that do not achieve sufficient accuracy are then replaced by new decision stumps, trained on the subset of the training corpus that is selected by the path to the leaf. For example, we could grow the decision tree in Figure 6-4 by replacing the leftmost leaf with a new decision stump, trained on the subset of the training set names that do not start with a k or end with a vowel or an l. Entropy and Information Gain As was mentioned before, there are several methods for identifying the most informa- tive feature for a decision stump. One popular alternative, called information gain, measures how much more organized the input values become when we divide them up using a given feature. To measure how disorganized the original set of input values are, we calculate entropy of their labels, which will be high if the input values have highly varied labels, and low if many input values all have the same label. In particular, entropy is defined as the sum of the probability of each label times the log probability of that same label: (1) H = Σl ∈ labelsP(l) × log2P(l). For example, Figure 6-5 shows how the entropy of labels in the name gender prediction task depends on the ratio of male to female names. Note that if most input values have the same label (e.g., if P(male) is near 0 or near 1), then entropy is low. In particular, labels that have low frequency do not contribute much to the entropy (since P(l) is small), and labels with high frequency also do not contribute much to the entropy (since log2P(l) is small). On the other hand, if the input values have a wide variety of labels, then there are many labels with a “medium” frequency, where neither P(l) nor log2P(l) is small, so the entropy is high. Example 6-8 demonstrates how to calculate the entropy of a list of labels. 6.4 Decision Trees | 243 Figure 6-5. The entropy of labels in the name gender prediction task, as a function of the percentage of names in a given set that are male. Example 6-8. Calculating the entropy of a list of labels. import math def entropy(labels): freqdist = nltk.FreqDist(labels) probs = [freqdist.freq(l) for l in nltk.FreqDist(labels)] return -sum([p * math.log(p,2) for p in probs]) >>> print entropy(['male', 'male', 'male', 'male']) 0.0 >>> print entropy(['male', 'female', 'male', 'male']) 0.811278124459 >>> print entropy(['female', 'male', 'female', 'male']) 1.0 >>> print entropy(['female', 'female', 'male', 'female']) 0.811278124459 >>> print entropy(['female', 'female', 'female', 'female']) 0.0 Once we have calculated the entropy of the labels of the original set of input values, we can determine how much more organized the labels become once we apply the decision stump. To do so, we calculate the entropy for each of the decision stump’s leaves, and take the average of those leaf entropy values (weighted by the number of samples in each leaf). The information gain is then equal to the original entropy minus this new, reduced entropy. The higher the information gain, the better job the decision stump does of dividing the input values into coherent groups, so we can build decision trees by selecting the decision stumps with the highest information gain. Another consideration for decision trees is efficiency. The simple algorithm for selecting decision stumps described earlier must construct a candidate decision stump for every possible feature, and this process must be repeated for every node in the constructed 244 | Chapter 6: Learning to Classify Text decision tree. A number of algorithms have been developed to cut down on the training time by storing and reusing information about previously evaluated examples. Decision trees have a number of useful qualities. To begin with, they’re simple to un- derstand, and easy to interpret. This is especially true near the top of the decision tree, where it is usually possible for the learning algorithm to find very useful features. De- cision trees are especially well suited to cases where many hierarchical categorical dis- tinctions can be made. For example, decision trees can be very effective at capturing phylogeny trees. However, decision trees also have a few disadvantages. One problem is that, since each branch in the decision tree splits the training data, the amount of training data available to train nodes lower in the tree can become quite small. As a result, these lower decision nodes may overfit the training set, learning patterns that reflect idiosyncrasies of the training set rather than linguistically significant patterns in the underlying problem. One solution to this problem is to stop dividing nodes once the amount of training data becomes too small. Another solution is to grow a full decision tree, but then to prune decision nodes that do not improve performance on a dev-test. A second problem with decision trees is that they force features to be checked in a specific order, even when features may act relatively independently of one another. For example, when classifying documents into topics (such as sports, automotive, or mur- der mystery), features such as hasword(football) are highly indicative of a specific label, regardless of what the other feature values are. Since there is limited space near the top of the decision tree, most of these features will need to be repeated on many different branches in the tree. And since the number of branches increases exponentially as we go down the tree, the amount of repetition can be very large. A related problem is that decision trees are not good at making use of features that are weak predictors of the correct label. Since these features make relatively small incremental improvements, they tend to occur very low in the decision tree. But by the time the decision tree learner has descended far enough to use these features, there is not enough training data left to reliably determine what effect they should have. If we could instead look at the effect of these features across the entire training set, then we might be able to make some conclusions about how they should affect the choice of label. The fact that decision trees require that features be checked in a specific order limits their ability to exploit features that are relatively independent of one another. The naive Bayes classification method, which we’ll discuss next, overcomes this limitation by allowing all features to act “in parallel.” 6.5 Naive Bayes Classifiers In naive Bayes classifiers, every feature gets a say in determining which label should be assigned to a given input value. To choose a label for an input value, the naive Bayes 6.5 Naive Bayes Classifiers | 245 classifier begins by calculating the prior probability of each label, which is determined by checking the frequency of each label in the training set. The contribution from each feature is then combined with this prior probability, to arrive at a likelihood estimate for each label. The label whose likelihood estimate is the highest is then assigned to the input value. Figure 6-6 illustrates this process. Figure 6-6. An abstract illustration of the procedure used by the naive Bayes classifier to choose the topic for a document. In the training corpus, most documents are automotive, so the classifier starts out at a point closer to the “automotive” label. But it then considers the effect of each feature. In this example, the input document contains the word dark, which is a weak indicator for murder mysteries, but it also contains the word football, which is a strong indicator for sports documents. After every feature has made its contribution, the classifier checks which label it is closest to, and assigns that label to the input. Individual features make their contribution to the overall decision by “voting against” labels that don’t occur with that feature very often. In particular, the likelihood score for each label is reduced by multiplying it by the probability that an input value with that label would have the feature. For example, if the word run occurs in 12% of the sports documents, 10% of the murder mystery documents, and 2% of the automotive documents, then the likelihood score for the sports label will be multiplied by 0.12, the likelihood score for the murder mystery label will be multiplied by 0.1, and the likeli- hood score for the automotive label will be multiplied by 0.02. The overall effect will be to reduce the score of the murder mystery label slightly more than the score of the sports label, and to significantly reduce the automotive label with respect to the other two labels. This process is illustrated in Figures 6-7 and 6-8. 246 | Chapter 6: Learning to Classify Text Figure 6-7. Calculating label likelihoods with naive Bayes. Naive Bayes begins by calculating the prior probability of each label, based on how frequently each label occurs in the training data. Every feature then contributes to the likelihood estimate for each label, by multiplying it by the probability that input values with that label will have that feature. The resulting likelihood score can be thought of as an estimate of the probability that a randomly selected value from the training set would have both the given label and the set of features, assuming that the feature probabilities are all independent. Figure 6-8. A Bayesian Network Graph illustrating the generative process that is assumed by the naive Bayes classifier. To generate a labeled input, the model first chooses a label for the input, and then it generates each of the input’s features based on that label. Every feature is assumed to be entirely independent of every other feature, given the label. Underlying Probabilistic Model Another way of understanding the naive Bayes classifier is that it chooses the most likely label for an input, under the assumption that every input value is generated by first choosing a class label for that input value, and then generating each feature, entirely independent of every other feature. Of course, this assumption is unrealistic; features are often highly dependent on one another. We’ll return to some of the consequences of this assumption at the end of this section. This simplifying assumption, known as the naive Bayes assumption (or independence assumption), makes it much easier 6.5 Naive Bayes Classifiers | 247 to combine the contributions of the different features, since we don’t need to worry about how they should interact with one another. Based on this assumption, we can calculate an expression for P(label|features), the probability that an input will have a particular label given that it has a particular set of features. To choose a label for a new input, we can then simply pick the label l that maximizes P(l|features). To begin, we note that P(label|features) is equal to the probability that an input has a particular label and the specified set of features, divided by the probability that it has the specified set of features: (2) P(label|features) = P(features, label)/P(features) Next, we note that P(features) will be the same for every choice of label, so if we are simply interested in finding the most likely label, it suffices to calculate P(features, label), which we’ll call the label likelihood. If we want to generate a probability estimate for each label, rather than just choosing the most likely label, then the easiest way to compute P(features) is to simply calculate the sum over labels of P(features, label): (3) P(features) = Σlabel ∈ labels P(features, label) The label likelihood can be expanded out as the probability of the label times the prob- ability of the features given the label: (4) P(features, label) = P(label) × P(features|label) Furthermore, since the features are all independent of one another (given the label), we can separate out the probability of each individual feature: (5) P(features, label) = P(label) × ⊓f ∈ featuresP(f|label) This is exactly the equation we discussed earlier for calculating the label likelihood: P(label) is the prior probability for a given label, and each P(f|label) is the contribution of a single feature to the label likelihood. Zero Counts and Smoothing The simplest way to calculate P(f|label), the contribution of a feature f toward the label likelihood for a label label, is to take the percentage of training instances with the given label that also have the given feature: (6) P(f|label) = count(f, label)/count(label) 248 | Chapter 6: Learning to Classify Text However, this simple approach can become problematic when a feature never occurs with a given label in the training set. In this case, our calculated value for P(f|label) will be zero, which will cause the label likelihood for the given label to be zero. Thus, the input will never be assigned this label, regardless of how well the other features fit the label. The basic problem here is with our calculation of P(f|label), the probability that an input will have a feature, given a label. In particular, just because we haven’t seen a feature/label combination occur in the training set, doesn’t mean it’s impossible for that combination to occur. For example, we may not have seen any murder mystery documents that contained the word football, but we wouldn’t want to conclude that it’s completely impossible for such documents to exist. Thus, although count(f,label)/count(label) is a good estimate for P(f|label) when count(f, label) is relatively high, this estimate becomes less reliable when count(f) becomes smaller. Therefore, when building naive Bayes models, we usually employ more so- phisticated techniques, known as smoothing techniques, for calculating P(f|label), the probability of a feature given a label. For example, the Expected Likelihood Estima- tion for the probability of a feature given a label basically adds 0.5 to each count(f,label) value, and the Heldout Estimation uses a heldout corpus to calculate the relationship between feature frequencies and feature probabilities. The nltk.prob ability module provides support for a wide variety of smoothing techniques. Non-Binary Features We have assumed here that each feature is binary, i.e., that each input either has a feature or does not. Label-valued features (e.g., a color feature, which could be red, green, blue, white, or orange) can be converted to binary features by replacing them with binary features, such as “color-is-red”. Numeric features can be converted to bi- nary features by binning, which replaces them with features such as “4