Python 3.5 is on its way

Please consider subscribing to LWN Subscriptions are the lifeblood of LWN.net. If you appreciate this content and would like to see more of it, your subscription will help to ensure that LWN continues to thrive. Please visit this page to join up and keep LWN on the net.

It has been nearly a year and a half since the last major Python release, which was 3.4 in March 2014—that means it is about time for Python 3.5. We looked at some of the new features in 3.4 at the time of its first release candidate, so the announcement of the penultimate beta release for 3.5 seems like a good time to see what will be coming in the new release. Some of bigger features are new keywords to support coroutines, an operator for matrix multiplication, and support for Python type checking, but there are certainly more. Python 3.5 is currently scheduled for a final release in September.

As usual, the "What's New In Python 3.5" document is the place to start for information about the release. At this point, it is still in draft form, with significant updates planned over the next couple of months. But there is lots to digest in what's there already.

Type hints

One of the more discussed features for 3.5 only showed up in the first beta back in May: type hints (also known as PEP 484). It provides an way to optionally annotate the types of Python functions, arguments, and variables in a way that can be used by various tools. Type hints have been championed by Python benevolent dictator for life (BDFL) Guido van Rossum and came about rather late in the 3.5 development cycle, so there was always a bit of a question whether it would make the feature freeze that came with the first beta release.

But, just prior to that deadline, BDFL-Delegate Mark Shannon accepted the PEP on May 22, with a nod to some of the opposition to the feature that has arisen:

Shannon continued by noting that other languages have an operator-overloading feature that often get used in ugly ways, but that Python's is generally used sensibly. That's because of Python's culture, which promotes the idea that "readability matters". He concluded: "Python is your language, please use type-hints responsibly :)".

Matrix multiplication

A new binary operator will be added to Python to support matrix multiplication (specified by PEP 465). "@" is a new infix operator that shares its precedence with the standard "*" operator for regular multiplication. The "@=" operator will perform the matrix multiplication and assignment, as the other, similar operators do.

The @ operator will be implemented with two "dunder" (double underscore) methods on objects: __matmul__() and __rmatmul__() (the latter is for right-side matrix multiplication when the left-side object does not support @). Unsurprisingly, @= is handled with the __imatmul__() method. That is all the new feature defines, since Python does not have a matrix type, nor does the language define what the @ operator actually does. That means no standard library or builtin types are being changed to support matrix multiplication, though a number of different scientific and numeric Python projects have reached agreement on the intended use and semantics of the operator.

Currently, developers of libraries like NumPy have to make a decision about how to implement Python's multiplication operator (*) for matrices. All of the other binary operators (e.g. +, -, /) can only reasonably be defined element-wise (i.e. applying the operation to corresponding elements in each operand), but multiplication is special in that both the element-wise and the specific 2D matrix varieties are useful. So, NumPy and others have resorted to using a method for 2D matrix multiplication, which detracts from readability.

The only other use of @ in Python is at the start of a decorator, so there will be no confusion in parsing the two different ways of using it. Soon, users of NumPy will be able to multiply matrices in a straightforward way using statements like:

    z = a @ b
    a @= b

async and await

Another fast-moving feature adds new keywords for coroutine support to the language. PEP 492 went from being proposed in early April to being accepted and having its code merged in early May. The idea is to make it easier to work with coroutines in Python, as the new async and await keywords don't really add new functionality that couldn't be accomplished with existing language constructs. But they do provide readability improvements, which is one of the main drivers of the addition.

Essentially, functions, for loops, and with statements can be declared as asynchronous (i.e. may suspend their execution) by adding the async keyword:

    async def foo(...):
        ...

    async for data in cursor:
        ...

    async with lock:
        ...
	await foo()
	...

The await statement that can be seen in the with example is similar to the yield from statement that was added to Python 3.3 (described in PEP 380). It suspends execution until the awaitable foo() completes and returns its result.

While async and await will eventually become full-fledged keywords in Python, that won't happen until Python 3.7 in roughly three years. The idea is to allow programmers time to switch away from using those two statement names as variables, which is not allowed for Python keywords. As it turns out, await can only appear in async constructs, so the parser will keep track of its context and treat those strings as variables outside of those constructs. It is a clever trick that allows the language to "softly deprecate" the keywords.

Zip applications

Over the last few years there has been a lot of discussion on python-dev, python-ideas, and elsewhere about how to easily distribute Python programs; the zipapp module (specified in PEP 441) is a step along that path. Partly, the PEP authors simply want to promote a feature that was added back in the Python 2.6 days. It provides the ability to execute a ZIP format file that contains a collection of Python files, including a __main__.py, which is where the execution starts.

Besides just publicizing a "relatively unknown" language feature, the zipapp module would be added to provide some tools to help create and maintain "Python Zip Applications", which is the formal name for these application bundles. The bundles will have a .pyz extension for console applications, while windowed applications (i.e. won't require a console on Windows) will end in .pyzw . The Windows installer will associate those extensions with the Python launcher. It is hoped that .pyz will be connected with Python on Unix-like systems (by way of the MIME-info database), but that is not directly under the control of the Python developers.

The appropriate interpreter (i.e. Python 2 or Python 3) can be associated with the file by prepending a "shebang" line to the ZIP format (which ZIP archivers will simply skip). For example (from the PEP):

    #!/usr/bin/env python3
    #  Python application packed with zipapp module
    (binary contents of archive)

In addition, the zipapp module can be directly accessed from the command line. It provides two helper functions (create_archive() and get_interpreter()) but will likely be used as follows:

    $ python -m zipapp dirname

That creates a dirname.pyz file that contains the files in the directory (which must include a __main__.py). There are also command-line options that govern the archive name, the interpreter to use, or the main function to call from a __main__.py that zipapp will generate.

Formatting bytes

The distinction between bytes and strings in Python 3 is one of the defining characteristics of the language. That was done to remove some of the ambiguities and "magic" from handling Unicode that are present in Python 2. "Bytes" are simply immutable sequences of integers, each of which is in the range 0–255, while "bytearrays" are their mutable counterpart. There is no interpretation placed on the byte types; strings, on the other hand, always contain Unicode.

But there are a number of "wire protocols" that combine binary and textual data (typically ASCII), so when writing code to deal with those, it would be convenient to be able to use the Python string formatting operations to do so. Today, though, trying to interpolate an integer value into a bytes type does not work:

    >>> b'%d' % 5
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: unsupported operand type(s) for %: 'bytes' and 'int'

The adoption of PEP 461 will change that. All of the numeric formatting codes, when applied to bytes, will be treated as if they were:

    ('%x' % val).encode('ascii')

Where x is any of the dozen or so numeric formatting codes. In addition, %c will insert a single byte (either from an integer in the 0–255 range or from a byte type of length 1) and %b can be used to interpolate a series of bytes. Neither of those will accept string types since converting a string to bytes requires an encoding, which Python 3 will not try to guess.

In addition, bytes and bytearrays will be getting a hex() method. That will allow turning bytes into hexadecimal strings:

    >>> bytes([92, 83, 80, 255]).hex()
    '5c5350ff'

The grab bag

There are, of course, lots more features coming in Python 3.5—way more than we can cover here—but there are some additional ones that caught our eye and we will look at briefly. For example, the ordered dictionary in collections.OrderedDict has been reimplemented in C, which provides a 4–100x performance increase. PEP 485, which has been accepted for 3.5, proposes adding a function to test approximate equality to both the math and cmath modules: math.is_close() and cmath.is_close().

The "system call interrupted" error code on POSIX systems (EINTR, which gets turned into the Python InterruptedError exception) is often unexpected by applications when they are making various I/O calls, so those exceptions may not be handled correctly. It would make more sense to centralize the handling of EINTR. So, in the future, the low-level system call wrappers will automatically retry operations that fail with EINTR (which is a feature that is described in PEP 475).

The process of initializing extension modules (which are implemented as shared libraries), built-in modules, and imported modules is subtly different in ways that can lead to import loops and other problems. PEP 489 was proposed to regularize this process so that all of the different "import-like" mechanisms behave in the same way; it will be part of Python 3.5.

Eliminating .pyo files is the subject of PEP 488 and it will be implemented for the upcoming release. Those files were meant to hold optimized versions of the Python bytecode (unoptimized bytecode is usually placed into .pyc files), but the extension was used for multiple different optimization levels, which meant manually removing all .pyo files to ensure they were all at the same level. In the future, these __pycache__ file names will all end in .pyc, but will indicate both the optimization level and interpreter used in the name, for example: foo.cpython-35.opt-2.pyc.

Lastly, at least for our look, Python 3.5 will add an os.scandir() function to provide a "better and faster directory iterator". The existing os.walk() does unnecessary work that results in roughly 2x the number of system calls required. os.scandir() will return a generator that produces file names as needed, rather than as one big list, and os.walk() will be implemented using the new function. That will result in performance increases of 2–3x on POSIX systems and 8–9x for Windows.

As can be seen, there is lots of good stuff coming in Python 3.5. Over the next few months, testing of the betas and release candidates should hopefully shake out all but the most obscure bugs, leading to a solid 3.5 release in mid-September.

---

(Log in to post comments)