ttul.org: Inline for Python

About PyInline

From the README:

The PyInline module allows you to put source code from other programming languages directly "inline" in a Python script or module. The code is automatically compiled as needed, and then loaded for immediate access from Python. PyInline is the Python equivalent of Brian Ingerson's Inline module for Perl ( inline.perl.org); indeed, this README file plagerizes Brian's documentation almost verbatim.

For example,

import PyInline, __main__

m = PyInline.build(code="""
  double my_add(double a, double b) {
    return a + b;
  }""",
  targetmodule=__main__, language="C")
		      
print my_add(4.5, 5.5) # Should print out "10.0"

Getting PyInline

PyInline is hosted on SourceForge. The latest release of PyInline is always available on the PyInline download page.

SourceForge Logo

News

Hats of to PyRex

November 20, 2004
PyRex is an alternative to PyInline that works in a slightly different way. Rather than allowing you to embed C source code within your Python script, PyRex extends the Python syntax a bit to allow you to specify the type of your variables. PyRex then pre-compiles you ".pyx" file, generating the appropriate Python and C code to create a hybrid module that you can use in your other Python code.

An Optimization Anecdote

March 9, 2002

Mitchell Charity writes in a USENET article on comp.lang.python recently:


Guido's "Python Patterns - An Optimization Anecdote"
(http://www.python.org/doc/essays/list2str.html) explores the
performance of various ways of converting a list of integers
into a string of characters.  Such as the straightforward

   def f1(list):
       string = ""
       for item in list:
           string = string + chr(item)
       return string

and the fastest version

   import array
   def f7(list):
       return array.array('B', list).tostring()

   # Use 'B' with Python2.2, 'b' with Python1.5

Guido writes
  [...] I had wanted to try one more approach: write the whole
  function in C.  [...]  Given the effort of writing and testing an
  extension (compared to whipping up those Python one-liners), as well
  as the dependency on a non-standard Python extension, I decided not
  to pursue this option...

I wanted to revisit this, because with PyInline, the effort has
diminished, and a non-standard extension is not required (just PyInline).
And, as Guido points out,

  If you feel the need for speed, [...] - you can't beat a loop
  written in C.

So, here then is a new version, f8(), which uses PyInline.

from PyInline import C
import __main__
C.Builder(code="""
PyObject* f8(PyObject *list)
{
    PyObject* string = NULL;
    unsigned char* buffer = NULL;
    int size, index;

#define RAISE(errtype,msg) { PyErr_SetString(errtype,msg); RE_RAISE; }
#define RE_RAISE           { Py_XDECREF(string); return NULL; }

    if(!PyList_Check(list))
       RAISE(PyExc_TypeError,"a list is required");

    size = PyList_Size(list);
    if(size < 0) RE_RAISE;

    string = PyString_FromStringAndSize(NULL, size);
    if(string == NULL) RE_RAISE;

    buffer = PyString_AsString(string);
    if(buffer == NULL) RE_RAISE;

    for(index = 0; index < size; index++) {
        PyObject *item = NULL;
        long number;

        item = PyList_GetItem(list,index);
        if(item == NULL) RE_RAISE;

        number = PyInt_AsLong(item);
        if(PyErr_Occurred() != NULL) RE_RAISE;

        if(number < 0 || number > 255)
            RAISE(PyExc_TypeError,"an integer was out of range");

        buffer[index] = (unsigned char) number;
    }

    return string;
}
""",targetmodule=__main__).build()

The test jig requires a slight change
   #for f in testfuncs: print f.func_name, f(testdata)
   for f in testfuncs: print f(testdata)
because PyInline currently generates glue which does not understand
f.func_name.

So, what is the result?

First, the downsides.  f8() is a C extension, and so only works with
C-based Pythons.  PyInline invokes an external C compiler, and
cruftily caches the result in the filesystem.  And, as PyInline is
alpha code, there are little misfeatures like not being able to use
func_name.  Oh yes, and being written in C, f8() is perhaps two orders
of magnitude more complex than the Python versions.  Though, this last
is perhaps a worst case, as f8() is almost all interface code, with
maybe just three lines of algorithm.  In more typical use, one might
see a one order of magnitude cost.  And with objects which provide a
nice C api, maybe even less.

The upside is f8() runs 5 times faster than f7(), the best alternative.
And uses half the memory.  That's 50 times faster than the naive versions.
And again, when python interface code is less dominant, the speedup
can be dramatically larger.

So, restoring the phrase I clipped out of the quote above, "If you
feel the need for speed, go for built-in functions - you can't beat a
loop written in C.".  And if there isn't one already, you might
consider creating one with PyInline.

Version 0.03 released

September 18, 2001

PyInline 0.03 (5869 bytes) provides Python 2.0 compatibility, an improved and simplified API, better parsing, and a much improved foundation for adding support for languages other than C. The canonical PyInline signature now looks like this:

import PyInline as P
P.build(code=r'void ja(char* J){printf("Just another %s hacker\n",J);}',\
language="C").ja("PyInline")

Version 0.02 released

August 22, 2001

PyInline 0.02 fixes some bugs and lets you do cool things like this:

from PyInline import C

# Use PyInline to build a module object.
# Then call methods on the module object.

# 1. Create a module builder.
x = C.Builder(code="""
#include <stdio.h>
void ja(char *str) {
  printf("Just another %s hacker\n", str);
}
""")

# 2. Build the module. build() returns a new module object.
m = x.build()

# 3. Call a method in the newly built module.
m.ja("Inline")
Version 0.02 also comes with two new examples, including a dynamic code generator which demonstrates PyInline's usefulness in scientific applications (see examples/example3.py).

Copyright © 2001 Ken Simpson, ksimpson@ttul.org