1.2 The _ _builtin_ _ Module

This module contains built-in functions that are automatically available in all Python modules. You usually don't have to import this module; Python does that for you when necessary.

1.2.1 Calling a Function with Arguments from a Tuple or Dictionary

Python allows you to build function argument lists on the fly. Just put all the arguments in a tuple, and call the built-in apply function, as illustrated in Example 1-1.

Example 1-1. Using the apply Function

File: builtin-apply-example-1.py

def function(a, b):
    print a, b

apply(function, ("whither", "canada?"))
apply(function, (1, 2 + 3))

whither canada?
1 5

To pass keyword arguments to a function, you can use a dictionary as the third argument to apply, as shown in Example 1-2.

Example 1-2. Using the apply Function to Pass Keyword Arguments

File: builtin-apply-example-2.py

def function(a, b):
    print a, b

apply(function, ("crunchy", "frog"))
apply(function, ("crunchy",), {"b": "frog"})
apply(function, (), {"a": "crunchy", "b": "frog"})

crunchy frog
crunchy frog
crunchy frog

One common use for apply is to pass constructor arguments from a subclass on to the base class, especially if the constructor takes a lot of arguments. See Example 1-3.

Example 1-3. Using the apply Function to Call Base Class Constructors

File: builtin-apply-example-3.py

class Rectangle:
    def _ _init_ _(self, color="white", width=10, height=10):
        print "create a", color, self, "sized", width, "x", height

class RoundedRectangle(Rectangle):
    def _ _init_ _(self, **kw):
        apply(Rectangle._ _init_ _, (self,), kw)

rect = Rectangle(color="green", height=100, width=100)
rect = RoundedRectangle(color="blue", height=20)

create a green <Rectangle instance at 8c8260> sized 100 x 100
create a blue <RoundedRectangle instance at 8c84c0> sized 10 x 20

Python 2.0 provides an alternate syntax. Instead of apply, you can use an ordinary function call, and use * to mark the tuple, and ** to mark the dictionary.

The following two statements are equivalent:

result = function(*args, **kwargs)
result = apply(function, args, kwargs)

1.2.2 Loading and Reloading Modules

If you've written a Python program larger than just a few lines, you know that the import statement is used to import external modules (you can also use the from-import version). What you might not know already is that import delegates the actual work to a built-in function called _ _import_ _.

The trick is that you can call this function directly. This can be handy if you have the module name in a string variable, as in Example 1-4, which imports all modules whose names end with "-plugin":

Example 1-4. Using the _ _import_ _ Function to Load Named Modules

File: builtin-import-example-1.py

import glob, os

modules = []

for module_file in glob.glob("*-plugin.py"):
    try:
        module_name, ext = os.path.splitext(os.path.basename(module_file))
        module = _ _import_ _(module_name)
        modules.append(module)
    except ImportError:
        pass # ignore broken modules

# say hello to all modules
for module in modules:
    module.hello()

example-plugin says hello

Note that the plug-in modules have hyphens. This means that you cannot import such a module using the ordinary import command, since you cannot have hyphens in Python identifiers.

Example 1-5 shows the plug-in used in Example 1-4.

Example 1-5. A Sample Plug-in

File: example-plugin.py

def hello():
    print "example-plugin says hello"

Example 1-6 shows how to get a function object, given that the module and function name are strings.

Example 1-6. Using the _ _import_ _ Function to Get a Named Function

File: builtin-import-example-2.py

def getfunctionbyname(module_name, function_name):
    module = _ _import_ _(module_name)
    return getattr(module, function_name)

print repr(getfunctionbyname("dumbdbm", "open"))

<function open at 794fa0>

You can also use this function to implement lazy module loading. In Example 1-7, the string module is imported when it is first used.

Example 1-7. Using the _ _import_ _ Function to Implement Lazy Import

File: builtin-import-example-3.py

class LazyImport:
    def _ _init_ _(self, module_name):
        self.module_name = module_name
        self.module = None
    def _ _getattr_ _(self, name):
        if self.module is None:
            self.module = _ _import_ _(self.module_name)
        return getattr(self.module, name)

string = LazyImport("string")

print string.lowercase

abcdefghijklmnopqrstuvwxyz

Python provides some basic support for reloading modules that you've already imported. Example 1-8 loads the hello.py file three times.

Example 1-8. Using the reload Function

File: builtin-reload-example-1.py

import hello
reload(hello)
reload(hello)

hello again, and welcome to the show
hello again, and welcome to the show
hello again, and welcome to the show

reload uses the module name associated with the module object, not the variable name. Even if you've renamed the original module, reload can still find it.

Note that when you reload a module, it is recompiled, and the new module replaces the old one in the module dictionary. However, if you have created instances of classes defined in that module, those instances will still use the old implementation.

Likewise, if you've used from-import to create references to module members in other modules, those references will not be updated.

1.2.3 Looking in Namespaces

The dir function returns a list of all members of a given module, class, instance, or other type. It's probably most useful when you're working with an interactive Python interpreter, but can also come in handy in other situations. Example 1-9 shows the dir function in use.

Example 1-9. Using the dir Function

File: builtin-dir-example-1.py

def dump(value):
    print value, "=>", dir(value)

import sys

dump(0)
dump(1.0)
dump(0.0j) # complex number
dump([]) # list
dump({}) # dictionary
dump("string")
dump(len) # function
dump(sys) # module

0 => []
1.0 => []
0j => ['conjugate', 'imag', 'real']
[] => ['append', 'count', 'extend', 'index', 'insert',
    'pop', 'remove', 'reverse', 'sort']
{} => ['clear', 'copy', 'get', 'has_key', 'items',
    'keys', 'update', 'values']
string => []
<built-in function len> => ['_ _doc_ _', '_ _name_ _', '_ _self_ _']
<module 'sys' (built-in)> => ['_ _doc_ _', '_ _name_ _',
    '_ _stderr_ _', '_ _stdin_ _', '_ _stdout_ _', 'argv',
    'builtin_module_names', 'copyright', 'dllhandle',
    'exc_info', 'exc_type', 'exec_prefix', 'executable',
...

In Example 1-10, the getmember function returns all class-level attributes and methods defined by a given class.

Example 1-10. Using the dir Function to Find All Members of a Class

File: builtin-dir-example-2.py

class A:
    def a(self):
        pass
    def b(self):
        pass

class B(A):
    def c(self):
        pass
    def d(self):
        pass

def getmembers(klass, members=None):
    # get a list of all class members, ordered by class
    if members is None:
        members = []
    for k in klass._ _bases_ _:
        getmembers(k, members)
    for m in dir(klass):
        if m not in members:
            members.append(m)
    return members

print getmembers(A)
print getmembers(B)
print getmembers(IOError)

['_ _doc_ _', '_ _module_ _', 'a', 'b']
['_ _doc_ _', '_ _module_ _', 'a', 'b', 'c', 'd']
['_ _doc_ _', '_ _getitem_ _', '_ _init_ _', '_ _module_ _', '_ _str_ _']

Note that the getmembers function returns an ordered list. The earlier a name appears in the list, the higher up in the class hierarchy it's defined. If order doesn't matter, you can use a dictionary to collect the names instead of a list.

The vars function is similar, but it returns a dictionary containing the current value for each member. If you use vars without an argument, it returns a dictionary containing what's visible in the current local namespace, as shown in Example 1-11.

Example 1-11. Using the vars Function

File: builtin-vars-example-1.py

book = "library2"
pages = 250
scripts = 350


print "the %(book)s book contains more than %(scripts)s scripts" % vars()

the library book contains more than 350 scripts

1.2.4 Checking an Object's Type

Python is a dynamically typed language, which means that a given variable can be bound to values of different types on different occasions. In the following example, the same function is called with an integer, a floating point value, and a string:

def function(value):
    print value
function(1)
function(1.0)
function("one")

The type function (shown in Example 1-12) allows you to check what type a variable has. This function returns a type descriptor, which is a unique object for each type provided by the Python interpreter.

Example 1-12. Using the type Function

File: builtin-type-example-1.py

def dump(value):
    print type(value), value

dump(1)
dump(1.0)
dump("one")

<type 'int'> 1
<type 'float'> 1.0
<type 'string'> one

Each type has a single corresponding type object, which means that you can use the is operator (object identity) to do type testing (as shown in Example 1-13).

Example 1-13. Using the type Function with Filenames and File Objects

File: builtin-type-example-2.py

def load(file):
    if isinstance(file, type("")):
        file = open(file, "rb")
    return file.read()

print len(load("samples/sample.jpg")), "bytes"
print len(load(open("samples/sample.jpg", "rb"))), "bytes"


4672 bytes
4672 bytes

The callable function, shown in Example 1-14, checks if an object can be called (either directly or via apply). It returns true for functions, methods, lambda expressions, classes, and class instances that define the _ _call_ _ method.

Example 1-14. Using the callable Function

File: builtin-callable-example-1.py

def dump(function):
    if callable(function):
        print function, "is callable"
    else:
        print function, "is *not* callable"

class A:
    def method(self, value):
        return value

class B(A):
    def _ _call_ _(self, value):
        return value

a = A()
b = B()

dump(0) # simple objects
dump("string")
dump(callable)
dump(dump) # function

dump(A) # classes
dump(B)
dump(B.method)

dump(a) # instances
dump(b)
dump(b.method)

0 is *not* callable
string is *not* callable
<built-in function callable> is callable
<function dump at 8ca320> is callable
A is callable
B is callable
<unbound method A.method> is callable
<A instance at 8caa10> is *not* callable
<B instance at 8cab00> is callable
<method A.method of B instance at 8cab00> is callable

Note that the class objects (A and B) are both callable; if you call them, they create new objects. However, instances of class A are not callable, since that class doesn't have a _ _call_ _ method.

You'll find functions to check if an object is of any of the built-in number, sequence, or dictionary types in the operator module. However, since it's easy to create a class that implements (for example, the basic sequence methods), it's usually a bad idea to use explicit type testing on such objects.

Things get even more complicated when it comes to classes and instances. Python doesn't treat classes as types per se; instead, all classes belong to a special class type, and all class instances belong to a special instance type.

This means that you cannot use type to test if an instance belongs to a given class; all instances have the same type! To solve this, you can use the isinstance function, which checks if an object is an instance of a given class (or of a subclass to it). Example 1-15 illustrates the isinstance function.

Example 1-15. Using the isinstance Function

File: builtin-isinstance-example-1.py

class A:
    pass

class B:
    pass

class C(A):
    pass

class D(A, B):
    pass

def dump(object):
    print object, "=>",
    if isinstance(object, A):
        print "A",
    if isinstance(object, B):
        print "B",
    if isinstance(object, C):
        print "C",
    if isinstance(object, D):
        print "D",
    print

a = A()
b = B()
c = C()
d = D()

dump(a)
dump(b)
dump(c)
dump(d)
dump(0)
dump("string")

<A instance at 8ca6d0> => A
<B instance at 8ca750> => B
<C instance at 8ca780> => A C
<D instance at 8ca7b0> => A B D
0 =>
string =>

The issubclass function is similar, but it instead checks whether a class object is the same as a given class, or is a subclass of it. The issubclass function is shown in Example 1-16.

Note that while isinstance accepts any kind of object, the issubclass function raises a TypeError exception if you use it on something that is not a class object.

Example 1-16. Using the issubclass Function

File: builtin-issubclass-example-1.py

class A:
    pass

class B:
    pass

class C(A):
    pass

class D(A, B):
    pass

def dump(object):
    print object, "=>",
    if issubclass(object, A):
        print "A",
    if issubclass(object, B):
        print "B",
    if issubclass(object, C):
        print "C",
    if issubclass(object, D):
        print "D",
    print

dump(A)
dump(B)
dump(C)
dump(D)
dump(0)
dump("string")

A => A
B => B
C => A C
D => A B D
0 =>
Traceback (innermost last):
  File "builtin-issubclass-example-1.py", line 29, in ?
  File "builtin-issubclass-example-1.py", line 15, in dump
TypeError: arguments must be classes

1.2.5 Evaluating Python Expressions

Python provides several ways to interact with the interpreter from within a program. For example, the eval function evaluates a string as if it were a Python expression. You can pass it a literal, simple expression, or use built-in functions, as shown in Example 1-17.

Example 1-17. Using the eval Function

File: builtin-eval-example-1.py

def dump(expression):
    result = eval(expression)
    print expression, "=>", result, type(result)

dump("1")
dump("1.0")
dump("'string'")
dump("1.0 + 2.0")
dump("'*' * 10")
dump("len('world')")

1 => 1 <type 'int'>
1.0 => 1.0 <type 'float'>
'string' => string <type 'string'>
1.0 + 2.0 => 3.0 <type 'float'>
'*' * 10 => ********** <type 'string'>
len('world') => 5 <type 'int'>

If you cannot trust the source from which you got the string, you may get into trouble using eval. For example, someone might use the built-in _ _import_ _ function to load the os module, and then remove files on your disk (as shown in Example 1-18).

Example 1-18. Using the eval Function to Execute Arbitrary Commands

File: builtin-eval-example-2.py

print eval("_ _import_ _('os').getcwd()")
print eval("_ _import_ _('os').remove('file')")

/home/fredrik/librarybook
Traceback (innermost last):
 File "builtin-eval-example-2", line 2, in ?
 File "<string>", line 0, in ?
os.error: (2, 'No such file or directory')

Note that you get an os.error exception, which means that Python actually tried to remove the file!

Luckily, there's a way around this problem. You can pass a second argument to eval, which should contain a dictionary defining the namespace in which the expression is evaluated. Let's pass in an empty namespace:

>>> print eval("_ _import_ _('os').remove('file')", {})
Traceback (innermost last):
  File "<stdin>", line 1, in ?
  File "<string>", line 0, in ?
os.error: (2, 'No such file or directory')

Hmm. We still end up with an os.error exception.

The reason for this is that Python looks in the dictionary before it evaluates the code, and if it doesn't find a variable named _ _builtins_ _ in there (note the plural form), it adds one:

>>> namespace = {}
>>> print eval("_ _import_ _('os').remove('file')", namespace)
Traceback (innermost last):
  File "<stdin>", line 1, in ?
  File "<string>", line 0, in ?
os.error: (2, 'No such file or directory')
>>> namespace.keys()
['_ _builtins_ _']

If you print the contents of the namespace variable, you'll find that they contain the full set of built-in functions.

The solution to this little dilemma isn't far away: since Python doesn't add this item if it is already there, simply add a dummy item called _ _builtins_ _ to the namespace before calling eval, as shown in Example 1-19.

Example 1-19. Using the eval Function to Evaluate Arbitrary Strings Safely

File: builtin-eval-example-3.py

print eval("_ _import_ _('os').getcwd()", {})
print eval("_ _import_ _('os').remove('file')", {"_ _builtins_ _": {}})

/home/fredrik/librarybook
Traceback (innermost last):
  File "builtin-eval-example-3.py", line 2, in ?
  File "<string>", line 0, in ?
NameError: _ _import_ _

Note that this doesn't protect you from CPU or memory-resource attacks (for example, something like eval("'*'*1000000*2*2*2*2*2*2*2*2*2") will most likely cause your program to run out of memory after a while).

1.2.6 Compiling and Executing Code

The eval function only works for simple expressions. To handle larger blocks of code, use the compile and exec functions (as demonstrated in Example 1-20).

Example 1-20. Using the compile Function to Check Syntax

File: builtin-compile-example-1.py

NAME = "script.py"

BODY = """
prnt 'owl-stretching time'
"""

try:
    compile(BODY, NAME, "exec")
except SyntaxError, v:
    print "syntax error:", v, "in", NAME

# syntax error: invalid syntax in script.py

When successful, the compile function returns a code object, which you can execute with the exec statement, as in Example 1-21.

Example 1-21. Compiling and Executing Compiled Code

File: builtin-compile-example-2.py

BODY = """
print 'the ant, an introduction'
"""

code = compile(BODY, "<script>", "exec")

print code

exec code

<code object ? at 8c6be0, file "<script>", line 0>
the ant, an introduction

To generate code on the fly, use the class shown in the Example 1-22. Use the write method to add statements, and the methods indent and dedent to add structure. The class will take care of the rest.

Example 1-22. A Simple Code Generator Tool

File: builtin-compile-example-3.py

import sys, string

class CodeGeneratorBackend:
    "Simple code generator for Python"

    def begin(self, tab="\t"):
        self.code = []
        self.tab = tab
        self.level = 0

    def end(self):
        self.code.append("") # make sure there's a newline at the end
        return compile(string.join(self.code, "\n"), "<code>", "exec")

    def write(self, string):
        self.code.append(self.tab * self.level + string)

    def indent(self):
        self.level = self.level + 1
        # in 2.0 and later, this can be written as: self.level += 1

    def dedent(self):
        if self.level == 0:
            raise SyntaxError, "internal error in code generator"
        self.level = self.level - 1
        # or: self.level -= 1

#
# try it out!

c = CodeGeneratorBackend()
c.begin()
c.write("for i in range(5):")
c.indent()
c.write("print 'code generation made easy!'")
c.dedent()
exec c.end()

code generation made easy!
code generation made easy!
code generation made easy!
code generation made easy!
code generation made easy!

Python also provides a function called execfile, a shortcut for loading code from a file, compiling it, and executing it. Example 1-23 shows how to use and emulate this function.

Example 1-23. Using the execfile Function

File: builtin-execfile-example-1.py

execfile("hello.py")

def EXECFILE(filename, locals=None, globals=None):
    exec compile(open(filename).read(), filename, "exec") in locals, globals

EXECFILE("hello.py")

hello again, and welcome to the show
hello again, and welcome to the show

The contents of the hello.py file used Example 1-23 are shown in Example 1-24.

Example 1-24. The hello.py Script

File: hello.py

print "hello again, and welcome to the show"

1.2.7 Overloading Functions from the _ _builtin_ _ Module

Since Python does not look among the built-in functions until after it has checked the local and module namespace, there may be situations in which you need to explicitly refer to the _ _builtin_ _ module. For instance, the script in Example 1-25 overloads the open function with a version that opens an ordinary file and checks that it starts with a "magic" string. To be able to use the original open function, the script explicitly refers to the function using the module name.

Example 1-25. Explicitly Accessing Functions in the _ _builtin_ _ Module

File: builtin-open-example-1.py

def open(filename, mode="rb"):
    import _ _builtin_ _
    file = _ _builtin_ _.open(filename, mode)
    if file.read(5) not in("GIF87", "GIF89"):
        raise IOError, "not a GIF file"
    file.seek(0)
    return file

fp = open("samples/sample.gif")
print len(fp.read()), "bytes"

fp = open("samples/sample.jpg")
print len(fp.read()), "bytes"

3565 bytes
Traceback (innermost last):
  File "builtin-open-example-1.py", line 12, in ?
  File "builtin-open-example-1.py", line 5, in open
IOError: not a GIF file