3.22 Replacing Python Code with the Results of Executing That Code

Credit: Joel Gould

3.22.1 Problem

You have a template string that may include embedded Python code, and you need a copy of the template in which any embedded Python code is replaced by the results of executing that code.

3.22.2 Solution

This recipe exploits the ability of the standard function re.sub to call a user-supplied replacement function for each match and to substitute the matched substring with the replacement function's result:

import re
import sys
import string

def runPythonCode(data, global_dict={}, local_dict=None, errorLogger=None):
    """ Main entry point to the replcode module """

    # Encapsulate evaluation state and error logging into an instance:
    eval_state = EvalState(global_dict, local_dict, errorLogger)

    # Execute statements enclosed in [!! .. !!]; statements may be nested by
    # enclosing them in [1!! .. !!1], [2!! .. !!2], and so on:
    data = re.sub(r'(?s)\[(?P<num>\d?)!!(?P<code>.+?)!!(?P=num)\]',
        eval_state.exec_python, data)

    # Evaluate expressions enclosed in [?? .. ??]:
    data = re.sub(r'(?s)\[\?\?(?P<code>.+?)\?\?\]',
        eval_state.eval_python, data)

    return data

class EvalState:
    """ Encapsulate evaluation state, expose methods to execute/evaluate """

    def _ _init_ _(self, global_dict, local_dict, errorLogger):
        self.global_dict = global_dict
        self.local_dict = local_dict
        if errorLogger:
            self.errorLogger = errorLogger
        else:
            # Default error "logging" writes error messages to sys.stdout
            self.errorLogger = sys.stdout.write

        # Prime the global dictionary with a few needed entries:
        self.global_dict['OUTPUT'] = OUTPUT
        self.global_dict['sys'] = sys
        self.global_dict['string'] = string
        self.global_dict['_ _builtins_ _'] = _ _builtins_ _

    def exec_python(self, result):
        """ Called from the 1st re.sub in runPythonCode for each block of
        embedded statements. Method's result is OUTPUT_TEXT (see also the OUTPUT
        function later in the recipe). """

        # Replace tabs with four spaces; remove first line's indent from all lines
        code = result.group('code')
        code = string.replace(code, '\t', '    ')
        result2 = re.search(r'(?P<prefix>\n[ ]*)[#a-zA-Z0-9''"]', code)
        if not result2:
            raise ParsingError, 'Invalid template code expression: ' + code
        code = string.replace(code, result2.group('prefix'), '\n')
        code = code + '\n'

        try:
            self.global_dict['OUTPUT_TEXT'] = ''
            if self.local_dict:
                exec code in self.global_dict, self.local_dict
            else:
                exec code in self.global_dict
            return self.global_dict['OUTPUT_TEXT']
        except:
            self.errorLogger('\n---- Error parsing statements: ----\n')
            self.errorLogger(code)
            self.errorLogger('\n------------------------\n')
            raise

    def eval_python(self, result):
        """ Called from the 2nd re.sub in runPythonCode for each embedded
        expression. The method's result is the expr's value as a string. """
        code = result.group('code')
        code = string.replace(code, '\t', '    ')

        try:
            if self.local_dict:
                result = eval(code, self.global_dict, self.local_dict)
            else:
                result = eval(code, self.global_dict)
            return str(result)
        except:
            self.errorLogger('\n---- Error parsing expression: ----\n')
            self.errorLogger(code)
            self.errorLogger('\n------------------------\n')
            raise

def OUTPUT(data):
    """ May be called from embedded statements: evaluates argument 'data' as
    a template string, appends the result to the global variable OUTPUT_TEXT """

    # a trick that's equivalent to sys._getframe in Python 2.0 and later but
    # also works on older versions of Python...:
    try: raise ZeroDivisionError
    except ZeroDivisionError:
        local_dict = sys.exc_info(  )[2].tb_frame.f_back.f_locals
        global_dict  = sys.exc_info(  )[2].tb_frame.f_back.f_globals

    global_dict['OUTPUT_TEXT'] = global_dict['OUTPUT_TEXT'] + runPythonCode(
        data, global_dict, local_dict)

3.22.3 Discussion

This recipe was originally designed for dynamically creating HTML. It takes a template, which is a string that may include embedded Python statements and expressions, and returns another string, in which any embedded Python is replaced with the results of executing that code. I originally designed this code to build my home page. Since then, I have used the same code for a CGI-based web site and for a documentation-generation program.

Templating, which is what this recipe does, is a very popular task in Python, for which you can find any number of existing Pythonic solutions. Many templating approaches aim specifically at the task of generating HTML (or, occasionally, other forms of structured text). Others, such as this recipe, are less specialized, and thus can be simultaneously wider in applicability and simpler in structure. However, they do not offer HTML-specific conveniences. See Recipe 3.23 for another small-scale approach to templating with general goals that are close to this one's but are executed in a rather different style.

Usually, the input template string is taken directly from a file, and the output expanded string is written to another file. When using CGI, the output string can be written directly to sys.stdout, which becomes the HTML displayed in the user's browser when it visits the script.

By passing in a dictionary, you control the global namespace in which the embedded Python code is run. If you want to share variables with the embedded Python code, insert the names and values of those variables into the global dictionary before calling runPythonCode. When an uncaught exception is raised in the embedded code, a dump of the code being evaluated is first written to stdout (or through the errorLogger function argument, if specified) before the exception is propagated to the routine that called runPythonCode.

This recipe handles two different types of embedded code blocks in template strings. Code inside [?? ??] is evaluated. Such code should be an expression and should return a string, which will be used to replace the embedded Python code. Code inside [!! !!] is executed. That code is a suite of statements, and it is not expected to return anything. However, you can call OUTPUT from inside embedded code, to specify text that should replace the executed Python code. This makes it possible, for example, to use loops to generate multiple blocks of output text.

Here is an interactive-interpreter example of using this replcode.py module:

>>> import replcode
>>> input_text = """
...     Normal line.
...     Expression [?? 1+2 ??].
...     Global variable [?? variable ??].
...     [!!
...         def foo(x):
...                 return x+x !!].
...     Function [?? foo('abc') ??].
...     [!!
...         OUTPUT('Nested call [?? variable ??]') !!].
...     [!!
...         OUTPUT('''Double nested [1!!
...                       myVariable = '456' !!1][?? myVariable ??]''') !!].
... """
>>> global_dict = { 'variable': '123' }
>>> output_text = replcode.runPythonCode(input_text, global_dict)
>>> print output_text

    Normal line.
    Expression 3.
    Global variable 123.
    .
    Function abcabc.
    Nested call 123.
    Double nested 456.

3.22.4 See Also

Recipe 3.23.