15.5 Importing Modules with Automatic End-of-Line Conversions

Credit: David Goodger

15.5.1 Problem

You need to move (or share, via a network-shared filesystem) .py modules between different platforms without worrying about the different ways the platforms represent end-of-line.

15.5.2 Solution

Python's import hooks are mostly used for advanced, complicated tasks, but can sometimes come in handy for a simple system-administration task, such as dealing with this recipe's problem. Put this code in your sitecustomize.py, which can be located anywhere on sys.path, and you'll be able to import Python modules with any Unix, Mac, or Windows line endings:

import ihooks, imp, py_compile

class MyHooks(ihooks.Hooks):
    def load_source(self, name, filename, file=None):
        """ Compile source files with any line ending. """
        if file:
            file.close(  )
        py_compile.compile(filename)    # Line-ending conversion is in here
        cfile = open(filename + (_ _debug_ _ and 'c' or 'o'), 'rb')
        try:
            return self.load_compiled(name, filename, cfile)
        finally:
            cfile.close(  )

class MyModuleLoader(ihooks.ModuleLoader):
    def load_module(self, name, stuff):
        """ Special-case package directory imports """
        file, filename, (suff, mode, type) = stuff
        path = None
        if type == imp.PKG_DIRECTORY:
            stuff = self.find_module_in_dir("_ _init_ _", filename, 0)
            file = stuff[0]             # package/_ _init_ _.py
            path = [filename]
        try:                            # Let superclass handle the rest
            module = ihooks.ModuleLoader.load_module(self, name, stuff)
        finally:
            if file:
                file.close(  )
        if path:
            module._ _path_ _ = path      # necessary for pkg.module imports
        return module

ihooks.ModuleImporter(MyModuleLoader(MyHooks())).install(  )

15.5.3 Discussion

This recipe's code eliminates the need to convert line endings when moving Python source modules between operating systems (or sharing sources via a network-shared filesystem, which is even harder for those who need to do conversions). Put this recipe's code in your sitecustomize.py file, which can be placed anywhere on Python's initial sys.path, and you'll be able to import Python modules with any Unix, Mac, or Windows line endings, on any operating system.

I develop code on a Mac and test it on Windows and Unix-like operating systems. Converting line endings used to be a pain. Delving deep into the innards of the import mechanism and ihooks.py, I was finally able to get this import hook working. No more line-ending conversion needed!

This hard work is performed in Python's standard library module py_compile in the compile function. It normalizes all line endings in the source before arranging for the rest of the compilation process. This recipe exploits the import hooks scheme to ensure that all the importing of sources goes through this function, and thus provides the benefits of normalized handling of line endings, rather than relying on Python's automatic compilation, which requires line endings that are compatible with the platform in use.

15.5.4 See Also

Documentation on the ihooks, imp, and py_compile standard library modules in the Library Reference.