Module serial

source code

Service to handle serialization and deserialization of python objects.

Object serialization is useful for long term storage, interlanguage communication and network transmission. In all cases, the process involves an initial encode() followed by a later decode().

The following properties are desirable for serialization/deserialization:

1. human readable so that disaster recovery is possible
2. readable/writable by other languages and environments
3. support for numerics: complex, nan, inf, arrays, full precision
4. version support: load object into newer versions of the program even if the class structure has changed
5. refactoring support: load object into newer versions of the program even if the classes have been moved or renamed
6. support self-referential data structures (not yet)

Python's builtin serialization, pickle/cPickle, cannot meet these needs. It is python specific, and not friendly to human readers or readers from other environments such as IDL which may want to load or receive data from a python program. Pickle inf/nan doesn't work on windows --- some of our models may use inf data, and some of our results may be nan. pickle has minimal support for versioning: users can write __setstate__ which accepts a dictionary and adjusts it accordingly. Beware though that version must be an instance variable rather than a class variable, since class variables are not seen by pickle. If the class is renamed, then pickle can do nothing to recover it.

Instead of pickle, we break the problem into parts: structucture and encoding. A pair of functions deconstruct and reconstruct work directly with the structure. Deconstruct extracts the state of the python object defined using a limited set of python primitives. Reconstruct takes an extracted state and rebuilds the complete python object. See documentation on the individual functions for details.

Object persistence for long term storage places particular burdens on the serialization protocol. In particular, the class may have changed since the instance was serialized. To aid the process of maintaining classes over the long term, the class definition can contain the following magic names:

__version__

Strict version number of the class. See isnewer() for details, or distutils.version.StrictVersion.

__factory__

Name of a factory function to return an empty instance of the class. This will be stored as the class name, and should include the complete path so that it can be imported by python. The class will be populated by the normal restore mechanism for that class, using either __reconstruct__, __setstate__ or setting the class dictionary.

__reconstruct__

Method which takes a structure tree and rebuilds the object. This is different from __setstate__ in that __setstate__ assumes its children have already been reconstructed. This is the difference between top-down and bottom-up interpretation. Bottom-up is usually easiear and sufficient, but top-down is required for radical restructuring of the object representation.

We need to coexist with third party libraries which may or may not use our object serialization technology, or even if they do, we may want to replace the dependence of one third party library with our own or another implementation. In order to do so we allow have class registry with the function refactor to add entries to the registry. Rather than blindly restoring the class we first see if the class and version are in the registry, and call the registered function to transform the data. For example:

import danse.util.serial as serial
serial.refactor('Numeric.array','numeric_converter','0.0')

This says that items stored as the class Numeric.array will be loaded by calling numeric_converter, which returns a

def data():
    from mylib.core.data import Data
    return Data()

from danse.util.serial import isnewer, reconstruct, setstate
class Data(object):
    __version__ = '1.2'
    __factory__ = 'mylib.data'
    def __reconstruct__(self,instance):
        '''
        Reconstruct the state from
        '''
        if isnewer('1.0',instance['version']):
            raise RuntimeError('pre-1.0 data objects no longer supported')
        if isnewer('1.1',instance['version']):
            # Version 1.1 added uncertainty; default it to zero
            instance['state']['uncertainty'] = 0
        setstate(self,reconstruct(instance['state']))

8.2 < 8.19a1 < 8.19 == 8.19.0

if isnewer(version, Model.__version__):
    raise IOError('software is older than model')
elif isnewer(xml.version, '2.0'):
    instantiate current model from xml
elif isnewer(xml.version, '1.0'):
    instantiate old model from xml
    copy old model format to new model format
else:
    raise IOError('pre-1.0 models not supported')