Oakland Data Transformer - Runtime

This section discusses the runtime execution of a map, and various behind the scenes internal issues.

Runtime

The runtime support is used to execute maps in a production environment. We provide both a Java API which can be used to execute a map.

A map can be executed for only one file (or whatever files it depends on) or for all of the files in a given directory.

The runtime environment depends only on having access to the map and structure definition files, which can be conveniently moved anywhere for deployment.

ESB Integration

The runtime supports integration with Apache Camel, OSGi Blueprint, Apache Karaf, and Mule ESB. Integration for each of these is provided using the standard means of plugging in components or transformers.

Map Execution

When a map is executed, the map metadata is translated into X/Query and compiled by the X/Query engine. Different X/Query engines can be plugged in as required. We currently use the Saxon X/Query engine, probably the most widely used open source engine and it has the best conformance to the latest editions of the standard.

All input for the execution of the map (flat files, database rows, EDI, etc) is translated into SAX events for processing by the X/Query engine. The actual map execution handles all types of data uniformly. Similarly, the output is translated from SAX events to the desired output format.

Because of this architecture it is easy to add additional adaptors for different formats of data.

Metadata

All metadata is stored in XML, some in local files for user projects and others as part of plugin projects (for the standards). The metadata format is fully versioned and extensible so the definition investment in maps and structures can be preserved permanently across all versions of the product.

As the amount of metadata can be quite large (consider the EDI X12 856 transaction with all of its code tables), there are many tricks and optimizations required to not store duplicate data.

These are accomplished both in the XML serialization process by detecting the removing redundant blocks and by the use of the inheritance mechanism to reuse definitions.

Compatibility Between Different Versions

A map or structure created with any prior version of the Oakland Data Transformer (or ODT-based products like the Mule Data Integrator) will always work correctly when used with the current version of the product. All of the metadata is versioned so that when an old version is opened, it is automatically brought current. We maintain many tests that execute maps created in the earliest versions of the product to ensure that we are maintaining compatibility.

In general the designer and runtime must correspond in order to successfully execute maps. For example if you have a newer designer than the runtime, it may not be possible for the runtime to execute the map because it does not understand the formats from the newer designer. This problem will be detected by the runtime and a clear error message will be given.

You can use the product version number to determine compatibility between the designer and runtime. Any products with the same major/minor version number will always be compatible (starting with version 3.1.0). So for example a 3.1.0 runtime will execute a map produced with a 3.1.7 designer, but will not necessarily be able to execute a map produced with a 3.2.1 designer.

Code Generation

Currently X/Query is generated from the map metadata. However the code generation portion of the product is isolated and generating alternative code is a relatively small task. XSLT could be generated to support high-performance appliances that do transformation. Native Java (or code in another language could be generated).

Diagnostics and Support

Extensive logging (using Log4J) is provided and can be enabled in the field. An automatic bug-reporting mechanism is provided in the designer to allow the user to submit a bug report if the product catches any sort of unexpected exception. The user can also directly request support with a comment or question through the designer.