Overview

Overview #

Introduction #

This facility, usually shortened to ‘ADH’, allows users to convert an electronic document in one format into another - the second format usually being one that SICS itself can understand, typically an eMessaging message (though other formats are supported). It is a powerful addition to SICS, as it widens the scope of the eMessaging interface. Users receiving non-ACORD messages and notifications can transform the original document into the ACORD format, and process the resulting message through the extensive eMessaging solution.

Figure 1 ADH Process Overview

The graphic above outlines the ADH process. The left hand side shows the tasks performed for each document to be transformed, the right hand side shows the set-up and administrative tasks required before a document is transformed. The basic assumption here is that you will receive documents of a known regular pattern, and the pattern is repeated in all documents of that type. You set up a plan for transforming documents of that type/pattern, then reuse the transformation again and again. To understand how ADH works, we need to start with the system set-up tasks.

Mappings #

At the heart of the ADH process is a mapping. The mapping describes both the incoming and outgoing documents and the transformation required to convert the former to the latter. The incoming and outgoing documents are described by patterns, but the method and process for creating output patterns is different from input ones. Between the input pattern and the output pattern are any validation rules and the transformation rules. The input pattern, validation and transformation rules are all defined within a Transformation mapping, but before we can create one, we must define the output pattern.

Output patterns #

By defining an independent output pattern, several different input documents can be set to create the same output pattern. SICS allows three different types of output pattern - eMessage, external or tabular. For simplicity in this overview, we will focus on eMessaging, and to be of any use, an eMessaging output pattern must create a document that is already known to SICS (e.g. an ACORD RCL XML message).

Figure 2 Output pattern

On the output pattern window, the left hand side defines a series of field groups and field names; these are the field names used in the transformation mapping. On the right hand side is an XSL Transformation stylesheet, that converts the output from the transformation mapping into an actual XML eMessage.

Fields within the output pattern can be defined as mandatory. When using the output pattern in a transformation mapping, a mandatory field must be supplied with data, otherwise the transformation mapping cannot be saved with status ‘Active’. Fields in the output pattern can be grouped, and the group itself can be mandatory and repeatable (individual fields are not repeatable). A repeatable group will be populated once for each corresponding incidence in the incoming document. Fields within a mandatory group need not be mandatory themselves, but at least one must receive data from the transformation mapping to ensure the group is created.

Transformation mapping #

Once we have defined an output pattern, we can use it in one or more transformation mappings. Each transformation mapping consists of three main parts:

• Input pattern
• Validation rules
• Transformation rules

An Input pattern is created as part of a transformation mapping. Again fields can be defined as part of a group, and groups can be repeatable. Fields can be mandatory or forbidden. Each field is given a name and an xPath. The xPath describes how to extract data from the incoming document for use in the validation and transformation rules.

The input pattern can support tabular data in the XML version. However, the format for this is very strictly defined -see the appendix for further details.

Figure 3 Input pattern

The input pattern is shown on the left hand side of the Edit mapping window. On the validation tab you can then add validation rules to the mapping. If, when processing a specific document, any of the rules are not true a validation error will be set preventing the document from being transformed. Validations are created using drag-and-drop and function blocks from the mapping palette.

The transformation tab also shows the input pattern, but significantly, also allows you to specify the output pattern. Using drag-and-drop, fields from the input pattern can be copied to the output pattern, and can be modified using transformation functions. These functions include options for managing flow, for setting constants (effectively hard-coded output), and many others. We will discuss these functions in greater detail later on.

Figure 4 Transformation mapping

Transforming documents #

Now we have defined a transformation mapping, we can use this to transform an incoming document into a useful output message that can be processed in SICS. We are moving from system set-up to day-to-day use.

On the SICS desktop there is a menu item for Automated Document Handling, and there are two options:

• Import
• Find

This follows a similar pattern to eMessaging. The import window allows you to specify a path to a folder containing the document (or documents) you want to transform. Selecting import will copy the documents into the SICS database, and will move the originals into the archive folder.

The find window shows the documents already loaded into the database. There are several find options, including identifier, import time and status. One or more documents can be selected for further action, and double clicking on a single row will open the document window.

Figure 5 Document view - content

Document life cycle and statuses #

After a document is loaded, it is given the status of ‘ready’. It will automatically try to find a suitable transformation mapping, but this can be assigned manually if necessary. If an attempt to transform is made without a mapping being assigned, the status is changed to ’no mapping’.

The transformation mapping will first apply the validation rules, and if these are all successful will then apply the transformation rules. If there’s a validation failure the status becomes ‘Invalid mapping’, or if there’s a transformation error the status becomes ‘Transformation error’. For a successful transformation the status is set to ‘Transformed’, and nothing more can be done with document. Any messages created by the transformation will be loaded directly into the eMessaging archive for further processing. When sub documents are enabled (see mapping detail in transformation mappings), a status of ‘Partially transformed’ indicates that one or more sub-documents in the incoming document have successfully created an output message, but that one or more pages have failed to do so, and remain to be transformed at a later time.

Documents can be transformed automatically, if the ‘process on load’ option is selected in ADH system parameter processing options, but the user can also manage the process manually using menu options.

Menu options #

The main option is to transform the document. This is only enabled if the document is linked to a transformation mapping, and this is another option in the menu. You can also opt to discard the document if you no longer need to transform it, or set a partially transformed document to ’transformed’ if you no longer want to transform those sub-documents that return errors during the transformation.

You can also choose to view or edit the transformation mapping - this is useful if trying to debug errors in the transformation, as the graphical representation of the mapping will show the values in the input pattern. Further options are provided to change the selected transformation mapping or to create a new one.

Document window #

The document window consists of a number of tabs. The content tab shows the input document in raw form - native XML in this case. The next tab will either be output or problems, depending on whether the transformation was successful. It is omitted altogether when the document is first loaded and no transformation has yet been attempted. Then comes history, which shows what actions have been taken, starting with loading the message, and finishing with the last action taken. These are shown with the latest action on top. Notes allows you to add your own annotations to the document whilst Rendered view allows you to see the incoming document in a formatted way.

When there are problems, these are shown on the problems tab. For a successful transformation, the output tab will show the transformed messages, which will already be loaded into the eMessaging archive.

Corrections #

An additional facility is the ability to correct problems encountered when transforming a document. Many digital documents presented for transformation are created from optical sources, and the optical reading is not always accurate enough to provide sufficient quality in the digital version. For example, commas (",") and periods (".") can get confused, which in text is acceptable, but is not in numbers. Dates sometimes have unexpected formats; text can get distorted.

The correction process allows the user to overcome many of these problems - either by entering a default value to be used instead of a missing value, or by entering a replacement input value to the input pattern. Some corrections, such as removing multiple decimal points from a number, can be applied automatically. Where corrections have been defined or applied, these are shown on the document window.

Sub documents and Leaf Repeating Groups #

As we have seen, the input pattern can include repeating groups.

Figure 6 Sub Documents and Repeating groups

In this example both ‘page’ and ‘instalment’ are defined as repeating groups. Either one of these groups can be chosen as the sub-document group, so that each new occurrence of the group within the incoming document forms a separate sub-document. The user can choose in the mapping process if all sub-documents be transformed at once, or if sub-documents that can be successfully transformed will create an output message, leaving those that cannot in error.

For each sub-document (or the whole document if no sub-document group has been set on the mapping), there is a notion of Leaf Repeating GroupandLeaf Repeating Group occurrences. The Leaf Repeating Group refers to the lowest level(s) of repeating group(s) in the Input Pattern. The Leaf Repeating Group occurrences are the actual occurrences of the Leaf Repeating Group(s) in the document. Transformation of a (sub) document takes place by processing each Leaf Repeating Group occurrence in turn, pushing the Input Pattern values for thisLeaf Repeating Group occurrence along the connections, through the blocks, and onto the Output Pattern fields. The term Leaf Repeating Group occurrence is also know as Input Position.

The relationship between Leaf Repeating Groups on the input side and output patterns will determine the number of messages produced from a document. If at least one field inside a repeating input pattern group is connected (possibly via blocks) to an output pattern field which is a direct child of the <root> output pattern field, then each occurrence of the repeating input pattern group will create one output pattern <root> occurrence. And each output pattern <root> occurrence in turn makes one output message. One exception to this rule would be the use of the @RetainData and @BlockOutput on the Output Pattern, as explained elsewhere in this documentation.

In the example above, if the incoming document has two pages, and the first page has two instalments whilst the second has just one, there will be three output messages created. Each instance of dueDate or premium from the instalment group will create another output message. Note this is separate from the sub-document - even if sub-document is set as page, there will still be three output messages, two for the first page / sub-document, and one for the second.