Transformation Mappings

Transformation mappings #

The [Overview]({{"/sicsdocs/technical/desktop/adh/overview/#" | relative_url}}) section described the basics about mappings. Here, further details are explained.

Figure 12 Transformation mapping - selection

From this table you can view, edit, copy, create or delete a transformation mapping. Other options include activating or inactivating a mapping, unlinking the mapping from existing documents, and importing/exporting the mapping. Only valid mappings can be activated.

Import and export are important for copying a mapping from one environment (e.g. test) to another (e.g. production). Multiple mappings can be imported/exported at the same time. The exported mappings are in XML format. The exported mappings are saved to a folder, one file for each mapping, each file named the same as the mapping.

With some limitations, it is possible to export mappings in an XML format that is compatible with a prior version of SICS. To do so, select the ‘Export To…’ context menu option.

When importing a mapping and linking to a referenced Output pattern, the mapping will be updated to fit the Output pattern. For example, assume the referenced Output pattern was recently updated by removing an Output pattern field. If the imported mapping has connections to this field, these connections will be removed.

Note: only XML mappings can be created in system parameters. Tabular mappings must be created directly from an imported tabular document.

The transformation mapping is at the heart of ADH. This is the part that links the input pattern to the output pattern, via a series of links and functional blocks. These links and blocks are graphical representations of the mapping process. A link directly between the input and output will copy the incoming value without modification to the output. Adding a block provides scope for transforming the incoming value in various ways. Blocks can be strung together to create complex mapping processes. A full description of the blocks is given here: [Transformation Blocks]({{"/sicsdocs/technical/desktop/adh/transformation_blocks/#" | relative_url}}).

Input Pattern #

Whilst the output pattern must be predefined before the mapping is created, the input pattern is created as part of the mapping. For a new transformation mapping, an Input pattern with a default root segment is automatically provided. Alternatively, if creating a mapping directly from a document, the system will automatically create input pattern elements from the source document, which can save a lot of effort when creating the input pattern. The input pattern is always shown on the top left of the validation and transformation tab windows, and no blocks or links may be placed below it.

Even when the pattern is created automatically, it may not contain all the required elements. The Palette section below describes how additional fields and groups can be added to the input pattern.

The input pattern consists of field groups and single fields.

Field Group #

A group is used to gather similar elements together, and the group may be set to be repeatable. It’s particularly used where the same data appears again and again in the document - if the document has multiple pages, the page is defined as a group.

Figure 22 Palette - new field group

The group is indicated by the rectangle around the elements in the group, with the children of the group indented to the right. The topmost element is the group header. It is possible to create further groups within a group.

Clicking on a field group header, you can select ‘properties’ from the menu. Double clicking the item has the same result.

Figure 23 Palette - field group properties

A group may be both mandatory and repeatable.

Single Field #

Clicking on a (single) field item, you can select ‘properties’ from the menu. Double clicking the item has the same result.

Figure 24 Palette - field item properties

Input Pattern single fields support xPath field (back)references. An xPath field (back)reference is a SICS concept, not an xPath concept, so is described here.

When SICS is about to process an xPath expression, SICS first looks for a substring surrounded by ‘$’. Such a substring is processed as a reference to the value of another Input Pattern field, and that value replaces the substring and the surrounding ‘$’ characters. This process repeats until all field references have been processed and replaced.

Note: The reference must refer to an Input Pattern field that is ‘above’ the current field (and thus already calculated), typically in a parent group. Hence this kind of reference is also called a backreference.

Note: Nested field (back)references is not supported.

Note: The use of ‘$’ for any other purpose than demarcating a sub-expression containing a backreference is not supported. For example, testing ‘value="$ Amount"’ is not possible.

Example: In a usecase where an Excel spreadsheet has been imported into ADH and a mapping has been created from the input data, there will be an Input Pattern structure with groups ‘Output/pages/page/table/row/cell’ (for the rows). Here, ‘page’, ‘row’ and ‘cell’ are repeatable groups. Typically, there are column headings in some row number N, and data values to be processed in subsequent rows. Say row N, column number M contains the column heading ‘U/W YEAR’. N and M could vary from one spreadsheet to another, thus they cannot be hardcoded but has to be calculated using suitable xPath expressions. Rather than processing cells one by one, we would want to process rows one by one. To this end, we would delete the ‘cell’ group and manually create a new Input Pattern field ‘_uwYear’ in the ‘row’ group, with xPath = cell[@columnNumber=number('%../row/cell[contains(value,"U/W YEAR")]/@columnNumber%')]/value. The issue here is that the lookup of column M (%../row/cell[contains(value,"U/W YEAR")]/@columnNumber%) would take place for every row, and thus slow down processing. Using a field reference, we can improve on this: In the ’table’ group (i.e. the parent group of ‘row’), create an Input Pattern field §uwYear with xPath = row/cell[contains(value,"U/W YEAR")]/@columnNumber. The first character of the field name, §, indicates that the field will later be used in a backreference and hence its xPath expression must be calculated, even if the field has no outgoing connections. Then, change the xPath of ‘_uwYear’ to: cell[@columnNumber=number('$../§uwYear$')]/value. This functionality applies to the Transformation tab only. On the Validation tab, as a performance improvement, fields with no outgoing connections will never be calculated, not even if they are fields whose name start with §.

Input Pattern single fields also support xPath sub-expressions. An xPath sub-expression is a SICS concept, not an xPath concept, so is described here.

When SICS is about to process an xPath expression, after resolving field references, SICS looks for a substring surrounded by ‘%’. Such a substring is processed as an xPath expression in its own right, in the context of the field’s xPath node, and the resulting value replaces the substring and the surrounding ‘%’ characters. This process repeats until all sub-expressions have been processed and replaced.

Note: Nested sub-expressions is not supported.

Note: The use of ‘%’ for any other purpose than demarcating a sub-expression is not supported. For example, testing ‘value="% SHARE"’ is not possible.

Example: Looking at the column group within a table, we have this Input Pattern fragment:

Figure 25 xPath sub-expression

Here, the ColumnName field would have this xPath expression:

../cell[@columnNumber = '%@number%' and @rowNumber = '1']/value

where %@number% is a sub-expression.

A field may not be both mandatory and forbidden.

Editing a mapping #

Figure 13 Transformation mapping - edit

The mapping can be opened in view mode or in edit mode. In edit mode, the name can be changed, as can the status and description. Note: changing the name does not create a new mapping - it simply changes the name of the existing mapping. Imported mappings are automatically set to inactive status. After import, the mapping must be activated from the mapping list, or be edited to change the status to active, before it can be used.

Note: Revision support is described in a separate chapter.

The window heading shows these items, left to right:

There are five tabs on this window:

and one menu button. The menu button can be used to open the Mapping Detail window (see below), or change Test Values Options (see below).

Validation #

The validation tab contains the Input pattern and various blocks from the palette (see below). Blocks are added and links made between the input pattern and the blocks to test the validity of the document being processed. For example, if the incoming document includes a fixed template name, an ‘In’ block can be added to test that the document template name is in a list of allowed template names for the mapping (as defined in the ‘In’ block). If the name is in the list, the output from the block is true, but if it is not, the output is false. An error is set if the output from the block is Boolean, is false, and is not connected to another block. Please note: although an unconnected false Boolean will create an error, the use of error blocks provides a better response to the user.

Transformation #

Figure 14 Transformation tab

The transformation tab contains the input pattern defined in validation, and the output pattern. It also contains the graphical links and blocks between the two. The structure of the input pattern can be created and modified using options from the palette. The output pattern (which is defined elsewhere) can be selected from the palette, as can the links and blocks.

Several output patterns may be added to the mapping. Also, a repeatable group within an output pattern may be copied (using the context menu), this allows for setting up different incoming connections to the fields in the group - for example creating multiple IndividualClaimAmtItems in the Tech Account message. The output pattern itself only has one instance, but in the mapping, several instances can be created and populated. The original group and any copies created are still treated as one repeatable group when the output pattern data goes through the final transformation to the output message (using XSL for an XML mapping).

When blocks are added to the validation or transformation, they are un-numbered, but when saved, they become numbered. A new mapping transformation will start with block number 1 (validation blocks also begin at 1, and use their own numbering, independent of the transformation numbering), but blocks added to an existing mapping continue with the sequence from the highest numbered block. If blocks are deleted before the highest numbered block, the number is not reused. For example, if the highest numbered block is 56, and block 42 is deleted, the next block to be added will be numbered 57. There will be no block 42 in the mapping. The block number is used in error correction processing.

Test Values #

On the validation and transformation tabs, hovering the select cursor over an item (a field in the input pattern, a link, a block or a field in the output pattern) will show the current value as a ’tooltip’, but only if there is a document linked to the mapping. If the mapping has been opened directly from the message (see View document below), the current document values will be shown. If the mapping is opened from the system parameter menu, there is no current document, and this tab allows you to choose a document previously loaded in the ADH archive to test with.

Further, you can modify the contents (note: the original document is not changed, only the copy being used for testing is altered) so that different paths through the validation and transformations can be tested. ‘Update’ must be pressed after changing the content before the mapping recognizes the changes made.

If, when hovering the select cursor over an item no value is shown, it implies there is no data - typically this means a block has failed to transform, and, if the mapping were run against a document an error would result. Finding empty values is a helpful technique when debugging a mapping.

Gate (Master) blocks will show red background color when closed, that is, when the currently selected Repeating Group occurrence will be excluded from transformation processing.

Any block which fails (raises a problem) - after applying corrections if selected - will show red background color.

Also, if the mapping is linked to a document, the Input Pattern fields will show input data (test values) according to the selected sub document / input position. Likewise, the Output Pattern fields will show the resulting output values.

Test Values Options #

The mapping menu button shows a number of options that affect how test values are calculated and displayed.

Figure 15 Test Value Options

If the mapping defines sub documents (see elsewhere), then the first drop-down after the menu button will show the sub documents that are present in the current test values. If sub documents are not defined, then this drop-down will be disabled.

The next drop-down shows the Leaf Repeating Group occurrences within the selected (sub) document of the test values.The Leaf Repeating Group occurrences will be listed in sorted order as specified by Sort blocks, if such are present. Otherwise they will be listed in the order given by the test values.

By selecting items from the drop-downs, and/or enabling/disabling menu options, the user can control the tooltip information based on the test values.

Automatic Corrections #

This tab lists any automatic corrections that apply when this mapping is used. Although a context menu appears if you select a correction and right-click, the options on the menu are always disabled. To add a correction to a mapping, it must be done from the Automated Document Handling menu in system parameters.

Notes #

Here you can add comments about the mapping. This is particularly useful for documenting complex parts of the mapping process.

Mapping detail #

Figure 16 Transformation mapping detail

Saving transformation mappings #

When trying to save a mapping, the system validates that the mapping is complete. Possible errors include:

• When a block has no input.
• When a block is missing a parameter.
• If the connections form a loop.
• When a mandatory output field is unpopulated.

Figure 17 Transformation mapping - errors on save

A list of problems will be shown when an attempt to save is made, and selecting an error from the list will highlight the block or field that is reporting the problem. (Note: this only works if the selected error occurs on the tab being viewed. If the selected error is on the transformation tab, and you are viewing the validation tab, you cannot see the highlighted problem on the other tab).

The problems window includes ‘Save’ and ‘Edit’ buttons. The ‘Edit’ option returns you to edit mode, where further changes can be made. The ‘Save’ option will persist the incomplete mapping, and will set the valid status to ‘No’, and the Status to ‘Inactive’. The mapping cannot be made active whilst ever there are problems with the mapping.

When reopening an invalid mapping for edit, the problems window is re-shown (without save or edit options). Close the problems window to edit the mapping.

When saving a new mapping (or a copy of an existing one), the status will default to ‘inactive’.

Palette #

The palette provides the tools to create and maintain the transformation mapping. Within this window are the means to add fields to the input pattern, add blocks to the mapping, select one or more output patterns, and add connections (links) between these.

Figure 18 Palette

Select #

This is the standard cursor mode, and is used to add fields to the input pattern, to add blocks to the mapping and to select one or more output patterns. It is also used to select blocks already placed in the mapping, and by double clicking (or right click and selecting from the menu), open the block properties. Hovering over an item with the select cursor will show the current value (only if there is a document attached to the mapping - see Test Values above).When adding a new block or output pattern to the mapping, and also when moving an existing block or output pattern, the object cannot be positioned below [and to the left of the right edge of] the input pattern.

Marquee #

This cursor mode allows you to select an area of the mapping window. Only items wholly within the select area are included. Using this tool, you can delete multiple blocks, or perhaps more usefully, move them around the window. This can important if you need to create some empty space so that new blocks are not overlaying existing ones.

Connection #

The connection cursor is used to make links between input pattern fields, blocks, and output pattern fields. When selected, the cursor symbol changes to one representing an electrical plug. If the cursor is in an area where no connection can be made ‘-’ is shown by the cursor. Connections must be made from a logical left-to right, from the input side towards the output side. In practice, this means linking from the ‘out’ item in a block to the ‘in’ item in the next block, and physically the first block may be placed anywhere in the middle of the window, including to the right of the next block. The connection remains logically from left-to-right, even though on the graphic, it starts further right than it finishes!

Figure 19 Logical left-to-right

When the connection is completed, the right hand end of the link is changed to an arrowhead.
If a connection, block, input field or output field is selected, the context menu offers a ‘Select Data Path’ option. This highlights the path from input to output.

Figure 20 Transformation mapping - select data path

The context menu also offers a ‘Show Data Path Only’ option. This will hide all connections and blocks that are not on the data path(s) of the selected mapping object(s).

Figure 21 Transformation mapping - show data path only

Finally, the context menu option ‘Show All’ will make all objects visible again.

Input Pattern Fields #

This group contains tools to add a field group or single field to the input pattern.

Field Group #

After selecting this option from the palette, and placing the cursor over the input pattern, a long bold horizontal line will appear in the pattern at the point the group will be inserted.

Field #

When selecting this option from the palette, and placing the cursor over the input pattern, a long bold horizontal line will appear in the pattern at the point the field will be inserted.

Transformation Blocks #

Please see [Transformation Blocks]({{"/sicsdocs/technical/desktop/adh/transformation_blocks/#" | relative_url}})

Output Patterns #

This is the list of active output patterns. Output patterns can only be placed on the transformation tab. It is possible to create multiple outputs from one input document in a single transform - for example, if creating TechAccount and Settlement messages from a single source document (as happens elsewhere for Lloyds accounting messages).

Output patterns have no properties. However, the context menu for the Output pattern allows the copying of repeatable output pattern groups. This is useful say, for TechAccountAmtItems in an ACORD TechAccount message. The output pattern may only include one instance of the TechAccountAmtItem group, but the mapping may need to populate several items in the message. Copying the TechAccountAmtItem group allows many items to be created in the outgoing message.

Automatic Corrections #

Figure 29 Automatic corrections

For common errors, rules can be provided within SICS to automatically correct the problem. For now, there are two types of automatic correction:

1. Keep only last decimal separator
2. Ignore ‘No output data’

Other rules can be added on request.

Automatic corrections are only applied automatically if they are assigned to a transformation mapping, and this is done from the Automatic Corrections option in system parameters. Selecting a row in the table and right-clicking opens context menu, where you can choose to:

• View
• Edit
• Create
• Delete

a correction type.

On selecting Create, Edit or View, a new window opens.

Figure 30 Automatic corrections - edit

In create mode, the window is empty, in view mode the Available column is not shown, but there is an edit button to change to edit mode.

Automatic corrections can also be applied manually during document transformation if they are not assigned to the mapping.

Automatic Correction types #

Keep only last decimal separator #

This rule will remove all but the last decimal separator from an input field, when this field is input to the following block types:

Each of these blocks includes a parameter that defines the decimal separator.

Example:

If the input document has been misread, and an amount appears in the input pattern as 123.456.78, the blocks above will normally fail, as the input is not considered to be numeric. It is not numeric because there are multiple decimal separators. By applying the auto correct rule, any input to the above blocks that contains more than one decimal separator will have all but the last one removed.

Note:

If this correction is not assigned to the transformation mapping, and one of the above blocks detects a nun-numeric input, the user has the opportunity to apply the correction manually.

Ignore ‘No output data’ #

This rule will ignore / suppress the ‘No output data’ error which would otherwise be raised when an input sub document does not produce any output.