Automated Document Handling in the SICS Workstation Desktop

Preceding documents relate to the administrative set-up required to create and manage the mapping tools needed to transform an incoming document into a format that can be understood by SICS. For the general user working with incoming documents, there are three main functions.

Import Document #

This window allows the user to specify an inbox path and an archive path, and to start the import process.

Figure 31 Import doucment

Inbox path: The location of the documents to be imported. This is simply a folder somewhere on the network.

Archive path: The location where documents should be placed after import. This can be the same path as in the inbox path, which is useful when testing. However, in production this can lead to the same document being imported repeatedly. It is recommended that a separate path be used. If the archive path is empty, the imported documents will not be moved.

** Import as Document Type:** Radio button group to select the document type to import, you can choose XML or Tabular.

If the XML radio button is selected, the following file types will be imported as XML Document Type: .xml, .txt, .xls and .xlsx. Note that excel spreadsheets (.xls and .xlsx) will be stored as XML in SICS.

If the Tabular radio button is selected, the following file types will be imported as Tabular Document Type: .xls, .xlsx and .csv.

Inbox files: This table shows a list of documents in the inbox path. Using control and shift keys, it is possible to select individual items, or a range of items

The following buttons control the operation of the window:

Refresh: Refreshes the inbox files table with the latest list of documents in the inbox path

Import selected: Imports the selected documents into the ADH area within the SICS database.

Import all: Imports all the documents in the inbox path into the ADH area within the SICS database.

Close: Closes the window without importing any document. Normally the window closes automatically once the import is complete.

If the ADH parameter ‘Transform on load’ is enabled, SICS will automatically try to find the appropriate transformation, and will apply this mapping to the document.

If the ADH parameter ‘Rename document file on import’ is enabled, the existing filename of the document will be appended with a date and timestamp. This avoids problems with duplicate filenames in the archive folder.

Various aspects of ADH will fail if namespace information is present in the internal ADH document that is created from the external document during import/loading. In order to avoid this, all namespace information will be removed when an internal ADH document is created from an XML file. This applies to online loading (import) from the SICS workstation, as well as loading files through SICS ADH Server.

Import of XML documents #

Import of an XML file requires that the file is UTF-8 encoded. Even so, the client may sometimes provide input files that are incorrectly encoded (contain not-UTF-8 byte sequences). Online loading (import) from the SICS workstation, as well as loading files through SICS ADH Server, will handle such situations according to the ADH Processing Option ‘Handling of invalid UTF-8 multi-byte sequences in XML documents’ (please see [Define Processing Options]({{"/sicsdocs/technical/desktop/adh/processing_options/#handling-of-invalid-utf-8-multi-byte-sequences-in-xml-documents" | relative_url}})).

Find Document #

The find document window is the place to start looking for a document previously loaded into the SICS database. There are two tabs, and three buttons.

Figure 32 Find document

The buttons behave in the usual way

• Find now returns the first 50 documents that meet the search criteria.
• More returns a further 50 documents. More is only enabled if there are further documents to add to the list
• New Search clears all the selection criteria.

The Properties tab is the main selection tab

Main search criteria

Identifier: If you know the identifier of the document to be found, this option will return only those documents matching the given identifier. Partial identifiers using wildcards are permitted. However, given the nature of many identifiers, (some are long incomprehensible strings of hexadecimal characters), the use of other criteria may be preferable.

Type: A drop down selection: The default is none, but you can choose Tabular or XML

Import Timestamp: From and to dates for the import date can be selected. If from date only is entered, all documents from that date up to present will be returned. If to date only is selected, all documents from the earliest date up to the entered date will be returned.
The Status tab allows you to select which document statuses are found. The default is to show all statues - if there are no statues in the selected column, all those in available are searched. Moving a status from Available to Selected will limit the search to those statuses in the Selected column.
Find table shows the list of documents returned by the search

Import Timestamp: The date and time the document was loaded into the SICS database.

Identifier: The identifier given to the document when imported. This usually taken from an element within the document.

Type: Either Tabular or XML, depending on the type of document when imported.

Status: The status of the document.

Documents in the SICS database are assigned one of the following statuses:

• Discarded - assigned when the message is discarded by the user. A message may subsequently be restored, when the status becomes ‘Ready’.
• Invalid mapping - the mapping detects an error in the validation phase. More detail is available by viewing the document.
• Multiple mappings - SICS is unable to automatically select a transformation mapping, because several are available.
• No mapping - No transformation mapping has been selected (automatically or otherwise) for this document
• Partially transformed - Only used if the transformation mapping handles sub documents. One or more sub documents have successfully been transformed, whilst one or more contains errors that prevent their transformation.
• Ready - the message is ready to be transformed.
• Transformation Error - the document passes the validation, but the transformation fails for some reason. More detail is available by viewing the document.
• Transformed - the transformation is complete, and nothing further can be done with this document.

Double clicking on a row in the find table will open the document properties (View document). You can also select single row or multiple rows. Right click with then open a pop-up menu.

Find document pop up menu options

View properties - opens the View document window for the first selected document. Transform - Begins the transformation process for all the selected documents. Not available for transformed or discarded status messages.

Set to Transformed - for documents linked to a transformation mapping that supports sub-documents, this option is enabled for partially transformed documents. One or more pages of the incoming document will have created output messages, but one or more pages will remain in error, unmapped. Use this option if the unmapped documents are not to be transformed. The document status will then become transformed.

Unlink mapping - remove the previously linked transformation mapping. Not available for transformed or discarded status messages.

Discard - Discards the selected documents. A comment must be added to complete the discard process - the same comment applies to all selected documents. Not available for transformed status messages. Restore - Restores the selected documents. A comment must be added to complete the restoration process - the same comment applies to all selected documents. Only available for discarded status messages.

View Document #

This is main window for viewing and progressing the transformation of a document.

Figure 33 View document history

The top of the window shows several identification attributes, and the document menu.

Identifier: The identifier given to the document when imported. This usually taken from an element within the document.

Original File Name: The filename from the which the document was imported. Note: if the ADH parameter ‘Rename document file on import’ is enabled, the current filename will be the original name with a timestamp added.

Mapping: the name of the transformation mapping assigned to this document.

Import timestamp: The date and time the document was loaded into the SICS database.

Current status: The status of the document. See ‘Find Document’ for a list of all possible statuses and their meanings.

Menu items #

Transform - begins the transformation process, including automatically locating the transformation mapping. An error will result if no transformation mapping is assigned. Not available for transformed or discarded status messages.

Set to Transformed - for documents linked to a transformation mapping that supports sub-documents, this option is enabled for partially transformed documents. One or more pages of the incoming document will have created output messages, but one or more pages will remain in error, unmapped. Use this option if the unmapped documents are not to be transformed. The document status will then become transformed.

Discard - Discards the document. A comment must be added to complete the discard process. Not available for transformed status messages.

Restore - Restores the document. A comment must be added to complete the restoration process. Only available for discarded status messages.

View Mapping-Opens the transformation mapping window in view mode. Not available if no transformation mapping is assigned to the document.

Edit Mapping - Opens the transformation mapping window in edit mode. Not available if no transformation mapping is assigned to the document.

Link Mapping - Opens a selection window to manually assign the transformation mapping to the document. Not available if a transformation mapping is already assigned to the document.

Figure 34 View document select mapping

Unlink Mapping - Removes the current transformation mapping from the document. Not available if no transformation mapping is assigned to the document.

Change Mapping - Opens the selection window to manually assign the transformation mapping to the document. Not available if no transformation mapping is assigned to the document.

Create Mapping - Creates a new transformation mapping based on the content of the document. Only the input pattern is created. The user must then select an output pattern and create all the links between input elements and output elements. See also Transformation Mappings above, which describes the process of creating a new mapping in full.
Note: this is the only method of creating a tabular mapping.

Content #

The content tab shows the raw document with no formatting or transformations applied.

Spreadsheet #

The spreadsheet tab is visible if the document was imported as XML Document Type and the original document is an excel spreadsheet (.xls and .xlsx). It shows the content of the spreadsheet in a tabular manner. You can navigate between sheets using the tabs located in the bottom left corner.

Problems #

This tab shows a list of any problems encountered when validating or transforming a document.

Figure 35 View document, problems tab

Type: The general type of problem - either Mapping, Validation or Transformation

SubType: For Validation and Transformation errors, a sub-classification of the error. The type of correction available depends on the sub- type.

SubDoc: Only used if the transformation mapping supports subdocuments. The subdoc number is the serial count of the sub document group (frequently defined as ‘page’) within the document. Please note this is not the same as the page number. Subdoc numbers are always a continuous series starting at 1. Page numbers need not start at 1 and do not need to be consecutive.

Input Fields: A list of input fields that may have caused the problem.

Description: An outline of the problem detected. Where the problem involves a mapping block, the specific block in the mapping is identified with the block name and number.
Selecting a single row and right-clicking will open a pop-up context menu. The options on the menu will depend on the SubType of the error.

Problem subtypes and possible correction options #

Block Transformation #

Block transformation errors mainly occur when the input to the block is not recognized. Typically this will be if an arithmetic block is receiving text, or a block expecting a date/time receives an input in the wrong format.

Correction options:

• Block Input Replacement Values
• Input Fields Replacement Values
• Keep only last Decimal Separator

Mandatory Output Field Group Missing Value #

A group of fields in the output pattern has no value, but the group is mandatory.

Correction options:

• Default values for group
• Input Fields Replacement Values

Mandatory Output Field Missing Value #

A mandatory output field has no value provided by the mapping.

Correction options:

• Default value
• Input Fields Replacement Values

Translation Table Missing Value #

A translate block, where ‘Fail on missing’ is true, has failed to find a matching value in the translation table. Note: if ‘Fail on missing’ is false, no error is set.

Correction options

• View/edit Translation table
• Input Fields Replacement Values

Error in sending generated document #

Applicable for external output pattern only. When document has been generated, but it could not be sent on its destination successfully. This will be considered as transformation error, problem sub type will be ‘Error in sending generated document’.

Description field will show detailed information about the error.

Possible reasons

• Folder path is not accessible
• REST or SOAP URL is not correct
• REST or SOAP server is not running or not accessible
• Parameter substitution in URL using xPath syntax is failed
• Generated document is not a valid XML, JSON, SOAP document, so either not parsed or rejected by server
• Server reponse was sent with status other than 200

Overview of correction options #

On selecting a correction option, a create correction window will open. The exact format of this window will vary according to the correction type selected.

Block Input Replacement Values #

Figure 36 View document, create correction

Document: The unique identifier of the document being corrected. Corrections made on this window will only apply to this document. They will not apply to other documents using the same transformation mapping.

User: The name of the user making the correction. This retained for audit purposes.

Timestamp: The date and time the correction window was opened. Note: when the correction is saved, the timestamp will be updated to the current time.

Type: Identifies the correction type to be applied.

Block: The type and number of the block that raised the error. Use the ‘Show in Mapping’ button to open the transformation mapping to see the block in context - particularly where it is getting its input from.

Input Field: The name of the input to the block. Where a block has several inputs that could be wrong (e.g. an arithmetic block), there will be several lines shown here.

Input value: The value submitted to the block when the error was created

Replacement value: Here the user can enter the new value to be used instead of the input value.

Applies to: A Specific sub Document or to all sub documents. Only available if the transformation mapping supports sub-documents. The specific sub-document is identified by number (subDocument 2 in the above example), and is the default value.

This correction type replaces the input values to the block identified in the error. It does not change the value in the input document, just the value being submitted to this block. In transformations where unit if work per sub-document is not true, an error is set on each occurrence where a repeating group in the input supplies an invalid value to the block, but the correction will apply to all occurrences that have the same input value. For example, if the incoming document has four repeats, with successive values abc, abc, xyz, and abc, all of which are invalid for an addition block say, four errors will be recorded (one for each repeat). However, if you correct the first abc to 123, this correction will apply to the other instances of abc (the second and fourth groups). It will not apply to the xyz instance.

Where unit of work per sub-document is true, you can choose to apply the replacement value to the single sub-document identified, or to all sub-documents. When applied to all sub-documents, it will only replace values identical to those on the problem - e.g. if the date expected by a block should have ‘/’ as a separator between day, month and year (01/07/2014), but the dates in the document are missing this symbol (01072014), the block will set an error. If you correct 01072014 to 01/07/2015 it will apply to any sub document where the input is 01072014, but will not apply if the date is different - 01012015 would need a separate correction.

Input Fields Replacement Values #

Figure 37 View document, input replacement correction

Document: The unique identifier of the document being corrected. Corrections made on this window will only apply to this document. They will not apply to other documents using the same transformation mapping.

User: The name of the user making the correction. This is retained for audit purposes.

Timestamp: The date and time the correction window was opened. Note: when the correction is saved, the timestamp will be updated to the current time.

Type: Identifies the correction type to be applied.

Show All Input Fields: A checkbox to show all input fields (ticked) or only the ones that may have caused the problem you are creating a correction for (unticked). This checkbox is disabled if the list of input fields for the selected problem is empty.

Applies to: A specific sub Document or to all sub documents. Only available if the transformation mapping supports sub-documents. The specific sub-document is identified by number (sub Document 2 in the above example), and is the default value. If All Sub Documents is selected, the navigation widget allows the user to move between sub-Documents.

Properties of failing block If the correction is linked to a problem that is related to a transformation block, the properties of said block is shown here for reference.

Input field: If the transformation mapping does not support sub-documents, the input field column will show the whole document in a tree structure. If the mapping does support sub-documents, the tree structure will start at the sub-document on which the error was reported. If Applies to all sub-documents is chosen, the navigation widget can be used to move between sub-documents, but only one will be shown at a time.
The input field tree structure will only show document contents for which an input pattern field exists. Data within the document (that can be seen on the content tab) that does not have an input pattern field will not be shown on the correction window.

Input value: The value of the corresponding input field in the selected page/sub document.

Replacement value: Here the user can enter the new value to replace the input value.
The input fields replacement values correction changes the value of the input data. (Note this is not reflected in document values shown on the contents tab). All the document data is open to change - any input field can be replaced, not just the data related to the problem. Where a transformation mapping does not support sub-documents, you must enter a correction for every page on which there is an error. However, the correction window allows to enter more than one correction at a time. For example, say there are errors for missing data on the input accounting start and accounting end dates, and both these have occurred on two pages of the document. The problems tab will report four errors. But on the correction window for one error, you can enter the missing dates (both start and end), for both pages. When the document is re-transformed all four errors will be cleared.

Default values for group #

Figure 38 View document default group correction

Document: The unique identifier of the document being corrected. Corrections made on this window will only apply to this document. They will not apply to other documents using the same transformation mapping.

User: The name of the user making the correction. This retained for audit purposes.

Timestamp: The date and time the correction window was opened. Note: when the correction is saved, the timestamp will be updated to the current time.

Type: Identifies the correction type to be applied.

Mandatory group: The name (and path) of the mandatory group in the output pattern for which no data has been provided.

Field table: There is one row in the table for each field in the group. All fields in the group are shown, whether or not they themselves are mandatory or not.

Field Name: The name of the field within the group.

Default Value: Here the user can enter a default value to be used if there is no value from the mapping.

Applies to: A Specific Sub Document or to all sub documents. Only available if the transformation mapping supports sub-documents. The specific sub-document is identified by number (sub Document 2 in the above example), and is the default value…
The default values for group correction is available when a mandatory group is unpopulated in the output pattern. A default value can be set for any or all the fields in the group. If the transformation mapping does not support sub-documents, the default values are used for all pages where the mandatory group is missing, but the default values do not override values provided by the mapping. If the transformation mapping does support sub-documents, the user can choose if the default values should apply to a single sub-document or to all. If all is chosen, then again the default values do not override values for sub-documents where a value is provided by the mapping.

It is not uncommon for a mandatory group to contain individual fields that are also mandatory. The mandatory group error will take precedent over the mandatory field error (which will not set a problem if the whole group is missing). If a default value is entered for a non-mandatory field in the group, but no value is entered for a mandatory one, the next time the transformation is made, the group error will be resolved, but new errors for any missing mandatory fields will be set instead.

Default value #

Figure 39 View document default value correction

Document: The unique identifier of the document being corrected. Corrections made on this window will only apply to this document. They will not apply to other documents using the same transformation mapping.

User: The name of the user making the correction. This retained for audit purposes.

Timestamp: The date and time the correction window was opened. Note: when the correction is saved, the timestamp will be updated to the current time.

Type: Identifies the correction type to be applied.

Mandatory field: The name (and path) of the mandatory field in the output pattern for which no data has been provided.

Default Value: Here the user can enter a default value to be used if there is no value from the mapping.

Applies to: A Specific Sub Document or to all sub documents. Only available if the transformation mapping supports sub-documents. The specific sub-document is identified by number (sub Document 2 in the above example), and is the default value…
The default value correction is available when a mandatory field is unpopulated in the output pattern. If the transformation mapping does not support sub-documents, the default value is used for all pages where the mandatory field is missing, but the default value does not override values provided by the mapping. If the transformation mapping does support sub-documents, the user can choose if the default value should apply to a single sub-document or to all. If all is chosen, then again the default value does not override values for sub-documents where a value is provided by the mapping.

View/edit Translation table #

Figure 40 View document, translation table correction

When a translate block with the parameter ‘Fail on Missing’ is true fails to find a row corresponding to the input data, one correction option is to edit the translation table. Selecting ‘View/edit Translation table’ from the context menu will open the View translation table window for the relevant table. From here, you can click the edit button and either change an existing value, or add a new one.

See the section in Reference Items - Translation Tables for further details.

Keep only last Decimal Separator #

Figure 41 View document keep last decimal correction

Document: The unique identifier of the document being corrected. Corrections made on this window will only apply to this document. They will not apply to other documents using the same transformation mapping.

User: The name of the user making the correction. This retained for audit purposes.

Timestamp: The date and time the correction window was opened. Note: when the correction is saved, the timestamp will be updated to the current time.

Type: Identifies the correction type to be applied.

Block: The type and number of the block that raised the error. Use the ‘Show in Mapping’ button to open the transformation mapping to see the block in context - particularly where it is getting its input from.

Applies to: A Specific Sub Document or to all sub documents. Only available if the transformation mapping supports sub-documents. The specific sub-document is identified by number (sub Document 2 in the above example), and is the default value.
The ‘Keep only last decimal separator’ correction is normally applied automatically. However, if a block expecting numeric input discovers the input has multiple decimal separators, and the automatic correction is not in force, the user has the opportunity to create a manual correction. There are no user inputs to this correction - see the section in Automatic Corrections above for a full description of the workings of this rule.

Limitations of the correction process #

The following problems cannot be resolved using the correction process:

• Error from Error (fixed) block.
• Error from Error (variable) block.
• Missing mandatory input field.
• Presence of forbidden input field.
• Presence of repeating element when Input Pattern field is not a repeating group.

Defined corrections #

Figure 42 View document, defined corrections tab

This tab shows the corrections that are currently in force for the document. The user can view or edit a correction (which will reopen the relevant correction window (see above). The user can also delete a correction. To view, edit or delete, select a single row in the table, an choose an option from the context menu.

No audit trail of corrections is maintained, so if you edit or delete a correction, the previous settings are lost when the edit is complete. Defined corrections are skipped in the transformation if they refer to a block which is subsequently removed from the transformation mapping.

Applied Corrections #

Figure 43 View document, applied corrections

The Applied Corrections tab shows the corrections that were applied to the last transformation attempt. Corrections from prior transformations that successfully processed one or more sub-documents, only may also be shown, but this tab is not intended to provide an audit trail of all applied corrections. Each time a [not yet transformed/processed] sub-document is attempted to be transformed, all ‘old’ applied corrections for this particular sub-document will be cleared at the start of the transformation.

In particular, where a transformation mapping supports sub-documents, it is possible for one sub-document to be completed using a correction that is later deleted in order to support the completion of another sub-document. By deleting the correction, it will no longer apply, and therefore is no longer shown on the applied corrections tab for the incomplete sub-documents. Applied corrections for successfully processed sub-documents will remain.

There are no facilities for the user to edit/view corrections on this tab.

Output #

Only present if the transformation was successful (or when processing using sub-documents, if partially successful), this tab shows the output messages/documents created from the incoming document. Clicking on a message/document will open the message/document properties.

If the document has been generated from external output pattern, an additional pop-up menu ‘Response’ will be enabled for each output document row. On click it will display response received from destination server.

History #

A new status history record is created each time a menu action is taken on a document. By default, the most recent action is first, and the record of the document being loaded is the last. Menu actions include transformation attempts or links to mappings. Corrections (and changes to corrections) are not recorded on the history tab.

Note #

An area where the user can add notes to the document. These notes are for information only, and have no effect on the transformation mapping process.

Rendered View #

Allows the user to view the incoming document in a formatted view. The view is managed by a stylesheet that renders the view in HTML. This makes the document more readable than in the content tab, and can be used to recreate a similar look to the original document before it was scanned. To see the rendered view, the Apply button must be pressed.

The stylesheet may be changed at any time, but the Apply button must be repressed each time before the view is shown. A default stylesheet may be defined for each transformation mapping. If there is no such default, the system default is used instead.