How do I create an import filter?

PackageGeneration Script:

A package generation script consists of an ISSUE line followed by any number of PacketSpecs.

CommentLine:

Comments can be placed almost anywhere to aid the readability. They are ignored by the filter function.

IssueLine:

The ISSUE line defines which issue of the Match-IT system the filter script is intended for. This is just passed through to the package script created by the filter.

PacketSpec:

A packet spec consists of a PACKET line followed by a StreamSpec, followed by any number of record or lookup specs, followed by an ENDPACKET line. The entire packet spec is processed for each CSV record. Each record spec represents a potential Match-IT record that can be created from a CSV record.

PacketName:

The packet name is used to identify which packet is to be processed when the filter script is run.

StreamSpec:

A StreamSpec consists of a STREAM line followed by up to 16 IgnoreSpec lines, followed by a FieldSpec line, followed by an optional GroupSpec, followed by an ENDSTREAM line. A stream spec defines how to interpret the CSV Records and Text Records presented to the filter. Each IgnoreSpec identifies lines that should be ignored. The FieldSpec identifies the field boundaries within the record. The GroupSpec defines the Group structure (if present). The fields defined in the FieldSpec at this level define the group header fields. These are given field numbers from 1. These are prepended to each group line detected.

IgnoreSpec:

An ignore spec defines a Condition for those records that should be ignored. If the condition is met the record is discarded and no information is extracted from it. There can be up to 16 ignore specs in total. Those defined before the GroupSpec apply only to the group header lines. Those defined within the group apply only to the group detail lines.

GroupSpec:

A group spec consists of a GROUP line followed by up to 16 IgnoreSpec lines followed by a FieldSpec line, followed by an ENDGROUP line. A group spec defines how to interpret the lines within a Group. A Group is all the lines between, and including, the lines that meet the Condition on the GROUP line and the Condition on the ENDGROUP line. The fields defined in the FieldSpec at this level define the group detail fields. These are given field numbers that follow on from the group header fields defined in the StreamSpec. For each group line, the header fields are prepended. A typical use for this is to break up a file that contains sub-sections, e.g.

 SubSection Header Line
         SubSection Detail Line 1
         SubSection Detail Line 2

         ...
         SubSection Detail Line N

Can be broken up into:

 SubSection Header Line        SubSection Detail Line 1
 SubSection Header Line        SubSection Detail Line 2

 ...
 SubSection Header Line        SubSection Detail Line N

Note: The 'WHEN' condition of a group start is referencing stream fields, but the 'WHEN' condition of a group end is referencing group fields.

FieldSpec:

A field spec defines the boundaries of a field in an input record. A field can be either a FixedField or a CSVField. The reference point for the start of each field is the end of the previous field. Thus: F#1,F#2[16] means field 1 extends to a separator, and field 2 is the next 16 characters. They must be defined in numerical order.

FixedField:

A fixed field is a field that is defined by a fixed number of characters. The number between the square brackets [] defines the number of characters occupied by the field. The start of the field is defined as the end of the preceding field, irrespective of the type of the previous field. Thus: F#1,F#2[22] defines that field 2 is 22 characters following on from wherever field 1 stopped. Similarly, F#1[10]F#2[22] defines that field 2 starts at position 11 and occupies the next 22 positions.

CSVField:

A CSV field is defined to extend from the end of the previous field to the next separator. Thus: F#1[10]F#2,F#3 defines that field 2 starts at position 11 and extends to the next separator, and field 3 starts from there, and so on.

LookupSpec:

A LookupSpec consists of a LOOKUP line, followed by any number of FieldNameSpec lines, followed by any number of RecordImage lines, followed by an ENDLOOKUP line. A LookupSpec where a 'FileId' is given defines a record to be looked up in the Match-IT database. The RecordImage must load the appropriate key fields. There may be multiple lookups on the same file. Fields looked up may be referenced in the RECORD: clause using the 'L#' context notation. Fields are referenced in the order specified in the field name list, e.g. L#?:F#1 is the first L#?:F#2 is the second, etc. The first fields in the list must match some key in the file. If no FIX: clause is present, the LOOKUP is exact, i.e. an exact match must be found. If FIX: is present, the lookup will find an exact match for the fields up to and including the FIX: field, then the nearest match for the rest. (It uses a zfSet/zfNext pair.) Lookups are performed in the order they are defined. A failed lookup leaves all fields blank when referenced. All lookups are done in lookup number order and before any record image is created. The lookup is only performed if the WHEN: conditions are met. If the conditions are not met, all lookup fields are blank when referenced.

The lookup number must be 1 or greater. The lookup number 0 is a special case used to access pre-defined values. There is a fixed reportoire of pre-defined values. As of 30/01/03 this reportoire is:

 F#1 = today's date
 F#2 = the time now
 F#3 = logged in user

A LookupSpec where the 'SCRIPT' option is given just evaluates fields for easier reference elsewhere. This is useful to minimise repetition and centralise 'knowledge'.

An example use might be to encapsulate things like: '[F#3][F#3![F#4]]'. These evaluations can also be cascaded, so they can be used to evaluate multiple choices.

Example:

LOOKUP:1 IN: SCRIPT
   [F#1=[X]Yes][F#1=[Y]No]
ENDLOOKUP:

!L#1:F#1 is now either 'Yes' or 'No' or blank.

LOOKUP:2 IN: SCRIPT
   [L#1:F#1][L#1:F#1!Illegal]
ENDLOOKUP:

!L#2:F#1 is now either 'Yes' or 'No' or 'Illegal'

RecordSpec:

A record spec consists of a RECORD line, followed by any number of FieldNameSpec lines, followed by any number of record images followed by an ENDRECORD line. Each record spec represents a record in one of the Match-IT files. A record spec is only considered if its 'WHEN:' conditions are TRUE. If no conditions are given then TRUE is assumed. The conditions are considered strictly in sequence. OR:'s are considered in turn, AND:s are considered until FALSE is detected. An FNumber is referring to the number of the field in the source record. The first is field 1. The whole input record is field 0.

FieldNameSpec:

A FieldNameSpec defines the logical order of the fields in a Match-IT database for the purposes of the import filter. It need not be the same as the physical order. Each name given must correspond to a name of a field in the associated file. The order the names are given defines the order they are being created in the RecordImage. See below for a full description of a field name spec. When the record is constructing a DO operation parameter list, the FieldNameSpec must still be present, with one name per parameter, but the actual names are arbitrary.

FileId:

A file ID is a short hand name for one of the Match-IT files. They consist of three letters. The list of valid file IDs is available within the on-line help system.

DropId

A drop Id is the name of a DO operation. The list of valid DO operations is available within the on-line help system.

SegmentName:

This is a name, for reference purposes within the script, of a group of related records. Every record created as a result of the associated record spec will be placed in the same segment. The import system will treat each segment as an indivisible unit for importing purposes. The whole segment or none of it will be imported.

Options:

These are just passed through to the package script. However, if the [DUPS] option is set, the filter will only add records to the associated segment if there isn’t  an identical record there already. The only significance of this is to save memory space while the filter script is being processed.

Condition:

A condition consists of a Field Value, an operation and a MatchPattern. If the match pattern matches the field value referenced according to the operation, the condition is TRUE. Otherwise it is FALSE. This is used to control records that are ignored, group boundaries and records created. The operation defines the test to perform.

FieldValue:

A field value is a reference to a field from the current input or a lookup, or a special value. The FieldContext defines where the field value is to come from. The FieldRef defines which field is being referenced. If the context is omitted, S#1 is assumed.

FieldRef:

A field ref. identifies which field in the context is being referenced. The first field in a record is field 1, the next field 2 and so on. The Fnumber indicates which field is being referenced. The [F#nnn] string is replaced by the current value of the field. If a FieldSymbol is present then only that element of the field is being referenced.

FieldContext:

This defines where the field is to come from. It can be either from the input stream, designated by ‘S#’, or a lookup, designated by ‘L#’.

FieldSymbol:

A field symbol identifies an element within the field. These elements correspond to the Match-IT LType elements. It is most often useful when taking looked-up fields apart, for example, extracting the units from a measure.

WhenOpCode:

This defines the type of match to perform on the match pattern. The possibilities are:
 ‘==‘        the field value must be the same as the pattern
 ‘<>‘        the field value must be different to the pattern
 ‘<<‘        the field value must contain the pattern somewhere
 ‘>>‘        the field value must not contain the pattern anywhere
 ‘!<‘        the field value must start with the pattern
 ‘>!’        the field value must end with the pattern.
 ‘++’        the field value must have changed form the last record

MatchPattern:

This is the pattern to match against the associated field value. The case of letters is ignored. Any leading and trailing single quote characters (‘) are removed. The MatchPattern for the ‘++’ operation must be [].

RecordImage:

The record image contains the fields of the associated record that are to be used to create, or lookup, a Match-IT file record. It consists of any number of Text strings or FieldText values. Each can either be placed in the image unconditionally or only when a specified field has a value or does not have a value. An implied package separator exists between each line of a record image. Thus:
 [F#1]
 [F#2]
is equivalent to:
 [F#1] | [F#2]

The changes made when constructing the image are: to replace FieldNumber strings with the current value of the field; to remove leading spaces from the line; and to translate sub-strings of the form '<##>' into the ASCII char that has the code '##'.

FieldText:

This represents a ‘conditional’ field. The first FieldValue defines the field whose value is to be checked. The FieldOpCode defines the type of check to apply. If the condition is ‘TRUE’ then the second FieldValue is placed in the record image, otherwise nothing is placed in the image.

FieldOpCode:

This defines the type of check to perform for a conditional field. A ‘?’ means the field condition is only ‘TRUE’ when the preceding FieldValue has a value. A ‘!’ means the opposite. The condition is ‘TRUE’ only when the associated FieldValue does not have a value. An ‘=[text]’ means the field condition is only TRUE when the preceding FieldValue has the value ‘text’, A ‘~[text]’ means the opposite to ‘=’, i.e. it’s TRUE when the field is not equal to ‘text’.