Chapter 2. Gateway concepts and processing 17
The next section goes through each stage and module of the data mediation
process for the gateway process. Note that we use the terminology IN_DIR,
INT_DIR, and OUT_DIR as variables that are specified from the gateway
Depending on whether the data extraction is push or pull, as explained in
“Extract” on page 7, this optional stage enables the configuration of the transfer
of data from the source. It supports the SFTP, SCP, FTP, RCP, and local CP
(copy) file transfer mechanisms. The transfer rule allows the configuration of
multiple instances, so extraction of files from different source locations are
possible. However, these rules are executed in sequence, not in parallel.
Configuration of this stage is optional when data is pushed to the gateway or
provided by another external tool. The configuration file for this stage is in
In this stage, the engine parses the raw data source file, which could be either in
a vendor specific data format, or a third-party data format, to produce the data in
Parser Intermediate Format (PIF). It picks up the raw data source file from
IN_DIR, and temporarily stores it as a PIF file in INT_DIR.
The parser engine executes based on a specific vendor specific engine rule.
Each vendor gateway provides its own configuration module. For example, to
parse Nokia NSS format data, the NOKIA_XML engine rule is used. Another rule
uses the CSV_GATEWAY that we use for parsing Motorola GPRS CSV format
data. The configuration file for this stage is in EngineConfig.pm.
The post parser stage further processes the data through multiple standard and
vendor specific post parser rules, transforming data into the final desired output
format for successful loading that conforms to the system requirements.
There are several transformation functions possible in this stage. These rules are
implemented in the gateway framework, so it is available for all vendors. The
following are some of the common rules:
Joining different PIF files using the JOIN rule.
Removing entire data records with the DATALINE_WHERE rule.