Delimited File Importer

The Delimited File Importer reads delimited files and imports them into TIES

How To Use

The Delimited File Importer is installed in\services\DelimitedFileImporter. The installer also installs it as a Windows Service or a Linux daemon so that the pipeline automatically starts when the server starts up.

Step 1 – Modify the DelimitedDescriptor.cfg to fit your needs

This file is found in the \pipelines\DelimitedFileImporter\classes\config folder and is used by the importer to know what columns are present in the input files. The importer can be used to load new reports or structured data for existing reports. Based on your specific need, you will need to include the following columns in your descriptor and input file. The default file is set for loading new reports.

DelimitedDescriptor.cfg File Structure

Column Description
Include Whether to process and import into database
Visible Whether it is visible to downstream processing and to the client.
PHI Whether the attribute contains protected health information.
Codeable Whether the values should be coded using the TIES coder pipeline.
Indexable Whether the attribute values are to be searchable.
Namespace Namespace for attribute. Must be unique within the TIES database. Reserved namespaces: ties.model, ties.image
Attribute Attribute name. Must be unique within namespace.
Data Type Data type of the attribute. Possible values:

  • TEXT
  • BOOLEAN
  • NUMBER
  • DATE
  • CATEGORY
Visible Researcher
Custom Data Additional data based on data type.

Data Type Custom Data
DATE Date format using Java SimpleDateFormat syntax. e.g. yyyyMMdd or yyyyMMddhhmm
CATEGORY relative path to mapping file that maps the source system values to standard values. For e.g. M can be mapped to Male Gender
Pretty Name Human readable name of the attribute
Description Description of the attribute

Sample Descriptor Files

 


At the minimum you must provide the following data columns in your input file:

  • ties.model.mrn or ties.model.ssn – Used in combination with patient name and birthdate as a unique key for patient
  • ties.model.birth_date – Patient birth date.
  • ties.model.name  ( in the format: LastName^FirstName^MiddleName )
  • ties.model.gender – Patient gender. Use config/GenderConfig.txt mapping file to map source system values to TIES values.
  • ties.model.record_id – Unique id for the report. for e.g. accession no. for pathology reports.
  • ties.model.report_date – Date of the report creation. for e.g. collection date time for pathology reports.
  • ties.model.text – Entire Report text or section text enclosed in double quotes(“).  If your source system provides report text as separate sections, it is preferable to load individual sections and let TIES concatenate it into a final report.
  • ties.model.section_type – section header name. Use config/DelimSectionHeaderConfig.txt to map different section headers to section types. For e.g. you can map both Final Pathological Diagnosis and Final Anatomical Diagnosis to the “Final Diagnosis” section. Not required if you are including the complete report text in ties.model.text column.

Sample Descriptor File

############# THIS IS TEXT CONFIG FILE FOR DELIMITED TEST DATA INCLUDED WITH TIES #############

#----------------------------------------------------------------------------------------------
# Include |Visible|PHI |Codeable |Indexable | Namespace | Attribute | Data Type | Custom Data |Pretty Name |Description
#----------------------------------------------------------------------------------------------
 true |true |true |true |false |ties.model | mrn | TEXT | | |
 true |true |true |true |false |ties.model | birth_date | DATE | yyyyMMdd | |
 true |true |false |true |false |ties.model | gender | CATEGORY | config/GenderConfig.txt | |
 true |true |false |true |false |ties.model | race | CATEGORY | config/RaceConfig.txt | |
 true |true |false |true |false |ties.model | ethnicity | CATEGORY | config/EthnicityConfig.txt | |
 true |true |true |true |false |ties.model | ssn | TEXT | | |
 true |true |true |true |false |ties.model | name | TEXT | | |
 true |true |true |true |false |ties.model | record_id | TEXT | | |
 true |true |false |true |false |ties.model | section_type | CATEGORY | config/DelimSectionHeaderConfig.txt | |
 true |true |true |true |false |ties.model | report_date | DATE | yyyyMMdd | |
 true |true |true |true |false |ties.model | text | TEXT | | |

A sample descriptor file and input file can be found in the sampledata folder of your TIES installation. The descriptor file listed below lists the minimum columns that need to be present in the input file. The required virtualslide metadata attributes use the ties.image namespace. The ties.model.record_id helps uniquely identify the report that the virtualslide is related to.


############# THIS IS TEXT CONFIG FILE FOR DELIMITED VIRTUAL SLIDE DATA IMPORT ##############
---------------------------------------------------------------------------------------------
# Include |Visible|PHI |Codeable |Indexable | Namespace | Attribute | Data Type | Custom Data |Pretty Name |Description
#---------------------------------------------------------------------------------------------
 true |true |true |true |false |ties.model | record_id | TEXT | | |
 true |true |false |false |false |ties.image | path | TEXT | |Image Path |
 true |true |false |false |false |ties.image | server_type | TEXT | |Server Type |
 true |true |false |false |false |ties.image | server_url | TEXT | |Server URL |
 true |true |false |false |false |ties.image | name | TEXT | |Name |
 true |true |false |false |false |ties.image | description | TEXT | |Description |

 

Report level metadata

You must have the ties.model.record_id column present in the input file. This is typically the source system unique id for the report  for e.g. accession number.


 

############# THIS IS TEXT CONFIG FILE FOR REPORT LEVEL STRUCTURED DATA IMPORT ##############
----------------------------------------------------------------------------------------------
# Include |Visible|PHI |Codeable |Indexable | Namespace | Attribute | Data Type | Custom Data |Pretty Name |Description
#---------------------------------------------------------------------------------------------
 true |true |true |true |false |ties.model | record_id | TEXT | | |
 true |true |false |false |true |nmvb | tumorsite | TEXT | |Tumor Site |Primary site of the tumor
 true |true |false |true |true |nmvb | tumordimension | TEXT | |Tumor Dimension |
 true |true |false |true |true |nmvb | hist_type | | TEXT | |Histological Type
 true |true |false |true |true |nmvb | surgical_margins | | BOOLEAN | |Surgical Margins
 true |true |false |true |true |nmvb | hist_stain_profile | TEXT | |Histochemical Special Stain Profile|

Patient level metadata

You must have either the ties.mode.mrn or ties.model.ssn columns present in the input file to help uniquely identify the patient.


 

############# THIS IS TEXT CONFIG FILE FOR PATIENT LEVEL STRUCTURED DATA IMPORT ##############
----------------------------------------------------------------------------------------------
# Include |Visible|PHI |Codeable |Indexable | Namespace | Attribute | Data Type | Custom Data |Pretty Name |Description
#---------------------------------------------------------------------------------------------
 true |true |true |true |false |ties.model | mrn | TEXT | | |
 true |true |true |true |false |gru | address | TEXT | | |
 true |true |true |true |false |gru | home_phone | TEXT | | |
 true |true |false |true |false |gru | death_date | DATE | YYYYMMDD | |
 true |true |false |true |false |gru | death_status | BOOL | 

 

Step 2 – Extract data from LIS into a delimited file format

The Delimited File importer can import data from any delimited file. It uses a delimited file descriptor file that describes the different columns in the delimited file and what type of content it is.

Step 3 – Configure the pipeline

The main configuration files for this component are:

classes\caTIES.properties This is the master configuration file. This file contains parameters for all pipelines, so scroll to the section of the file that is relevant to this pipeline to make any changes. This file is configured by installer and you do not generally require to modify this file unless you want to tweak these parameters. Comments and inactive lines in the file start with a #. caties.document.type The document type your are importing. Default: PATHOLOGY. caties.hl7importer.directory.home The directory where the input HL7 files are stored. caties.hl7importer.sleepsize The time in ms. for which the pipeline sleeps between checks for new files. Default: one minute. caties.hl7importer.tag An optional tag to be applied to the imported reports.
classes\SectionHeaderConfig.txt Used to map section headers in the data to TIES standard section names. The LIMS column in the file should match the section headers in your HL7 file. See Section Configuration for more information
classes\EthnicityConfig.txt Specifies the mapping between your institution’s ethnicity value domain and the TIES standard ethnicity value domain.
classes\GenderConfig.txt Specifies the mapping between your institution’s gender value domain and the TIES standard gender value domain.
classes\RaceConfig.txt Specifies the mapping between your institution’s race value domain and the TIES standard race value domain.

Step 4 – Start the pipeline

On Windows you can start the pipeline by starting the HL7ImportPipeController service from Windows Services.  Alternatively you can also use the batch file in the \bin folder.

Step 4 – Monitor the log files

Monitor the log files to make sure that the files are being processed and there are no errors.

Troubleshooting

The Delimited File Importer is installed as a Windows service or a Linux daemon during installation. Check to make sure the service is running. In the absence of any input files, the Delimited Importer goes to sleep, but continues to run in the background.

Next, check the log files for any errors. When files are successfully processed, you will see messages in the service.log indicating the number of reports imported.

The log files are located under the logs folder of the pipeline.

logs\service.log Detailed diagnostic information about the pipeline.
logs\wrapper.log Windows service log

 Common Errors

Error

STATUS | wrapper  | 2006/12/05 16:35:27 | Launching a JVM... 
INFO   | jvm 5    | 2006/12/05 16:35:27 | Error occurred during initialization of VM 
INFO   | jvm 5    | 2006/12/05 16:35:27 | Could not reserve enough space for object heap 
ERROR  | wrapper  | 2006/12/05 16:35:27 | JVM exited while loading the application.

Solution

This occurs when the Java memory heap settings are requesting more memory than is available in the system. Edit the conf\wrapper.conf file. Towards the end of the file you should find the initial and maximum heap size parameters. Optimal settings would be at 80% of the available system memory available.