Configuration and Maintenance

Learn how to configure and maintain a TIES installation

Configuration

The TIES node can be configured using the TIES Control Panel that is bundled with every TIES installation. The control panel is found in the <tiesinstallation>\configuration folder. There are two tabs in the control panel: The Configuration tab and the Utilities Tab. In the Configuration tab, you can see a list of TIES modules that are available for configuration on the current server. If you do not see a module, make sure it is in fact installed on this server and you are running the control panel from within the correct installation folder. The possible configuration modules are:

Database
DatabaseControlPanel

This selection lets you configure the public and private database parameters for this server. They are configured on separate tabs.  If you have multiple servers, you will need to run the control panel and make changes on each of the servers.
Public Database Tab
Provide the database connection parameters for the public/de-identified database. If you have changed your database password, or server location, you can specify it here.
Private Database Tab
Provide the database connection parameters for the private/identified database. If you have changed your database password, or server location, you can specify it here.
Processing Status Tab
This tab shows you the current processing state of the documents in the database. You can refresh it by clicking the Refresh button at the bottom. See Maintenance for more information.

TerminologyControlPanel

Lets you configure the terminology to be used by the coder and the search service.

DelimitedControlPanel

Configure properties for the Delimited File Importer.

HL7ControlPanel

DeidControlPanelConfigures the DeidPipeController.

CoderControlPanelConfigures the TiesPipeController

IndexerControlPanel

Configures the DBIndexPipeController

LDAP/Active Directory Configuration

TIES supports LDAP authentication. LDAP is a protocol that is supported by leading directory service providers, including Microsoft Active Directory. Before you begin, you will need to know the following information:

  • LDAP Server URL for your network.
  • LDAP username format. Typically this involves the domain name along with the username in a specific format.

Once you have these, you can proceed as follows:


  1. Modify to include the LDAP parameters in the ‘ActivityProdAnonymousEncryptedServicesDispatcher.properties’ file found in the <tiesinstallation>/tomcatPub/webapps/wsrf/WEB-INF/etc/dai folder.

    LDAP configuration requires three parameters for each network.

    Parameter Description
    caties.ldap.provider.domain.0 Email Domain identifying the network. User accounts with and email address with this domain will use this LDAP configuration for authentication
    caties.ldap.provider.url.0  LDAP Server URL for this network.
    caties.ldap.principal.pattern.0  How username is included in the Principal. Username is indicated with @USERNAME@.
    Example
     caties.ldap.provider.domain.0 = name.edu
     caties.ldap.provider.url.0 = ldaps://namead.univ.domain1.edu:636
     caties.ldap.principal.pattern.0 = @USERNAME@@name.edu

    Configuring two or more networks

    If your users may belong to more than one network, you can provide another set of configuration parameters by incrementing the number at the end of the parameter name.

    Example
     #Network A
     caties.ldap.provider.domain.0 = domain1.edu
     caties.ldap.provider.url.0 = ldaps://domain1ad.univ.domain1.edu:636
     caties.ldap.principal.pattern.0 = @USERNAME@@domain.edu
     #Network B
     caties.ldap.provider.domain.1 = domain2.edu
     caties.ldap.provider.url.1 = ldap://ldap.domain2.com:389
     caties.ldap.principal.pattern.1 = 1domain2\\@USERNAME@

  2. Copy the LDAP configuration parameters to ‘ActivityTestAnonymousEncryptedServicesDispatcher.properties’ found in the same folder.

  3. Restart Public Tomcat for the changes to take effect.

  4. Login into the TIES client as Administrator and Enable LDAP authentication for your organization in the Node Settings Dialog. nodeSettings

  5. Enable LDAP authentication for each user account separately. You will have the opportunity to change their username and email address at this time to match their network username and email address.
    enableLDAPUser


Maintenance

From the perspective of maintenance, TIES has two components: data processing and web services. The data processing component consists of the database and the processing pipelines. The web services component consists of the tomcat web servers.


Monitor Processing Status

You can monitor the status of the processing from the TIES Control Panel’s Database Configuration screen’s ‘Processing Status’ tab. Database Status Identified Documents Status These status types are set on the ties_private.ident_document table’s application_status column.

  • NOPATIENT – No patient was found attached to this document
  • DEIDENTIFYING – Pending de-identification by DeidPipeController
  • PURGE – Manually set by user to indicate this document needs to be deleted. Run Database Cleanup Utility to do actual deletion of PURGE records.
  • DELETING – Data Cleanup Utility is in the process of deleting this document.
  • IDLING – All processing finished for this document.
  • NOSECTIONS – No sections were found for this document.
  • DEIDERR – De-identification Error
  • SAVEERROR – Save Error. Occurs when any pipeline has a problem saving changes to database.
  • SYNTHERROR – Synthesis error.
  • SYNTHESIZING – Pending concatenation of document sections into document text by HL7PipeController or DelimitedFileImporter
  • IMPORTERROR – Import Error
  • OMITTING – Skipping this document from further processing due to missing required data fields like observation date or patient information.
  • EMPTYREPORT – No text found or no usable sections for this documents

De-Identified Documents Status These status types are set on the ties_private.ident_document table’s application_status column.

  • NOPATIENT – No patient was found attached to this document
  • CODING -Pending coding by TIESPipeController
  • PURGE – Manually set by user to indicate this document needs to be deleted. Run Database Cleanup Utility to do actual deletion of PURGE records.
  • DELETING – Data Cleanup Utility is in the process of deleting this document.
  • DEINDEXING – Data Cleanup Utility is in the process of de-indexing this document.
  • INDEXING – Pending indexing by the IndexPipeController
  • IDLING – All processing finished for this document.
  • CODINGERR – Coding Error
  • NOSECTIONS – No sections were found for this document.
  • INDEXNGERR – Indexing Error
  • SAVEERROR – Save Error. Occurs when any pipeline has a problem saving changes to database.
  • EMPTYREPORT – No text found or no usable sections for this documents
  • <DOCTYPE>_CODING_INPROGRESS – Is currently being coded by the TiesPipeController.
  • <DOCTYPE>_INDEXING_INPROGRESS – Is currently being indexed by the IndexPipeController.

For installations using the JavaMessaging Service Queue and JMS processing pipelines:

  • <DOCTYPE>_QUEUEING_ERR – Error when attempting to add this document to the queue.
  • <DOCTYPE>_CODING_QUEUE – In the CODING queue awaiting processing.
  • <DOCTYPE>_INDEXING_QUEUE – In the INDEXING queue awaiting processing.

Reprocessing Records

Documents are processed in several stages. each stage is run by a different processing pipeline as show below. The HL7ImportPipeController in the figure below, can be replaced by the DelimitedFileImporter, or your own custom import service.  See Loading You Data for more information. tiespipe You can force the reprocessing of documents through any stage, by setting the appropriate value in the APPLICATION_STATUS column for either the ties_private.IDENT_DOCUMENT or ties_public.DOCUMENT tables. After re-processing is triggered at a particular stage, the record has to go through all downstream stages before any changes are reflected in the search results. Select a stage below for instructions:

To re-import your documents. you can run the input files again through the importer. You can do that by copying the files from the ‘processed’ directory back into the input directory for the importer.

The HL7 Importer and the Delimited File importer both check if a report is already present in the database, and only overwrite if the report in the input file is newer than the copy in the database. So if you want a true re-import, you must clear all existing data from the database before starting the import process.

Set the ties_private.ident_document table’s APPLICATION_STATUS column value to ‘DEIDENTIFY’ for all rows that you want re-processed by the DeidPipeController.

 

Why would you need to re-deidentify?

  • The document text was changed.
  • De-identifier was changed or updated.
Set the ties_public.document table’s APPLICATION_STATUS column value to ‘CODING’ for all rows that you want re-processed by the TIESPipeController.

 

Why would you need to recode?

  • The terminology was updated.
  • The document text was changed.
Set the ties_public.document table’s APPLICATION_STATUS column value to ‘INDEXING’ for all rows that you want re-processed by the IndexPipeController.

Why would you need to re-index?

  • Terminology was changed.
  • Indexer algorithm was changed.

Remove Tags

Tags information is stored in the document_attr_value table. The full_name is set to ties.model.tag and the short_value column contains the tag name. You can remove specific tags from TIES as follows:

  1. Turn off the DeidPipeController.
  2. Set the application status of the ident_document rows which will be updated to DEIDENTIFYING as follows:
    UPDATE ties_private.ident_document 
    SET application_status = 'DEIDENTIFYING' 
    WHERE id IN (
           SELECT * FROM ties_private.document_attr_value 
           WHERE full_name = 'ties.model.tag' AND short_value = ‘<tagnametodelete>’
    );
  3. Delete the document_attr_value rows
    DELETE FROM ties_private.document_attr_value 
    WHERE full_name = 'ties.model.tag' AND short_value = ‘<tagnametodelete>’

Remove Individual Records

You can remove specific records from TIES as follows:

  1. Identify the rows in the ties_private.ident_document table for the records to be removed.
  2. Set the APPLICATION_STATUS to ‘PURGE’ for all the rows identified in (1).
  3. Run the Data Cleanup Utility.

Remove All Data

If you need to clear out all data from the database as follows:

  1. Connect to the database server and run the following SQL script. This will delete and re-create empty database schemas:
    DELETE ties_private;
    DELETE ties_public;
    CREATE SCHEMA ties_private;
    CREATE SCHEMA ties_public;
    
  2. Run the Database Updater Utility. This will recreate empty TIES tables in the ties_private and ties_public schemas.
  3. Delete all files in /indices/PRODINDEX. This will delete the search index.

Utilities

The HIPAA Scrubber Utility looks for patient names, initials, SSNs, MRNs and report identifiers in the de-identified text of the report and optionally removes them from the text. HIPAA Scrubber

 

The Database Updater Utility, updates existing TIES database schemas to be compatible with the installed version of TIES. This utility can also be run to create empty TIES tables in a new schema.

The Document Exporter Utility lets you export identified and de-identified documents from the TIES database to text files. You can specify a list of IDs to export.

The data cleanup utility performs general database maintenance for the TIES database. It will cleanup any PURGE records  check the database and index integrity, and make sure the index and the databases are in sync. You can run the utility in the Analyze mode or Cleanup Mode. The former just performs an analysis and shows you results but does not change anything in the database or index. The Cleanup mode goes ahead and fixes any issues it finds.

Data Cleanup

The Structured Data Loader Utility can be used to load structured data into TIES. It is very similar to the Delimited File Importer service, the only difference being that this utility is for one off imports. See DelimitedFileImporter for more information.Structured Data Loader

 

Report level metadata

You must have the ties.model.record_id column present in the input file. This is typically the source system unique id for the report  for e.g. accession number.


############# THIS IS TEXT CONFIG FILE FOR REPORT LEVEL STRUCTURED DATA IMPORT ##############
----------------------------------------------------------------------------------------------
# Include |Visible|PHI |Codeable |Indexable | Namespace | Attribute | Data Type | Custom Data |Pretty Name |Description
#---------------------------------------------------------------------------------------------
 true |true |true |true |false |ties.model | record_id | TEXT | | |
 true |true |false |false |true |nmvb | tumorsite | TEXT | |Tumor Site |Primary site of the tumor
 true |true |false |true |true |nmvb | tumordimension | TEXT | |Tumor Dimension |
 true |true |false |true |true |nmvb | hist_type | | TEXT | |Histological Type
 true |true |false |true |true |nmvb | surgical_margins | | BOOLEAN | |Surgical Margins
 true |true |false |true |true |nmvb | hist_stain_profile | TEXT | |Histochemical Special Stain Profile|

Patient level metadata

You must have either the ties.mode.mrn or ties.model.ssn columns present in the input file to help uniquely identify the patient.


############# THIS IS TEXT CONFIG FILE FOR PATIENT LEVEL STRUCTURED DATA IMPORT ##############
----------------------------------------------------------------------------------------------
# Include |Visible|PHI |Codeable |Indexable | Namespace | Attribute | Data Type | Custom Data |Pretty Name |Description
#---------------------------------------------------------------------------------------------
 true |true |true |true |false |ties.model | mrn | TEXT | | |
 true |true |true |true |false |gru | address | TEXT | | |
 true |true |true |true |false |gru | home_phone | TEXT | | |
 true |true |false |true |false |gru | death_date | DATE | yyyyMMdd | |
 true |true |false |true |false |gru | death_status | BOOLEAN |