Configuration and Maintenance
Learn how to configure and maintain a TIES installationConfiguration
The TIES node can be configured using the TIES Control Panel that is bundled with every TIES installation. The control panel is found in the <tiesinstallation>\configuration folder. There are two tabs in the control panel: The Configuration tab and the Utilities Tab. In the Configuration tab, you can see a list of TIES modules that are available for configuration on the current server. If you do not see a module, make sure it is in fact installed on this server and you are running the control panel from within the correct installation folder. The possible configuration modules are:
- Database
- Terminology
- Delimited File Importer Service
- HL7 Importer Service
- Deidentifier Service
- Coder Service
- Indexer Service
- LDAP/Active Directory
This selection lets you configure the public and private database parameters for this server. They are configured on separate tabs. If you have multiple servers, you will need to run the control panel and make changes on each of the servers.
Public Database Tab
Provide the database connection parameters for the public/de-identified database. If you have changed your database password, or server location, you can specify it here.
Private Database Tab
Provide the database connection parameters for the private/identified database. If you have changed your database password, or server location, you can specify it here.
Processing Status Tab
This tab shows you the current processing state of the documents in the database. You can refresh it by clicking the Refresh button at the bottom. See Maintenance for more information.
LDAP/Active Directory Configuration
TIES supports LDAP authentication. LDAP is a protocol that is supported by leading directory service providers, including Microsoft Active Directory. Before you begin, you will need to know the following information:
- LDAP Server URL for your network.
- LDAP username format. Typically this involves the domain name along with the username in a specific format.
Once you have these, you can proceed as follows:
- Modify to include the LDAP parameters in the ‘ActivityProdAnonymousEncryptedServicesDispatcher.properties’ file found in the <tiesinstallation>/tomcatPub/webapps/wsrf/WEB-INF/etc/dai folder.
LDAP configuration requires three parameters for each network.
Parameter Description caties.ldap.provider.domain.0 Email Domain identifying the network. User accounts with and email address with this domain will use this LDAP configuration for authentication caties.ldap.provider.url.0 LDAP Server URL for this network. caties.ldap.principal.pattern.0 How username is included in the Principal. Username is indicated with @USERNAME@. Example caties.ldap.provider.domain.0 = name.edu caties.ldap.provider.url.0 = ldaps://namead.univ.domain1.edu:636 caties.ldap.principal.pattern.0 = @USERNAME@@name.edu
Configuring two or more networks
If your users may belong to more than one network, you can provide another set of configuration parameters by incrementing the number at the end of the parameter name.
Example #Network A caties.ldap.provider.domain.0 = domain1.edu caties.ldap.provider.url.0 = ldaps://domain1ad.univ.domain1.edu:636 caties.ldap.principal.pattern.0 = @USERNAME@@domain.edu #Network B caties.ldap.provider.domain.1 = domain2.edu caties.ldap.provider.url.1 = ldap://ldap.domain2.com:389 caties.ldap.principal.pattern.1 = 1domain2\\@USERNAME@
- Copy the LDAP configuration parameters to ‘ActivityTestAnonymousEncryptedServicesDispatcher.properties’ found in the same folder.
- Restart Public Tomcat for the changes to take effect.
- Login into the TIES client as Administrator and Enable LDAP authentication for your organization in the Node Settings Dialog.
- Enable LDAP authentication for each user account separately. You will have the opportunity to change their username and email address at this time to match their network username and email address.
Maintenance
From the perspective of maintenance, TIES has two components: data processing and web services. The data processing component consists of the database and the processing pipelines. The web services component consists of the tomcat web servers.
Monitor Processing Status
You can monitor the status of the processing from the TIES Control Panel’s Database Configuration screen’s ‘Processing Status’ tab. Identified Documents Status These status types are set on the ties_private.ident_document table’s application_status column.
- NOPATIENT – No patient was found attached to this document
- DEIDENTIFYING – Pending de-identification by DeidPipeController
- PURGE – Manually set by user to indicate this document needs to be deleted. Run Database Cleanup Utility to do actual deletion of PURGE records.
- DELETING – Data Cleanup Utility is in the process of deleting this document.
- IDLING – All processing finished for this document.
- NOSECTIONS – No sections were found for this document.
- DEIDERR – De-identification Error
- SAVEERROR – Save Error. Occurs when any pipeline has a problem saving changes to database.
- SYNTHERROR – Synthesis error.
- SYNTHESIZING – Pending concatenation of document sections into document text by HL7PipeController or DelimitedFileImporter
- IMPORTERROR – Import Error
- OMITTING – Skipping this document from further processing due to missing required data fields like observation date or patient information.
- EMPTYREPORT – No text found or no usable sections for this documents
De-Identified Documents Status These status types are set on the ties_private.ident_document table’s application_status column.
- NOPATIENT – No patient was found attached to this document
- CODING -Pending coding by TIESPipeController
- PURGE – Manually set by user to indicate this document needs to be deleted. Run Database Cleanup Utility to do actual deletion of PURGE records.
- DELETING – Data Cleanup Utility is in the process of deleting this document.
- DEINDEXING – Data Cleanup Utility is in the process of de-indexing this document.
- INDEXING – Pending indexing by the IndexPipeController
- IDLING – All processing finished for this document.
- CODINGERR – Coding Error
- NOSECTIONS – No sections were found for this document.
- INDEXNGERR – Indexing Error
- SAVEERROR – Save Error. Occurs when any pipeline has a problem saving changes to database.
- EMPTYREPORT – No text found or no usable sections for this documents
- <DOCTYPE>_CODING_INPROGRESS – Is currently being coded by the TiesPipeController.
- <DOCTYPE>_INDEXING_INPROGRESS – Is currently being indexed by the IndexPipeController.
For installations using the JavaMessaging Service Queue and JMS processing pipelines:
- <DOCTYPE>_QUEUEING_ERR – Error when attempting to add this document to the queue.
- <DOCTYPE>_CODING_QUEUE – In the CODING queue awaiting processing.
- <DOCTYPE>_INDEXING_QUEUE – In the INDEXING queue awaiting processing.
Reprocessing Records
Documents are processed in several stages. each stage is run by a different processing pipeline as show below. The HL7ImportPipeController in the figure below, can be replaced by the DelimitedFileImporter, or your own custom import service. See Loading You Data for more information. You can force the reprocessing of documents through any stage, by setting the appropriate value in the APPLICATION_STATUS column for either the ties_private.IDENT_DOCUMENT or ties_public.DOCUMENT tables. After re-processing is triggered at a particular stage, the record has to go through all downstream stages before any changes are reflected in the search results. Select a stage below for instructions:
The HL7 Importer and the Delimited File importer both check if a report is already present in the database, and only overwrite if the report in the input file is newer than the copy in the database. So if you want a true re-import, you must clear all existing data from the database before starting the import process.
Why would you need to re-deidentify?
- The document text was changed.
- De-identifier was changed or updated.
Why would you need to recode?
- The terminology was updated.
- The document text was changed.
Why would you need to re-index?
- Terminology was changed.
- Indexer algorithm was changed.
Remove Tags
Tags information is stored in the document_attr_value table. The full_name is set to ties.model.tag and the short_value column contains the tag name. You can remove specific tags from TIES as follows:
- Turn off the DeidPipeController.
- Set the application status of the ident_document rows which will be updated to DEIDENTIFYING as follows:
UPDATE ties_private.ident_document SET application_status = 'DEIDENTIFYING' WHERE id IN ( SELECT * FROM ties_private.document_attr_value WHERE full_name = 'ties.model.tag' AND short_value = ‘<tagnametodelete>’ );
- Delete the document_attr_value rows
DELETE FROM ties_private.document_attr_value WHERE full_name = 'ties.model.tag' AND short_value = ‘<tagnametodelete>’
Remove Individual Records
You can remove specific records from TIES as follows:
- Identify the rows in the ties_private.ident_document table for the records to be removed.
- Set the APPLICATION_STATUS to ‘PURGE’ for all the rows identified in (1).
- Run the Data Cleanup Utility.
Remove All Data
If you need to clear out all data from the database as follows:
- Connect to the database server and run the following SQL script. This will delete and re-create empty database schemas:
DELETE ties_private; DELETE ties_public; CREATE SCHEMA ties_private; CREATE SCHEMA ties_public;
- Run the Database Updater Utility. This will recreate empty TIES tables in the ties_private and ties_public schemas.
- Delete all files in /indices/PRODINDEX. This will delete the search index.
Utilities
The Structured Data Loader Utility can be used to load structured data into TIES. It is very similar to the Delimited File Importer service, the only difference being that this utility is for one off imports. See DelimitedFileImporter for more information.
Report level metadata
You must have the ties.model.record_id column present in the input file. This is typically the source system unique id for the report for e.g. accession number.
############# THIS IS TEXT CONFIG FILE FOR REPORT LEVEL STRUCTURED DATA IMPORT ############## ---------------------------------------------------------------------------------------------- # Include |Visible|PHI |Codeable |Indexable | Namespace | Attribute | Data Type | Custom Data |Pretty Name |Description #--------------------------------------------------------------------------------------------- true |true |true |true |false |ties.model | record_id | TEXT | | | true |true |false |false |true |nmvb | tumorsite | TEXT | |Tumor Site |Primary site of the tumor true |true |false |true |true |nmvb | tumordimension | TEXT | |Tumor Dimension | true |true |false |true |true |nmvb | hist_type | | TEXT | |Histological Type true |true |false |true |true |nmvb | surgical_margins | | BOOLEAN | |Surgical Margins true |true |false |true |true |nmvb | hist_stain_profile | TEXT | |Histochemical Special Stain Profile|
Patient level metadata
You must have either the ties.mode.mrn or ties.model.ssn columns present in the input file to help uniquely identify the patient.
############# THIS IS TEXT CONFIG FILE FOR PATIENT LEVEL STRUCTURED DATA IMPORT ############## ---------------------------------------------------------------------------------------------- # Include |Visible|PHI |Codeable |Indexable | Namespace | Attribute | Data Type | Custom Data |Pretty Name |Description #--------------------------------------------------------------------------------------------- true |true |true |true |false |ties.model | mrn | TEXT | | | true |true |true |true |false |gru | address | TEXT | | | true |true |true |true |false |gru | home_phone | TEXT | | | true |true |false |true |false |gru | death_date | DATE | yyyyMMdd | | true |true |false |true |false |gru | death_status | BOOLEAN |