Automated Ingestion¶

Automated Ingestion of test files can be set up for your PHC project.

Automated Ingestion Process¶

The automating ingestion process consists of uploading sequencing result files to a PHC Project via SFTP, File Service, cli or Jupyter Notebooks. The uploaded file triggers a pre-configured File Action that will run the vendor conversion CWL workflow to create the Omics files. One of these output files is a test manifest file which will start the ingestion of the Omics data.

Steps¶

1. Generate a CWL workflow and any required steps in the workflow to process the test data. The workflow steps should consist of Docker Containers.
   • The completed workflow should be uploaded to the PHC Tool Registry Service.
   • The individual Docker Containers (steps) may be uploaded to the PHC Tool Registry Service or any publicly available Container repository, i.e. DockerHub.
   • PHC currently supports (has the workflows and steps already generated) the following vendors: Ashion, Caris, Foundation Medicine, and NantOmics.
2. Create a File Action that triggers the workflow you had added to the PHC Tool Registry Service. This is done by uploading a file or set of files by following the directions found in the File Action article.
3. The configuration can be verified by simply uploading a vendor file to the specified directory that matches the file pattern used in the File Action creation. The upload will trigger the workflow which can be monitored from the Automation View. Once the workflow completes, the test indexing progress can be monitored from the Omics Dashboard.

Note: A manifest must have an MRN (Medical Record Number) for proper automated ingestion. If a manifest is missing the MRN, the Omics Dashboard will display an error in the Manifest file, such as, "No patient found for test". To resolve this, add the MRN by hovering your cursor on the row of the error file and a purple oval with tools appears. Click the edit icon. The New Patient Record window opens. Add the MRN number, then click Create.