Automated Ingestion¶

This guide walks you through the steps to configure a PHC Project to automate the process of ingesting test files from different vendors.

Ashion¶

This section describes how to automate the processing of Ashion test files that are bundled as TAR files. The general flow for the automation is as follows:

• The Ashion TAR file is uploaded to a PHC Project via SFTP.
• A preconfigured File Action is triggered that will run the Ashion conversion CWL workflow to create the omic files.
• The output files from the CWL workflow are added to the PHC project. One of these output files is a test manifest file which will start the ingestion of the omic data.

Steps¶

NOTE: You must have Administer Rules access for the PHC Project that will be receiving the Ashion TAR files to perform these steps.

1. From the PHC Web UI, set the project that is to receive the Ashion TAR files as the active project.
2. Create a SFTP user following the instructions from the SFTP user guide. When creating the SFTP user you can specify a folder where all uploaded files will be stored. It is recommended to specify a folder here to keep the TAR files from being stored at the project root. A value of ashion would store all of the TAR files in a folder named ashion.
3. Download ashion_ingest_workflow.cwl located on Dockstore from here. There is an Export as ZIP option as well as a direct TRS link that you can click to save the CWL file.
4. Upload the ashion_ingest_workflow.cwl file to the target PHC project that will be storing the Ashion test files.
5. Create a File Action that will trigger a the ashion_ingest_workflow.cwl when a new TAR file is uploaded by following the directions here. For the File Name Pattern field enter the folder you specified for the SFTP user in step 1 along with a wildcard pattern to match the TAR files that will be uploaded. E.g. ashion/*.tar.gz. For the workflow master file pick the ashion_ingest_workflow.cwl file that you just uploaded to the project. For the workflow output folder, enter: .lifeomic/ashion. On the parameters screen, enter {{name}} for the source string parameter and GRCh37 for the reference parameter. There are no additional dependencies that need to be added for this workflow.

With the SFTP user created and the File Action in place for the project, you can now verify the configuration by simply uploading an Ashion TAR file to the project using the SFTP user created in step 1. Monitoring the ingestion process for progress and errors from the Omics Dashboard in the PHC Web UI.

Foundation Medicine¶

This section describes how to automate the processing of Foundation Medicine XML test files. The general flow for the automation is as follows:

• The XML file is uploaded to a PHC Project via SFTP.
• A preconfigured File Action is triggered that will run the Foundation conversion CWL workflow to create the omic files.
• The output files from the CWL workflow are added to the PHC project. One of these output files is a test manifest file which will start the ingestion of the omic data.

Steps¶

NOTE: You must have Administer Rules access for the PHC Project that will be receiving the Foundation XML files to perform these steps.

1. From the PHC Web UI, set the project that is to receive the Foundation XML files as the active project.
2. Create a SFTP user following the instructions from the SFTP user guide. When creating the SFTP user you can specify a folder where all uploaded files will be stored. It is recommended to specify a folder here to keep the XML files from being stored at the project root. A value of foundation would store all of the XML files in a folder named foundation.
3. Download foundation_ingest_workflow.cwl located on Dockstore from here. There is an Export as ZIP option as well as a direct TRS link that you can click to save the CWL file.
4. Upload the foundation_ingest_workflow.cwl file to the target PHC project that will be storing the Foundation XML files.
5. Create a File Action that will trigger a the foundation_ingest_workflow.cwl when a new XML file is uploaded by following the directions here. For the File Name Pattern field enter the folder you specified for the SFTP user in step 1 along with a wildcard pattern to match the XML files that will be uploaded. E.g. foundation/*.xml. For the workflow master file pick the foundation_ingest_workflow.cwl file that you just uploaded to the project. For the workflow output folder, enter: .lifeomic/foundation. On the parameters screen, enter {{name}} for the source string parameter and GRCh37 for the reference parameter. There are no additional dependencies that need to be added for this workflow.

With the SFTP user created and the File Action in place for the project, you can now verify the configuration by simply uploading an XML file to the project using the SFTP user created in step 1. Monitoring the ingestion process for progress and errors from the Omics Dashboard in the PHC Web UI.