Automated Ingestion¶
This guide walks you through the steps to configure a PHC Project to automate the process of ingesting test files from different vendors.
Ashion¶
This section describes how to automate the processing of Ashion test files that are bundled as TAR files. The general flow for the automation is as follows:
- The Ashion TAR file is uploaded to a PHC Project via SFTP.
- A preconfigured File Action is triggered that will run the Ashion conversion CWL workflow to create the omic files.
- The output files from the CWL workflow are added to the PHC project. One of these output files is a test manifest file which will start the ingestion of the omic data.
Steps¶
NOTE: You must have Administer Rules
access for the PHC Project that will be receiving the Ashion TAR files to perform these steps.
- From the PHC Web UI, set the project that is to receive the Ashion TAR files as the active project.
- Create a SFTP user following the instructions from the SFTP user guide. When creating the SFTP user you can specify a folder where all uploaded files will be stored. It is recommended to specify a folder here to keep the TAR files from being stored at the project root. A value of
ashion
would store all of the TAR files in a folder namedashion
. - Download
ashion_ingest_workflow.cwl
located on Dockstore from here. There is anExport as ZIP
option as well as a direct TRS link that you can click to save the CWL file. - Upload the
ashion_ingest_workflow.cwl
file to the target PHC project that will be storing the Ashion test files. - Create a File Action that will trigger a the
ashion_ingest_workflow.cwl
when a new TAR file is uploaded by following the directions here. For theFile Name Pattern
field enter the folder you specified for the SFTP user in step 1 along with a wildcard pattern to match the TAR files that will be uploaded. E.g.ashion/*.tar.gz
. For the workflow master file pick theashion_ingest_workflow.cwl
file that you just uploaded to the project. For the workflow output folder, enter:.lifeomic/ashion
. On the parameters screen, enter{{name}}
for thesource
string parameter andGRCh37
for thereference
parameter. There are no additional dependencies that need to be added for this workflow.
With the SFTP user created and the File Action in place for the project, you can now verify the configuration by simply uploading an Ashion TAR file to the project using the SFTP user created in step 1. Monitoring the ingestion process for progress and errors from the Omics Dashboard in the PHC Web UI.
Foundation Medicine¶
This section describes how to automate the processing of Foundation Medicine XML test files. The general flow for the automation is as follows:
- The XML file is uploaded to a PHC Project via SFTP.
- A preconfigured File Action is triggered that will run the Foundation conversion CWL workflow to create the omic files.
- The output files from the CWL workflow are added to the PHC project. One of these output files is a test manifest file which will start the ingestion of the omic data.
Steps¶
NOTE: You must have Administer Rules
access for the PHC Project that will be receiving the Foundation XML files to perform these steps.
- From the PHC Web UI, set the project that is to receive the Foundation XML files as the active project.
- Create a SFTP user following the instructions from the SFTP user guide. When creating the SFTP user you can specify a folder where all uploaded files will be stored. It is recommended to specify a folder here to keep the XML files from being stored at the project root. A value of
foundation
would store all of the XML files in a folder namedfoundation
. - Download
foundation_ingest_workflow.cwl
located on Dockstore from here. There is anExport as ZIP
option as well as a direct TRS link that you can click to save the CWL file. - Upload the
foundation_ingest_workflow.cwl
file to the target PHC project that will be storing the Foundation XML files. - Create a File Action that will trigger a the
foundation_ingest_workflow.cwl
when a new XML file is uploaded by following the directions here. For theFile Name Pattern
field enter the folder you specified for the SFTP user in step 1 along with a wildcard pattern to match the XML files that will be uploaded. E.g.foundation/*.xml
. For the workflow master file pick thefoundation_ingest_workflow.cwl
file that you just uploaded to the project. For the workflow output folder, enter:.lifeomic/foundation
. On the parameters screen, enter{{name}}
for thesource
string parameter andGRCh37
for thereference
parameter. There are no additional dependencies that need to be added for this workflow.
With the SFTP user created and the File Action in place for the project, you can now verify the configuration by simply uploading an XML file to the project using the SFTP user created in step 1. Monitoring the ingestion process for progress and errors from the Omics Dashboard in the PHC Web UI.