Nightly file uploads¶
The LifeOmic platform allows its users to upload files via the LifeOmic CLI. There are multiple ways to authenticate with the CLI, but for this document we’ll be using API Keys. Once an API Key is generated, the CLI can be used in conjunction with a cron job to automatically upload files each night.
API Key Creation¶
See the API Key Guide
Download and verify CLI¶
wget "https://github.com/lifeomic/cli/releases/download/v6.11.0/lo-linux-x64.zip" sudo apt-get install unzip unzip lo-linux-x64.zip rm lo-linux-x64.zip sudo mv lo /usr/local/bin lo --help
Setup CLI with API Key¶
lo setup ? Pick a LifeOmic environment: prod-us ? Specify a default LifeOmic account to use: myaccount ? Use API key for authentication? Yes ? API Key:
After pasting the API Key, you should be ready to use the CLI for file uploading. Issue the following command as a test to verify authentication is all setup:
lo projects list
Prepare local folders and PHC project¶
mkdir uploads mkdir logs lo projects create "Files Test"
Replace "Files Test" with whatever the project should be called.
Prepare upload script¶
Copy project ID
lo projects list Example response: items: - id: f3a220a0-b22b-41c3-9f77-2ce7cf37c93b name: Files Test
Copy the id of the project you created.
#!/bin/bash set -u set -e set -x if pgrep --full "lo files upload" ; then echo "Already running" exit 0 fi trap "echo Process killed" SIGINT SIGTERM SIGABRT echo `date '+%Y-%m-%d %H:%M:%S'` cd /home/ubuntu/uploads /usr/local/bin/lo files upload ./ f3a220a0-b22b-41c3-9f77-2ce7cf37c93b --recursive --overwrite --delete-after-upload
NOTE: In this example we’re using /home/ubuntu/uploads. You may need to change that line of the script to match the directory you’re using.
NOTE: Make sure to replace the UUID (which starts with
f3a220a0-) above with your project ID.
Add execution permissions and verify a file upload¶
chmod +x lo-upload.sh echo "test" > uploads/test.txt ./lo-upload.sh
Verify you see output similar to this:
Upload complete: test.txt, ID: `d4d9d8fc-b96b-49ad-a43c-c24f54b29941`
Also, issuing the following command should verify the uploaded file:
lo files list <projectId>
Create a cron job to execute the script¶
sudo mv lo-upload.sh /etc/cron.daily/ crontab -e
Add the following line (the script will run nightly at midnight) and then save:
0 0 * * * /usr/bin/timeout -s 2 172800 /etc/cron.daily/lo-upload.sh >> /home/ubuntu/logs/lo-upload.log 2>&1
Email success or failure¶
Instead of sending output to the upload.log file as shown above, it can be better from a monitoring perspective to receive nightly emails upon success or failure.
You will need to configure your server to be able to send email. One tool commonly used for this purpose is postfix. For postfix installation and configuration, search the web for setup instructions specific to the type of SMTP server you’ll be using (Office 365, gmail, and so on).
Make sure to test an example email before moving on from this step:
sudo apt-get install mailutils echo "Test email contents" | mail -s "Test email subject" "firstname.lastname@example.org,email@example.com"
Reconfigure cron job to email¶
Update your crontab -e entry to use mail:
0 0 * * * /usr/bin/timeout -s 2 172800 /etc/cron.daily/lo-upload.sh 2>&1 | mail -s "PHC <companyName> Files Upload" "firstname.lastname@example.org,email@example.com"