Nightly file uploads¶

The LifeOmic platform allows its users to upload files via the LifeOmic CLI. There are multiple ways to authenticate with the CLI, but for this document we’ll be using API Keys. Once an API Key is generated, the CLI can be used in conjunction with a cron job to automatically upload files each night.

API Key Creation¶

See the API Key Guide

Download and verify CLI¶

wget "https://github.com/lifeomic/cli/releases/download/v6.11.0/lo-linux-x64.zip"
sudo apt-get install unzip
unzip lo-linux-x64.zip
rm lo-linux-x64.zip
sudo mv lo /usr/local/bin
lo --help

Setup CLI with API Key¶

lo setup
? Pick a LifeOmic environment: prod-us
? Specify a default LifeOmic account to use: myaccount
? Use API key for authentication? Yes
? API Key:

After pasting the API Key, you should be ready to use the CLI for file uploading. Issue the following command as a test to verify authentication is all setup:

lo projects list

Prepare local folders and PHC project¶

mkdir uploads
mkdir logs
lo projects create "Files Test"

Replace "Files Test" with whatever the project should be called.

Prepare upload script¶

Copy project ID

lo projects list
Example response:
items:
  -
    id:   f3a220a0-b22b-41c3-9f77-2ce7cf37c93b
    name: Files Test

Copy the ID of the project you created.

Create script¶

vi lo-upload.sh

#!/bin/bash

set -u
set -e
set -x

if pgrep --full "lo files upload" ; then
        echo "Already running"
        exit 0
fi

trap "echo Process killed" SIGINT SIGTERM SIGABRT

echo `date '+%Y-%m-%d %H:%M:%S'`
cd /home/ubuntu/uploads
/usr/local/bin/lo files upload ./ f3a220a0-b22b-41c3-9f77-2ce7cf37c93b --recursive --overwrite --delete-after-upload

NOTE: In this example we’re using /home/ubuntu/uploads. You may need to change that line of the script to match the directory you’re using.

NOTE: Make sure to replace the UUID (which starts with f3a220a0-) above with your project ID.

Add execution permissions and verify a file upload¶

chmod +x lo-upload.sh
echo "test" > uploads/test.txt
./lo-upload.sh

Verify you see output similar to this:

Upload complete: test.txt, ID: `d4d9d8fc-b96b-49ad-a43c-c24f54b29941`

Also, issuing the following command should verify the uploaded file:

lo files list <projectId>

Create a cron job to execute the script¶

sudo mv lo-upload.sh /etc/cron.daily/
crontab -e

Add the following line (the script will run nightly at midnight) and then save:

0 0 * * * /usr/bin/timeout -s 2 172800 /etc/cron.daily/lo-upload.sh >> /home/ubuntu/logs/lo-upload.log 2>&1

Email success or failure¶

Instead of sending output to the upload.log file as shown above, it can be better from a monitoring perspective to receive nightly emails upon success or failure.

Configure SMTP¶

You will need to configure your server to be able to send email. One tool commonly used for this purpose is postfix. For postfix installation and configuration, search the web for setup instructions specific to the type of SMTP server you’ll be using (Office 365, Gmail, and others).

Make sure to test an example email before moving on from this step:

sudo apt-get install mailutils
echo "Test email contents" | mail -s "Test email subject" "email1@domain.com,email2@domain.com"

Reconfigure cron job to email¶

Update your crontab -e entry to use mail:

0 0 * * * /usr/bin/timeout -s 2 172800 /etc/cron.daily/lo-upload.sh 2>&1 | mail -s "PHC <companyName> Files Upload" "email1@domain.com,email2@domain.com"