Uploading compressed IQ files to Dropbox (or any other cloud storage provider)

SatNOGS Software
1

Thanks to the work by other community members in Uploading IQ files to S3 I was able to create a script that uploads IQ files to Dropbox using rclone.

Rclone supports a lot of cloud storage providers (Dropbox, AWS S3, Google Drive, Azure blob storage, Microsoft OneDrive, NextCloud, etc) as well as generic upload methods such as SFTP, FTP, HTTP). This is great, since the same script can be used for all supported targets. See https://rclone.org/ for a full list.

rclone install

rclone can be installed by running apt-get install rclone if you are using the SATNOGS Raspberry Pi image (or any other Ubuntu-baed system).

rclone setup

rclone needs to be configured. Note that the configuration needs to be run as the satnogs user. This is what I did:
edit /etc/passed; change

satnogs:x:999:995::/var/lib/satnogs:/bin/false

to

satnogs:x:999:995::/var/lib/satnogs:/bin/bash

Run sudo su - satnogs to get a shell as the satnogs user. Click on the icon that looks like a book next to your storage provider on https://rclone.org/ and follow the instructions. Instructions for Dropbox: Dropbox
After configuring you can change /etc/passwd back to /bin/false for the satnogs user.

post observation script

This is the script I use:

#!/bin/bash
RC_REMOTE=mfdropbox # This needs to be the same name you specified when configuring rclone
if [ -n "$IQ_DUMP_FILENAME" ] && [ -f "$IQ_DUMP_FILENAME" ]
then
   IQ_DIR=$(dirname "${IQ_DUMP_FILENAME}")
   IQ_NAME="$1".wav.xz
   IQ_FILE="$IQ_DIR"/"$IQ_NAME"
   xz -c -7 -T2 "$IQ_DUMP_FILENAME" > "$IQ_FILE"
   rclone move "$IQ_FILE" $RC_REMOTE:
fi

I placed it in /usr/local/bin/satnogs-post and made it executable by the satnogs user:

chgrp satnogs /usr/local/bin/satnogs-post
chmod g+x /usr/local/bin/satnogs-post

satnogs-setup

To enable IQ file saving and activate the post observation script, run sudo satnogs-setup and set the following settings:

Advanced -> SATNOGS_POST_OBSERVATION_SCRIPT /usr/local/bin/satnogs-post {{ID}}
Advanced -> ENABLE_IQ_DUMP YES
Advanced -> IQ_DUMP_FILENAME /tmp/.satnogs/iq.raw

Let satnogs-setup apply the configuration.

Contents of the configuration file (/etc/default/satnogs-client):

SATNOGS_POST_OBSERVATION_SCRIPT="/usr/local/bin/satnogs-post {{ID}}"
ENABLE_IQ_DUMP="True"
IQ_DUMP_FILENAME="/tmp/.satnogs/iq.raw"

Compressing the IQ files save about 50% based on my tests. I chose xz because it often performs better than gz and bzip2, but xz uses more cpu and more ram, so using a different compressor might be useful.

After each observation, the iq file will be uploaded to my dropbox account:
image

Things I’d like to add at some point in the future:

{{ID}} {{FREQ}} {{TLE}} {{TIMESTAMP}} {{BAUD}} {{SCRIPT_NAME}}"

example data:

1582787 437250000 {"tle2": "2 39161 97.9744 109.0828 0009620 355.2085 4.9027 14.72273109360518", "tle1": "1 39161U 13021C 20023.16198395 .00000100 00000-0 23673-4 0 9992", "tle0": "ESTCUBE 1"} 2020-01-23T10-54-56 None satnogs_cw_decoder.py
1574047 437424000 {"tle2": "2 41460 98.0961 127.5580 0155052 334.7290 24.6413 15.08103934205599", "tle1": "1 41460U 16025E 20023.19743888 .00001022 00000-0 54631-4 0 9992", "tle0": "AAUSAT 4"} 2020-01-23T11-32-39 2400.0 satnogs_msk_ax25.py
6 Likes
2

I have now added the satellite name to the filename:

image
image831×507 14.9 KB

Updated script:

#!/bin/bash
RC_REMOTE=mfdropbox
if [ -n "$IQ_DUMP_FILENAME" ] && [ -f "$IQ_DUMP_FILENAME" ]
then
   SATNAME=$(echo "$3" | jq .tle0 | sed -e 's/ /_/g' | sed -e 's/[^A-Za-z0-9._-]//g')
   IQ_DIR=$(dirname "${IQ_DUMP_FILENAME}")
   IQ_NAME="$1"_"$SATNAME".wav.xz
   IQ_FILE="$IQ_DIR"/"$IQ_NAME"
   xz -c -7 -T2 "$IQ_DUMP_FILENAME" > "$IQ_FILE"
   rclone move "$IQ_FILE" $RC_REMOTE:
fi

Updated SATNOGS_POST_OBSERVATION_SCRIPT setting:

/usr/local/bin/satnogs-post {{ID}} {{FREQ}} {{TLE}} {{TIMESTAMP}} {{BAUD}} {{SCRIPT_NAME}}
7 Likes
3

Nice work! Next iteration of gr-satnogs will use this https://gitlab.com/librespacefoundation/sdrmakerspace/iqzip so IQ will be directly stored compressed and will have enough metadata to fully describe the parameters of the capture (bandwidth, dynamic range, center frequency, etc) directly from the radio frontend.

4 Likes
4

Thanks! Compressing the IQ files by default will be great.

1 Like
5

The IQ file upload has been working well, and the size seems manageable so I created a public link: https://www.dropbox.com/sh/1ed4wopcphqjmbh/AAB-O2qhu5SbV6rwdgC7Xsxha?dl=0&lst=
My station will upload IQ files of all observations to this folder, so feel free to schedule something and download the IQ file.

2 Likes
6

I noticed that when observations are too close, xz doesn’t always finish in time, which causes a failed observation (waterfall and other files aren’t uploaded).

It would probably be a good idea to run the compression in the background.

1 Like
7

I should probably create some sort of queue for compressing and uploading (uploading could take a long time if bandwidth is low). This is a small modification to at least save the IQ file before it gets overwritten by the next observation:

#!/bin/bash
RC_REMOTE=dropbox
if [ -n "$IQ_DUMP_FILENAME" ] && [ -f "$IQ_DUMP_FILENAME" ]
then
   SATNAME=$(echo "$3" | jq .tle0 | sed -e 's/ /_/g' | sed -e 's/[^A-Za-z0-9._-]//g')
   IQ_DIR=$(dirname "${IQ_DUMP_FILENAME}")
   IQ_NAME="$1"_"$SATNAME".wav.xz
   IQ_FILE="$IQ_DIR"/"$IQ_NAME"
   TEMPFILE=$(mktemp -p /tmp/.satnogs/)
   mv "$IQ_DUMP_FILENAME" "$TEMPFILE"
   nice -n19 xz -c -6 -T2 "$TEMPFILE" > "$IQ_FILE" && rm "$TEMPFILE"
   rclone move "$IQ_FILE" $RC_REMOTE:
fi
2 Likes
9

Does this line above have an error? I can not get the regexp to work in a simulator

10

It works for me.

You can try it on the command line.

echo -n 'hello world' | sed -e 's/ /_/g'

should yield the string hello_world

11

your script requires jq which was not installed by default on my satnogs.

If I wanted to only save raw files for certain satellites, could I parse the TLE file for the satellite catalog number and then do some logic based on that?
My desire is to only save ones that have linear transponders since satnogs can’t decode them on it’s own yet.

For me, your above script seems to be saving the file as the passNumber.wav.xz and I thought it was supposed to be parsing something from the TLE.

12

The first script only uses the observation id, yes. The later script adds the satellite name. Example:
2059378_KKS-1_KISEKI.wav.xz

Yes, doing some logic based on the variables (not necessarily the TLE; BAUD and SCRIPT_NAME might be more useful) is what I was getting at in Raw IQ files - options for converting? Sorry for being unclear.

13

yes, you were very clear. I think what I was trying to document is that your final script on this page, makes use of jq . jq, was not part of the satnogs image. So, if you ran the script as it was without jq, it would produce file names that were just the pass ID.

thank you for all of you hard work and for being patient with me.

14

I see. Thanks for explaining.

1 Like
15

My station has uploaded 64.5GB IQ files so far. Seems to work well.

The Dropbox web interface isn’t very good when browsing the files though: no way to search or filter the file list. I’d like to find a way to create a better interface, so people can easily find the files they are interested in. On the other hand, when/if the Satnogs project implements support for IQ files there is no need for a separate interface so maybe I’ll just wait.

1 Like
16

My station has almost reached 200GB of IQ files.
image

3 Likes
17

Total compressed size of IQ files is now 836GB.
image

5 Likes
18

1.26TB
image

2 Likes
19

Sorry I have been silent. I had medical issues in my family which have now passed. Happy New Year to you and may you have health and happiness in 2023 !

1 Like
20

Thanks @kk4yel, the same to you and your family!