After the helpful pointers from @Acinonyx and @fredy I set about attempting to run two simultaneous satnogs-clients on a single RPi4, a few weeks back with partial success.
My impression so far is there’s no obvious compute or I/O bandwidth issues when receiving two SVs simultaneously, but there is some sort of interaction occurring. I’ll elaborate on this further below, but first the basic approach I took in case it’s of interest to others.
I created a duplicate service, called (imaginatively) satnogs-client-2. I did this by making a copy of /etc/systemd/system/satnogs-client.service called /etc/systemd/system/satnogs-client-2.service This file differs only from the original in having the EnvironmentFile entry pointing to a 2nd environment file which contains the different settings for the 2nd instance;
[Unit]
Description=SatNOGS client two
[Service]
EnvironmentFile=-/home/pi/satnogs-client-2/etc-default-satnogs-client
ExecStart=/var/lib/satnogs/bin/satnogs-client
Restart=on-failure
User=satnogs
Group=satnogs
[Install]
WantedBy=multi-user.target
Then in the appropriate directory the environment file for the 2nd instance;
SATNOGS_API_TOKEN=“XXXXXX”
SATNOGS_STATION_ID=“1891”
SATNOGS_STATION_LAT=“-38.x”
SATNOGS_STATION_LON=“142.x”
SATNOGS_STATION_ELEV=“51”
SATNOGS_SOAPY_RX_DEVICE=“driver=rtlsdr,serial=00000001”
SATNOGS_RX_SAMP_RATE=“2.048e6”
SATNOGS_RF_GAIN=“32”
SATNOGS_ANTENNA=“RX”
SATNOGS_APP_PATH=“/tmp/.satnogs2”
SATNOGS_OUTPUT_PATH=“/tmp/.satnogs2/data”
SATNOGS_INCOMPLETE_OUTPUT_PATH=“/tmp/.satnogs2/data/incomplete”
SATNOGS_COMPLETE_OUTPUT_PATH=“/tmp/.satnogs2/data/complete”
SATNOGS_LOG_LEVEL=“INFO”
Of note in the above;
- You need to use your existing API_TOKEN
- Create a new station ID through the Satnogs Dashboard
- Create new temporary directories for the 2nd instance
- Adjust the SOAPY_RX_DEVICE settings to suit your 2nd SDR dongle - I used rtl_eeprom to re-write the serial number of the different kerberos_SDR channels
This should be enough that with sudo systemctl daemon-reload and sudo systemctl start satnogs-client-2.service you’ll be up and running.
Variations on journalctl -f -u satnogs-client-2.service will let you see what is going on.
Two important caveats:
Firstly I’m new to satnogs and systemd wrangling, so this might not be the best way to do this, constructive feedback of course welcome. It might break your system!
Secondly there is an interaction between the two instances if they are running simultaneously which, best I can tell is not I/O or compute bandwidth related and only seems to occur if different RX frequencies in use.
I’ve looked at load averages, USB bandwidth utilisation etc. and nothing suggests issues there so am beggining to suspect something not quite right in the way I’m running the second client or, possibly, a lockfile contention in satnogs/GNUradio, but this latter less likely.
To elaborate;
Pull up these two observations side by side https://network.satnogs.org/observations/3289133/ and https://network.satnogs.org/observations/3289134/
The VZLUSAT-1 pass (3289133, UHF) overlaps with the first 6 minutes or so of the ERS-15 pass (3289134, VHF) During the time there is overlap there are horizontal bands in the waterfall plots which disappear once the overlap period ends (the VHF station isn’t fully commissioned so ignore the actual received signal or absence thereof)
By contrast comparing simultaneous UHF RX in observations https://network.satnogs.org/observations/3285917/ and https://network.satnogs.org/observations/3285918/ there is no apparent interaction (the VHF station doing observation …918 has no gain at UHF at all so won’t see a signal, but I have verified in other passes that it is “live”)
I got a chance this weekend to try have a decent go at debugging what is going on. Perhaps due to the way I’ve got things set up, I have observed that the two stations both end up sharing/writing to a single “FFT Wisdom” file and associated lock in /var/lib/satnogs/.gr_fft_wisdom and /var/lib/satnogs/.gr_fft_wisdom.lock and when running on different frequencies they write different data there.
I haven’t been able to work out a way to tell satnogs or GNU Radio to use a different location for these files to force them to be unique short of (presumably) rebuilding the library to add in a PID or some other unique value. It appears to be set fairly deeply in GR in wisdom_filename() in gnuradio/gr-fft/lib/fft.cc at master · gnuradio/gnuradio · GitHub As a result my conclusions that this is the underlying issue is a hypothesis only at this stage.
I have been able to reproduce this same “interaction” behaviour running the two underlying satnogs decoders by hand from a shell on the system - e.g.
sudo -u satnogs /usr/bin/python3 /usr/bin/satnogs_fm.py --soapy-rx-device=driver=rtlsdr,serial=00000001 --samp-rate-rx=2.048e6 --rx-freq=136801000 --file-path=receiving_satnogs_3286515.out --waterfall-file-path=receiving_waterfall_3286515.dat --decoded-data-file-path=data_3286515 --rigctl-port=4532 --gain-mode=Overall --gain=32 --antenna=RX --enable-iq-dump=0
sudo -u satnogs /usr/bin/python3 /usr/bin/satnogs_bpsk.py --soapy-rx-device=driver=rtlsdr,serial=2702 --samp-rate-rx=2.048e6 --rx-freq=436400000 --file-path=receiving_satnogs_3286516.out --waterfall-file-path=receiving_waterfall_3286516.dat --decoded-data-file-path=data_3286516 --rigctl-port=4532 --gain-mode=Overall --gain=32 --antenna=RX --enable-iq-dump=0 --baudrate=9600 --framing=ax25
This post already way longer than I planned so will leave it there for now as other duties call this arvo.
Any insights would be most welcome - I do appreciate that I am trying to do something that’s not officially supported but figure if I can contribute to the project by making simultaneous clients possible in the future that’d be a useful contribution as we start to enjoy faster compute hardware.