Two or more SatNOGS clients on a single RPi/computer (reprise?)

hughhalf · November 22, 2020, 11:16pm

Hi,

I realise this general topic has come up before but it seems compute resources, even in the humble RPI have also moved along, so please bear with me.

In short - would like to run two more satnogs-clients on a single host (RPI4 or, later, a server class boxen) each tied to a separate USB device.

My current setup is a RPi4 with a Kerberos SDR (4 RTL-SDRs in a single box) each of which I’ve set to a unique serial number.

The setup runs fine with a single satnogs-client tied to by serial number to the relevant channel on the KSDR (e.g. with “driver=rtlsdr,serial=2702”) 2702 being the serial number I set manually for that channel in EEPROM.

I have an unrelated SDR app (rtl_433) running on one of the other channels without any apparent issues with USB bandwidth. Empirically I’ve been able to run all four channels in parallel without apparent USB issues - but it’s a Pi4 which has a better USB controller I think.

To SatNOGS then - to run multiple stations and stay compatible with the upstream SatNOGS scheduling infrastructure, my understanding is that I should create a unique station ID/API Key for each station.

I’m less sure about how to run multiple satnogs-clients on the same 'pi - is there a preferred method ?

It seems that using the satnogs-setup ansible magic to configure one, then run any second and subsequent instances manually would be an approach. Best I can tell some sort of magic with environment variables as proposed in this earlier post might be the go ? Or might it be feasible to run second and subsequent instances in a container of some form ?

Before I go too far down the rabbit hole would welcome feedback on this approach - perhaps the SatNOGS devs have plans for this support “officially” in the future in which case I’d be happy to help test. Or perhaps the advice is “buy a 2nd Pi Hugh and stop being silly”, though I’d allow this seems a shame given the growing compute power available (and coming soon)

Cheers,
Hugh

Acinonyx · November 23, 2020, 6:17pm

SatNOGS Client Ansible currently supports only a single SatNOGS Client to be running on the RPi. You can however manually duplicate the systemd script, tweak the configuration and set different directories to write temporary files for the second instance.

fredy · November 24, 2020, 3:34am

Just a quick comment on this. You will need indeed two separate Station IDs if you are planning to run them at the same time. However you will need only one API Key if both station are under the same user, as API Keys are generated per user not per station.

hughhalf · November 24, 2020, 6:08am

Thank you both - @Acinonyx for the pointers on where to start with systemd scripts and @fredy for the StationID vs API key distinction. Hopefully I’ll get a chance to tinker with this over next day or two.

hughhalf · December 13, 2020, 4:26am

After the helpful pointers from @Acinonyx and @fredy I set about attempting to run two simultaneous satnogs-clients on a single RPi4, a few weeks back with partial success.

My impression so far is there’s no obvious compute or I/O bandwidth issues when receiving two SVs simultaneously, but there is some sort of interaction occurring. I’ll elaborate on this further below, but first the basic approach I took in case it’s of interest to others.

I created a duplicate service, called (imaginatively) satnogs-client-2. I did this by making a copy of /etc/systemd/system/satnogs-client.service called /etc/systemd/system/satnogs-client-2.service This file differs only from the original in having the EnvironmentFile entry pointing to a 2nd environment file which contains the different settings for the 2nd instance;

[Unit]
Description=SatNOGS client two
[Service]
EnvironmentFile=-/home/pi/satnogs-client-2/etc-default-satnogs-client
ExecStart=/var/lib/satnogs/bin/satnogs-client
Restart=on-failure
User=satnogs
Group=satnogs
[Install]
WantedBy=multi-user.target

Then in the appropriate directory the environment file for the 2nd instance;

SATNOGS_API_TOKEN=“XXXXXX”
SATNOGS_STATION_ID=“1891”
SATNOGS_STATION_LAT=“-38.x”
SATNOGS_STATION_LON=“142.x”
SATNOGS_STATION_ELEV=“51”
SATNOGS_SOAPY_RX_DEVICE=“driver=rtlsdr,serial=00000001”
SATNOGS_RX_SAMP_RATE=“2.048e6”
SATNOGS_RF_GAIN=“32”
SATNOGS_ANTENNA=“RX”
SATNOGS_APP_PATH=“/tmp/.satnogs2”
SATNOGS_OUTPUT_PATH=“/tmp/.satnogs2/data”
SATNOGS_INCOMPLETE_OUTPUT_PATH=“/tmp/.satnogs2/data/incomplete”
SATNOGS_COMPLETE_OUTPUT_PATH=“/tmp/.satnogs2/data/complete”
SATNOGS_LOG_LEVEL=“INFO”

Of note in the above;

You need to use your existing API_TOKEN
Create a new station ID through the Satnogs Dashboard
Create new temporary directories for the 2nd instance
Adjust the SOAPY_RX_DEVICE settings to suit your 2nd SDR dongle - I used rtl_eeprom to re-write the serial number of the different kerberos_SDR channels

This should be enough that with sudo systemctl daemon-reload and sudo systemctl start satnogs-client-2.service you’ll be up and running.

Variations on journalctl -f -u satnogs-client-2.service will let you see what is going on.

Two important caveats:
Firstly I’m new to satnogs and systemd wrangling, so this might not be the best way to do this, constructive feedback of course welcome. It might break your system!

Secondly there is an interaction between the two instances if they are running simultaneously which, best I can tell is not I/O or compute bandwidth related and only seems to occur if different RX frequencies in use.

I’ve looked at load averages, USB bandwidth utilisation etc. and nothing suggests issues there so am beggining to suspect something not quite right in the way I’m running the second client or, possibly, a lockfile contention in satnogs/GNUradio, but this latter less likely.

To elaborate;

Pull up these two observations side by side https://network.satnogs.org/observations/3289133/ and https://network.satnogs.org/observations/3289134/

The VZLUSAT-1 pass (3289133, UHF) overlaps with the first 6 minutes or so of the ERS-15 pass (3289134, VHF) During the time there is overlap there are horizontal bands in the waterfall plots which disappear once the overlap period ends (the VHF station isn’t fully commissioned so ignore the actual received signal or absence thereof)

By contrast comparing simultaneous UHF RX in observations https://network.satnogs.org/observations/3285917/ and https://network.satnogs.org/observations/3285918/ there is no apparent interaction (the VHF station doing observation …918 has no gain at UHF at all so won’t see a signal, but I have verified in other passes that it is “live”)

I got a chance this weekend to try have a decent go at debugging what is going on. Perhaps due to the way I’ve got things set up, I have observed that the two stations both end up sharing/writing to a single “FFT Wisdom” file and associated lock in /var/lib/satnogs/.gr_fft_wisdom and /var/lib/satnogs/.gr_fft_wisdom.lock and when running on different frequencies they write different data there.

I haven’t been able to work out a way to tell satnogs or GNU Radio to use a different location for these files to force them to be unique short of (presumably) rebuilding the library to add in a PID or some other unique value. It appears to be set fairly deeply in GR in wisdom_filename() in gnuradio/gr-fft/lib/fft.cc at master · gnuradio/gnuradio · GitHub As a result my conclusions that this is the underlying issue is a hypothesis only at this stage.

I have been able to reproduce this same “interaction” behaviour running the two underlying satnogs decoders by hand from a shell on the system - e.g.

sudo -u satnogs /usr/bin/python3 /usr/bin/satnogs_fm.py --soapy-rx-device=driver=rtlsdr,serial=00000001 --samp-rate-rx=2.048e6 --rx-freq=136801000 --file-path=receiving_satnogs_3286515.out --waterfall-file-path=receiving_waterfall_3286515.dat --decoded-data-file-path=data_3286515 --rigctl-port=4532 --gain-mode=Overall --gain=32 --antenna=RX --enable-iq-dump=0
sudo -u satnogs /usr/bin/python3 /usr/bin/satnogs_bpsk.py --soapy-rx-device=driver=rtlsdr,serial=2702 --samp-rate-rx=2.048e6 --rx-freq=436400000 --file-path=receiving_satnogs_3286516.out --waterfall-file-path=receiving_waterfall_3286516.dat --decoded-data-file-path=data_3286516 --rigctl-port=4532 --gain-mode=Overall --gain=32 --antenna=RX --enable-iq-dump=0 --baudrate=9600 --framing=ax25

This post already way longer than I planned so will leave it there for now as other duties call this arvo.

Any insights would be most welcome - I do appreciate that I am trying to do something that’s not officially supported but figure if I can contribute to the project by making simultaneous clients possible in the future that’d be a useful contribution as we start to enjoy faster compute hardware.

hughhalf · January 4, 2021, 9:54am

This weekend I was able to spend a bit more time digging into the apparent interaction between two simultaneously running instances of the satnogs-client. For further background please see the reply above.

Initially I attempted to build my own version of the gnu-radio library libgnuradio-fft.so.3.8.2.0 which contains the fft function I suspect is interacting between instances. The change I wanted to test was simple enough - to place the gr_fftw_wisdom and gr_fftw_wisdom.lock files in the temporary directory rather than the app path - modifying line 61 of https://github.com/gnuradio/gnuradio/blob/master/gr-fft/lib/fft.cc from;

path = fs::path(gr::appdata_path()) / ".gr_fftw_wisdom";

to

path = fs::path(gr::tmp_path()) / ".gr_fftw_wisdom";

but after an inordinate amount of trying different things including building gr- from source entire I just couldn’t get anything usable built. I think my .deb packaging-fu just not good enough

I went for plan B and tried running the 2nd instance unmodified in a chroot environment (recall from above the issue appears to be both instances modifying the same .gr_fftw_wisdom/.gr_fftw_wisdom.lock files - this did the trick, each instance now had its own copy of the file and the interactions I was seeing (typified by radical bands in the waterfall plots and corresponding audio) seem to have disappeared.

Where to from here is less clear to me. I don’t know the software stack well enough to be sure that the change I propose above is the right approach - it may for instance mess up other use cases of GNU-Radio. Perhaps a safer option is to have a patch that allows an implicit path to be set for the fft_wisdom files if desired but otherwise the default of app_data remains.

I’m actively doing further testing of the station with trying to run as many simultaneous receives as possible and am grateful to @ansi for his help with this as he has a script designed for just this purpose.

I’ll wait until I have a few more days of conclusive lack-of-interactions and then report back including writing a bit more detail on the (hackish) way I did the chroot.

If nothing else it does appear to suggest that a single RPi-4 can handle at least two independent simultaneous SDR receivers at once.

Acinonyx · January 4, 2021, 10:05am

Are you using the same rigctl server for both clients?

hughhalf · January 4, 2021, 10:09am

Err, have to confess ignorance here sorry @Acinonyx - I’m not intentionally using rigctl as both channels are RTL-SDR V3s - not external radios - or does rigctl drive the frequency settings for the SDRs too ?

Quick look at the logs and only the 2nd instance seems to log any rigctl related messages and I suspect just a side effect of having more logging on that instance.

Happy to dig further though if you can suggest where I should look ?

Edited to add: Happened to have two passes being recorded when I looked - here’s the command line for them both;

satnogs 4762 63.9 5.5 369824 105776 ? Sl 21:08 2:34 /usr/bin/python3 /usr/bin/satnogs_cw_decoder.py --soapy-rx-device=driver=rtlsdr,serial=2703 --samp-rate-rx=2.048e6 --rx-freq=145843800 --file-path=/tmp/.satnogs2/data/receiving_satnogs_3413032_2021-01-04T10-08-09.out --waterfall-file-path=/tmp/.satnogs2/data/receiving_waterfall_3413032_2021-01-04T10-08-09.dat --decoded-data-file-path=/tmp/.satnogs2/data/data_3413032 --rigctl-port=4532 --gain-mode=Overall --gain=9 --antenna=RX --enable-iq-dump=0

satnogs 5431 88.5 5.5 444584 106548 ? Sl 21:09 2:26 /usr/bin/python3 /usr/bin/satnogs_fsk.py --soapy-rx-device=driver=rtlsdr,serial=00000001 --samp-rate-rx=2.048e6 --rx-freq=437421500 --file-path=/tmp/.satnogs/data/receiving_satnogs_3412953_2021-01-04T10-09-26.out --waterfall-file-path=/tmp/.satnogs/data/receiving_waterfall_3412953_2021-01-04T10-09-26.dat --decoded-data-file-path=/tmp/.satnogs/data/data_3412953 --rigctl-port=4532 --gain-mode=Overall --gain=16 --antenna=RX --enable-iq-dump=0 --baudrate=9600 --framing=ax25

and only a single instance of rigctld running;

hamlib-+ 7907 0.1 0.4 51428 8060 ? Ssl Jan01 6:47 /usr/bin/rigctld -T 127.0.0.1 -m 1

Acinonyx · January 4, 2021, 10:29am

Yeah, the rigctl server tunes the SDRs as well so you need two instances of them and the respective configuration on SatNOGS client and radio.

Copy /etc/systemd/system/rigctld.service to /etc/systemd/system/rigctld2.service
Edit /etc/systemd/system/rigctld2.service to update EnvironmentFile=-/etc/default/hamlib-utils2
Copy /etc/default/hamlib-utils to /etc/default/hamlib-utils2
Edit /etc/default/hamlib-utils2 to update RIG_OPTS="-t 4542 -T 127.0.0.1 -m 1"
In /home/pi/satnogs-client-2/etc-default-satnogs-client, add SATNOGS_RIG_PORT="4542"
systemctl start rigctld2 and systemctl restart satnogs-client

hughhalf · January 4, 2021, 10:41am

Ah, ok, that is handy to know and makes sense. Might also explain why I would sometimes see the interactions, othertimes not, might have just been getting lucky there.

Wonder if the .gr_fft_wisdom thing was a red herring ?

In any case will give will give the 2nd rigtctl instance a try as soon as possible, thanks very much!

hughhalf · January 4, 2021, 12:09pm

Started a 2nd rigctl instance and scheduled some passes, first few look good, will see how rest go overnight. Thanks for the suggestion @Acinonyx.

Once have this debugged will write up in wiki if that’s helpful (noting it’s not an officially supported config so ymmv)

cgbsat · January 4, 2021, 6:55pm

SatNOGS stations #39 and #40 run from the same x86 machine, each using a separate RTL SDR. As the installation is manual and does not use ansible, this approach may not be directly applicable to a Raspberry Pi.

However, to separate the operation of the two stations, the following environment variables are important:

paths: SATNOGS_APP_PATH, SATNOGS_OUTPUT_PATH, SATNOGS_COMPLETE_OUTPUT_PATH and SATNOGS_INCOMPLETE_OUTPUT_PATH need to be different for each station
Each RTL SDR needs to be given a different serial number (use rtl_eeprom to set this), and is specified through SATNOGS_DEV_ARGS
The rigctld and rotctld ports need to be different, set SATNOGS_ROT_PORT and SATNOGS_RIG_PORT to different values and make sure the right rigctld and rotctld are started before starting satnogs-client.
Obviously, each station eeds a different SATNOGS_STATION_ID.

hughhalf · January 4, 2021, 11:26pm

Thanks for that @cgbsat, looks like I had things pretty much right other than the 2nd rigctld and rotctld instances. When I write it up I’ll make sure to add that extra info.

As at this morning it looks like the setup is working well other than the RF side of the UHF station which seems to have a damaged preamp or something going on, but only when temperatures drop below about 15C. Some soldering in my future

hughhalf · February 14, 2021, 4:17am

I’ve done a first pass at a Wiki page to cover this, hopefully of use to others. Have endeavoured to attribute the assistance received correctly and to make it clear that it’s not an officially supported configuration

hb9fxx · February 14, 2021, 2:26pm

I’ve also successfully ran 2 clients in 2 separate dockers using ansible deployement to footloose dockers as described in https://gitlab.com/librespacefoundation/satnogs/satnogs-client-ansible. I’ve been running one for many month for some testing. Once running they can be used as standalone clients and can be updated/upgraded individually.

But it rather consider this for testing/development on PC than a ‘production’ client.