Client crashes after observation of mode BPSK 2320000 (Station 858)

SatNOGS Software
1

Hi,
some days ago I did an update of my client through satnogs-setup. After this update I have problem that the client seams to crasch after observation of satellite 43857 - Da Vinci mode
BPSK 2320000
see SatNOGS Network - Observation 6357946

Only solution is to hard reset (power off/on) the Raspberry PI. See log below “Satellite not supported”:

Any solutions for this? As now, my station can not be used.
regards
Lars SM0TGU

aug 21 04:07:12 SM0TGUSATNOGS satnogs-client[369]: gr-satellites: Observation: 6357947, Norad: 43857, Name: 0_DAVINCI, Script: satnogs_bpsk.py
aug 21 04:07:12 SM0TGUSATNOGS satnogs-client[369]: gr-satellites: Satellite not supported
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: [INFO] Using format CF32.
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: gr::log 2022-08-21 04:07:15,196 :INFO: satnogs.doppler_compensation: Output sampling rate should be less or equal the device sampling rate
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: Traceback (most recent call last):
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: File “/usr/bin/satnogs_bpsk.py”, line 583, in
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: main()
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: File “/usr/bin/satnogs_bpsk.py”, line 566, in main
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: tb = top_block_cls(antenna=options.antenna, baudrate=options.baudrate, bb_freq=options.bb_freq, bw=options.bw, dc_removal=options.dc_removal, decoded_data_file_path
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: File “/usr/bin/satnogs_bpsk.py”, line 144, in init
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: self.satnogs_doppler_compensation_0 = satnogs.doppler_compensation(samp_rate_rx, rx_freq, lo_offset, baudrate*decimation, 1, 0)
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: File “/usr/lib/python3/dist-packages/satnogs/doppler_compensation.py”, line 91, in init
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: raise AttributeError
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: AttributeError
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: gr-satellites: Observation: 6357947, Norad: 43857, Name: 0_DAVINCI, Script: satnogs_bpsk.py
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: apscheduler.executors.default - ERROR - Job “spawn_observer (trigger: date[2022-08-21 02:07:12 UTC], next run at: 2022-08-21 02:07:12 UTC)” raised an exception
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: Traceback (most recent call last):
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: File “/var/lib/satnogs/lib/python3.7/site-packages/apscheduler/executors/base.py”, line 125, in run_job
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: retval = job.func(*job.args, **job.kwargs)
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py”, line 65, in spawn_observer
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: observer.observe()
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 209, in observe
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: waterfall = Waterfall(self.observation_waterfall_file)
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/waterfall.py”, line 105, in init
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: self.data = _get_waterfall(datafile_path)
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/waterfall.py”, line 80, in _get_waterfall
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: waterfall = _read_waterfall(datafile_path)
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/waterfall.py”, line 36, in _read_waterfall
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: ‘timestamp’: np.fromfile(datafile, dtype=‘|S32’, count=1)[0],
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: IndexError: index 0 is out of bounds for axis 0 with size 0
aug 21 04:49:01 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 05:13:11 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 05:26:18 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 06:00:21 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 06:23:17 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 07:17:09 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 07:25:34 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 07:37:52 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 08:12:27 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 08:35:28 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 08:50:31 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 09:26:23 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 09:40:22 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 10:15:15 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.

Support information:
{
“versions”: {
“satnogs-client”: “1.6”,
“satnogs-client-ansible”: “202205101826”,
“satnogs-flowgraphs”: “1.4-1”,
“gr-satnogs”: “2.3.1.1-1”,
“gr-soapy”: “2.1.3.1-1”,
“gnuradio”: “3.8.2.0-14satnogs2”,
“satnogs-config”: “0.12”
},
“state”: {
“is-applied”: true,
“pending-tags”: null
},
“system”: {
“date”: “2022-08-21T08:29:06.065949+00:00”,
“distribution”: {
“DESCRIPTION”: “Raspbian GNU/Linux 10 (buster)”,
“RELEASE”: “10”,
“CODENAME”: “buster”,
“ID”: “Raspbian”
},
“pending-updates”: true,
“platform”: {
“system”: “Linux”,
“node”: “SM0TGUSATNOGS”,
“release”: “5.10.103-v7+”,
“version”: “#1529 SMP Tue Mar 8 12:21:37 GMT 2022”,
“machine”: “armv7l”,
“processor”: “”
},
“memory”: {
“total”: 968052736,
“available”: 755314688,
“percent”: 22.0,
“used”: 151207936,
“free”: 432898048,
“active”: 147927040,
“inactive”: 335581184,
“buffers”: 29384704,
“cached”: 354562048,
“shared”: 6496256,
“slab”: 32251904
},
“disk”: {
“total”: 31090814976,
“used”: 3764711424,
“free”: 26015645696,
“percent”: 12.6
}
},
“configuration”: {
“satnogs_antenna”: “RX”,
“satnogs_api_token”: “[redacted]”,
“satnogs_gain_mode”: “Settings Field”,
“satnogs_other_settings”: “LNA=10,MIX=10,VGA=10”,
“satnogs_post_observation_script”: “/usr/local/bin/satnogs-post {{ID}} {{FREQ}} {{TLE}} {{TIMESTAMP}} {{BAUD}} {{SCRIPT_NAME}}”,
“satnogs_pre_observation_script”: “/usr/local/bin/satnogs-pre {{ID}} {{FREQ}} {{TLE}} {{TIMESTAMP}} {{BAUD}} {{SCRIPT_NAME}}”,
“satnogs_rf_gain”: “36.4”,
“satnogs_rx_samp_rate”: “3e6”,
“satnogs_soapy_rx_device”: “driver=airspy”,
“satnogs_station_elev”: “5”,
“satnogs_station_id”: “858”,
“satnogs_station_lat”: “59.453”,
“satnogs_station_lon”: “17.890”,
“satnogs_waterfall_min_value”: “-90”,
“udp_dump_host”: “127.0.0.1”
}
}

2

These instructions might get your station working without reboot: Troubleshooting - SatNOGS Wiki
Not a fix, but could make troubleshooting a bit easier.

2 Likes
3

I’m not sure what’s up with it, but it appears your ground station does VHF and UHF up to 500 MHz, but the observation was scheduled at 1616 MHz (1.616 GHz).

1 Like
4

I changed the frq after I noticed that the satellite 43857 crasched my station, so no new observation with that satellite can be done.
Tomorrow I will start up the station again and schedule some known satellites at 435 MHz and 145 MHz and see if everything is stable.

2 Likes
5

Can you check which version of rtl-sdr is installed by running

dpkg -s rtl-sdr

I assume it should be 0.6-1+rpt1satnogs3. There has been a bug triggering the same error message, fixed by a patched version of rtl-sdr and deployed to stable stations many months ago (see satnogs-client#415 (comment)). Checking the rtl-sdr version will allow us to see if the patched version is installed (maybe the satnogs-update accidentally installed an unpatched version).

2 Likes
6

Yes it is version 0.6-1+rpt1satnogs3 installed.

7

I have tested today with satellites with CW and BPSK without any problem. It seams that the satellite “Da vinci” and mode “BPSK 2320000” fails and in my case locks up all upcoming observations. Locking at other stations they also have failed observations of “BPSK 2320000”.

This mode should be removed from the dB.

Edit! I see that it has been set as “inactive” 2022-08-22 07:25 by pierros

Thanks all for help!
/Lars

3 Likes
8

Thank you for checking the rtl-sdr version, it confirmed that the patched version is installed. My suspicion that this issue is related to the previous rtl-sdr issue turned out to be wrong. Indeed, satellites with BPSK at a high baudrate (lately a lot of them were added to SatNOGS and many observations were scheduled) were the cause. Apparently they can cause SatNOGS stations to get into a broken state which can only be fixed by restarting the satnogs-client service. This happened to multiple stations and there will be or already has been communication with affected stations.

The underlying issue with the BPSK flowgraph is tracked in satnogs-flowgraphs#32. The issue of satnogs-client not recovering itself is tracked in satnogs-client#434 (comment).

As you have seen, the transmitters which were causing issues have been set to “inactive” in satnogs-db for now, and it’s being worked on a fix for SatNOGS stations so that the transmitters can be set to “active” again.

4 Likes
9

@kerel Thank you for detailed information about the problem. Also thank you for continue working with a fix.

Best regards
Lars