Client crashes after observation of mode BPSK 2320000

SatNOGS Software
1

Hi,
some days ago I did an update of my client through satnogs-setup. After this update I have problem that the client seams to crasch after observation of satellite 43857 - Da Vinci mode
BPSK 2320000
see SatNOGS Network - Observation 6357946

Only solution is to hard reset (power off/on) the Raspberry PI. See log below “Satellite not supported”:

Any solutions for this? As now, my station can not be used.
regards
Lars SM0TGU

aug 21 04:07:12 SM0TGUSATNOGS satnogs-client[369]: gr-satellites: Observation: 6357947, Norad: 43857, Name: 0_DAVINCI, Script: satnogs_bpsk.py
aug 21 04:07:12 SM0TGUSATNOGS satnogs-client[369]: gr-satellites: Satellite not supported
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: [INFO] Using format CF32.
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: gr::log 2022-08-21 04:07:15,196 :INFO: satnogs.doppler_compensation: Output sampling rate should be less or equal the device sampling rate
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: Traceback (most recent call last):
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: File “/usr/bin/satnogs_bpsk.py”, line 583, in
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: main()
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: File “/usr/bin/satnogs_bpsk.py”, line 566, in main
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: tb = top_block_cls(antenna=options.antenna, baudrate=options.baudrate, bb_freq=options.bb_freq, bw=options.bw, dc_removal=options.dc_removal, decoded_data_file_path
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: File “/usr/bin/satnogs_bpsk.py”, line 144, in init
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: self.satnogs_doppler_compensation_0 = satnogs.doppler_compensation(samp_rate_rx, rx_freq, lo_offset, baudrate*decimation, 1, 0)
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: File “/usr/lib/python3/dist-packages/satnogs/doppler_compensation.py”, line 91, in init
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: raise AttributeError
aug 21 04:07:15 SM0TGUSATNOGS satnogs-client[369]: AttributeError
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: gr-satellites: Observation: 6357947, Norad: 43857, Name: 0_DAVINCI, Script: satnogs_bpsk.py
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: apscheduler.executors.default - ERROR - Job “spawn_observer (trigger: date[2022-08-21 02:07:12 UTC], next run at: 2022-08-21 02:07:12 UTC)” raised an exception
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: Traceback (most recent call last):
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: File “/var/lib/satnogs/lib/python3.7/site-packages/apscheduler/executors/base.py”, line 125, in run_job
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: retval = job.func(*job.args, **job.kwargs)
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py”, line 65, in spawn_observer
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: observer.observe()
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 209, in observe
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: waterfall = Waterfall(self.observation_waterfall_file)
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/waterfall.py”, line 105, in init
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: self.data = _get_waterfall(datafile_path)
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/waterfall.py”, line 80, in _get_waterfall
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: waterfall = _read_waterfall(datafile_path)
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/waterfall.py”, line 36, in _read_waterfall
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: ‘timestamp’: np.fromfile(datafile, dtype=‘|S32’, count=1)[0],
aug 21 04:07:18 SM0TGUSATNOGS satnogs-client[369]: IndexError: index 0 is out of bounds for axis 0 with size 0
aug 21 04:49:01 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 05:13:11 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 05:26:18 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 06:00:21 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 06:23:17 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 07:17:09 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 07:25:34 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 07:37:52 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 08:12:27 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 08:35:28 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 08:50:31 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 09:26:23 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 09:40:22 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
aug 21 10:15:15 SM0TGUSATNOGS satnogs-client[369]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.

Support information:
{
“versions”: {
“satnogs-client”: “1.6”,
“satnogs-client-ansible”: “202205101826”,
“satnogs-flowgraphs”: “1.4-1”,
“gr-satnogs”: “2.3.1.1-1”,
“gr-soapy”: “2.1.3.1-1”,
“gnuradio”: “3.8.2.0-14satnogs2”,
“satnogs-config”: “0.12”
},
“state”: {
“is-applied”: true,
“pending-tags”: null
},
“system”: {
“date”: “2022-08-21T08:29:06.065949+00:00”,
“distribution”: {
“DESCRIPTION”: “Raspbian GNU/Linux 10 (buster)”,
“RELEASE”: “10”,
“CODENAME”: “buster”,
“ID”: “Raspbian”
},
“pending-updates”: true,
“platform”: {
“system”: “Linux”,
“node”: “SM0TGUSATNOGS”,
“release”: “5.10.103-v7+”,
“version”: “#1529 SMP Tue Mar 8 12:21:37 GMT 2022”,
“machine”: “armv7l”,
“processor”: “”
},
“memory”: {
“total”: 968052736,
“available”: 755314688,
“percent”: 22.0,
“used”: 151207936,
“free”: 432898048,
“active”: 147927040,
“inactive”: 335581184,
“buffers”: 29384704,
“cached”: 354562048,
“shared”: 6496256,
“slab”: 32251904
},
“disk”: {
“total”: 31090814976,
“used”: 3764711424,
“free”: 26015645696,
“percent”: 12.6
}
},
“configuration”: {
“satnogs_antenna”: “RX”,
“satnogs_api_token”: “[redacted]”,
“satnogs_gain_mode”: “Settings Field”,
“satnogs_other_settings”: “LNA=10,MIX=10,VGA=10”,
“satnogs_post_observation_script”: “/usr/local/bin/satnogs-post {{ID}} {{FREQ}} {{TLE}} {{TIMESTAMP}} {{BAUD}} {{SCRIPT_NAME}}”,
“satnogs_pre_observation_script”: “/usr/local/bin/satnogs-pre {{ID}} {{FREQ}} {{TLE}} {{TIMESTAMP}} {{BAUD}} {{SCRIPT_NAME}}”,
“satnogs_rf_gain”: “36.4”,
“satnogs_rx_samp_rate”: “3e6”,
“satnogs_soapy_rx_device”: “driver=airspy”,
“satnogs_station_elev”: “5”,
“satnogs_station_id”: “858”,
“satnogs_station_lat”: “59.453”,
“satnogs_station_lon”: “17.890”,
“satnogs_waterfall_min_value”: “-90”,
“udp_dump_host”: “127.0.0.1”
}
}

2

These instructions might get your station working without reboot: Troubleshooting - SatNOGS Wiki
Not a fix, but could make troubleshooting a bit easier.

2 Likes
3

I’m not sure what’s up with it, but it appears your ground station does VHF and UHF up to 500 MHz, but the observation was scheduled at 1616 MHz (1.616 GHz).

1 Like
4

I changed the frq after I noticed that the satellite 43857 crasched my station, so no new observation with that satellite can be done.
Tomorrow I will start up the station again and schedule some known satellites at 435 MHz and 145 MHz and see if everything is stable.

2 Likes
5

Can you check which version of rtl-sdr is installed by running

dpkg -s rtl-sdr

I assume it should be 0.6-1+rpt1satnogs3. There has been a bug triggering the same error message, fixed by a patched version of rtl-sdr and deployed to stable stations many months ago (see satnogs-client#415 (comment)). Checking the rtl-sdr version will allow us to see if the patched version is installed (maybe the satnogs-update accidentally installed an unpatched version).

2 Likes
6

Yes it is version 0.6-1+rpt1satnogs3 installed.

7

I have tested today with satellites with CW and BPSK without any problem. It seams that the satellite “Da vinci” and mode “BPSK 2320000” fails and in my case locks up all upcoming observations. Locking at other stations they also have failed observations of “BPSK 2320000”.

This mode should be removed from the dB.

Edit! I see that it has been set as “inactive” 2022-08-22 07:25 by pierros

Thanks all for help!
/Lars

3 Likes
8

Thank you for checking the rtl-sdr version, it confirmed that the patched version is installed. My suspicion that this issue is related to the previous rtl-sdr issue turned out to be wrong. Indeed, satellites with BPSK at a high baudrate (lately a lot of them were added to SatNOGS and many observations were scheduled) were the cause. Apparently they can cause SatNOGS stations to get into a broken state which can only be fixed by restarting the satnogs-client service. This happened to multiple stations and there will be or already has been communication with affected stations.

The underlying issue with the BPSK flowgraph is tracked in satnogs-flowgraphs#32. The issue of satnogs-client not recovering itself is tracked in satnogs-client#434 (comment).

As you have seen, the transmitters which were causing issues have been set to “inactive” in satnogs-db for now, and it’s being worked on a fix for SatNOGS stations so that the transmitters can be set to “active” again.

4 Likes
9

@kerel Thank you for detailed information about the problem. Also thank you for continue working with a fix.

Best regards
Lars

10

Hello,
I see that this is still a problem, I’m trying to follow the tracked issues but not sure what the current status is for fix (or not fix).
Maybe someone can follow the links in Gitlab and explain? Thanks.
//Lars

11

It is fixed and still a problem (:
This is because it isn’t a released version, so the stable is 1.8.1 and fixes in master.
Previously one could just enable experimental mode and you would be on master in all the codebases, but unfortunately it seems broken now.

You could try using the master branch on -client separately, hop in to satnogs-setup and set
SATNOGS_CLIENT_URL = git+https://gitlab.com/librespacefoundation/satnogs/satnogs-client.git@master

12

Thanks Daniel.
After setting SATNOGS_CLIENT_URL, should I just do an “update” and “apply” after?

13

Yep, should do the trick. iirc the observations client version should identify as 1.8.1+githash after that

14

Seams not to work. I have reverted to the release version.

Install satnogs-client...
  Retrying... (1 of 4)
  Retrying... (2 of 4)
  Retrying... (3 of 4)
  localhost failed | msg: stdout: Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting git+https://gitlab.com/librespacefoundation/satnogs/satnogs-client.git@master
  Cloning https://gitlab.com/librespacefoundation/satnogs/satnogs-client.git (to revision master) to ./pip-req-build-8a1hl69k
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Building wheels for collected packages: satnogs-client
  Building wheel for satnogs-client (PEP 517): started
  Building wheel for satnogs-client (PEP 517): finished with status 'error'
Failed to build satnogs-client

:stderr:   Running command git clone -q https://gitlab.com/librespacefoundation/satnogs/satnogs-client.git /tmp/pip-req-build-8a1hl69k
  ERROR: Command errored out with exit status 1:
   command: /var/lib/satnogs/bin/python /tmp/tmpqnos51tm_in_process.py build_wheel /tmp/tmpm5aw0ct8
       cwd: /tmp/pip-req-build-8a1hl69k
  Complete output (82 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/satnogsclient
  copying satnogsclient/waterfall.py -> build/lib/satnogsclient
  copying satnogsclient/artifacts.py -> build/lib/satnogsclient
  copying satnogsclient/rotator.py -> build/lib/satnogsclient
  copying satnogsclient/_version.py -> build/lib/satnogsclient
  copying satnogsclient/settings.py -> build/lib/satnogsclient
  copying satnogsclient/config.py -> build/lib/satnogsclient
  copying satnogsclient/__main__.py -> build/lib/satnogsclient
  copying satnogsclient/rig.py -> build/lib/satnogsclient
  copying satnogsclient/__init__.py -> build/lib/satnogsclient
  creating build/lib/satnogsclient/scheduler
  copying satnogsclient/scheduler/tasks.py -> build/lib/satnogsclient/scheduler
  copying satnogsclient/scheduler/__init__.py -> build/lib/satnogsclient/scheduler
  creating build/lib/satnogsclient/observer
  copying satnogsclient/observer/obsever_test.py -> build/lib/satnogsclient/observer
  copying satnogsclient/observer/observer.py -> build/lib/satnogsclient/observer
  copying satnogsclient/observer/worker.py -> build/lib/satnogsclient/observer
  copying satnogsclient/observer/orbital.py -> build/lib/satnogsclient/observer
  copying satnogsclient/observer/__init__.py -> build/lib/satnogsclient/observer
  creating build/lib/satnogsclient/radio
  copying satnogsclient/radio/flowgraphs.py -> build/lib/satnogsclient/radio
  copying satnogsclient/radio/__init__.py -> build/lib/satnogsclient/radio
  creating build/lib/satnogsclient/locator
  copying satnogsclient/locator/locator.py -> build/lib/satnogsclient/locator
  copying satnogsclient/locator/__init__.py -> build/lib/satnogsclient/locator
  UPDATING build/lib/satnogsclient/_version.py
  set build/lib/satnogsclient/_version.py to '1.8.1+99.g4ca3571'
  running install
  running install_lib
  Traceback (most recent call last):
    File "/tmp/tmpqnos51tm_in_process.py", line 280, in <module>
      main()
    File "/tmp/tmpqnos51tm_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/tmpqnos51tm_in_process.py", line 204, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 216, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 202, in _build_with_temp_dir
      self.run_setup()
    File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 253, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 145, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 16, in <module>
      setup(**setup_args)
    File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 967, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 259, in run
      self.run_command('install')
    File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/usr/lib/python3/dist-packages/setuptools/command/install.py", line 61, in run
      return orig.install.run(self)
    File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/install.py", line 578, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 985, in run_command
      cmd_obj.ensure_finalized()
    File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
      self.finalize_options()
    File "/usr/lib/python3/dist-packages/setuptools/command/install_lib.py", line 17, in finalize_options
      self.set_undefined_options('install',('install_layout','install_layout'))
    File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 290, in set_undefined_options
      setattr(self, dst_option, getattr(src_cmd_obj, src_option))
    File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 103, in __getattr__
      raise AttributeError(attr)
  AttributeError: install_layout
  ----------------------------------------
  ERROR: Failed building wheel for satnogs-client
ERROR: Could not build wheels for satnogs-client which use PEP 517 and cannot be installed directly
15

Hmm, I recognize the error, I don’t remember what I did to solve that however.
In my opinion it should be a release 1.8.2 or something like that, or at least a easy route to run the latest client.

1 Like