Satnogs-client failed

SatNOGS Software
1

Dec 8 06:27:04 satnogs satnogs-client[748]: 2019-12-08 06:27:04,041 - apscheduler.executors.default - ERROR - Job “spawn_observer (trigger: date[2019-12-08 05:16:30 UTC], next run at: 2019-12-08 05:16:30 UTC)” raised an exception
Dec 8 06:27:04 satnogs satnogs-client[748]: Traceback (most recent call last):
Dec 8 06:27:04 satnogs satnogs-client[748]: File “/usr/local/lib/python2.7/dist-packages/apscheduler/executors/base.py”, line 125, in run_job
Dec 8 06:27:04 satnogs satnogs-client[748]: retval = job.func(*job.args, **job.kwargs)
Dec 8 06:27:04 satnogs satnogs-client[748]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/scheduler/tasks.py”, line 110, in spawn_observer
Dec 8 06:27:04 satnogs satnogs-client[748]: observer.observe()
Dec 8 06:27:04 satnogs satnogs-client[748]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/observer.py”, line 301, in observe
Dec 8 06:27:04 satnogs satnogs-client[748]: self.plot_waterfall()
Dec 8 06:27:04 satnogs satnogs-client[748]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/observer.py”, line 393, in plot_waterfall
Dec 8 06:27:04 satnogs satnogs-client[748]: tmin, tmax = np.min(t_idx), np.max(t_idx)
Dec 8 06:27:04 satnogs satnogs-client[748]: File “/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py”, line 2618, in amin
Dec 8 06:27:04 satnogs satnogs-client[748]: initial=initial)
Dec 8 06:27:04 satnogs satnogs-client[748]: File “/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py”, line 86, in _wrapreduction
Dec 8 06:27:04 satnogs satnogs-client[748]: return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
Dec 8 06:27:04 satnogs satnogs-client[748]: ValueError: zero-size array to reduction operation minimum which has no identity

Station failed for all observations after that and running normaly after restart. No error from RTL device driver, but seems failed return values.
After several searches in the log files, it seems that the dongle no longer returned data. Would it be possible on this type of error to reset the USB port ?

Uptime before crash : 1 month
I test command “usbreset RTL2838UHIDIR” at pre and post observation script.

2

RTL device failed and usb on my pc cannot support usb on/off by command, after unplug / plug device station restart.

3

There is a way to “reboot” only USB system (see How do I troubleshoot missing waterfalls?).

1 Like
4

Station failed today, i’ve resolve RTL device failed without restart all system.

RTL Device is connected to usb1
echo “usb1” > /sys/bus/usb/drivers/usb/unbind
echo “usb1” > /sys/bus/usb/drivers/usb/bind

I’ve add this on my satnogs_post_observation_script for test if i can resolve problem for long time.

3 Likes
5

Unbinding and binding the rtlsdr helped when my station stopped working.

The satnogs user, which runs the post observation script, did not have sufficient permissions on my system. This gives the satnogs user permissions:
sudo chgrp satnogs /sys/bus/usb/drivers/usb/{un,}bind
sudo chmod g+w /sys/bus/usb/drivers/usb/{un,}bind

But this change does not persist over reboots and I am unable to find a way to do that. I tried rc.local and cron @reboot but that seems to be too early (I assume the sys tree hasn’t been populated yet). If anyone knows the right way to allow a user access to bind/unbind, feel free to share.

6

Use sudo command :
https://askubuntu.com/questions/159007/how-do-i-run-specific-sudo-commands-without-a-password
satnogs ALL=(ALL) NOPASSWD: {here your path}/pre-observation.sh
satnogs ALL=(ALL) NOPASSWD: {here your path}/post-observation.sh

and in satnog-setup / advanced / scripts:
sudo {here your path}/pre-observation.sh
sudo {here your path}/post-observation.sh

Pre and post observation script run with root privileges

For found usb number use command : dmesg | grep usb

1 Like
7

If anyone follows the above advice, make sure the pre- and post observation scripts only can be edited by the root user. Otherwise, the satnogs user effectively has root privileges.

1 Like
8

Station failed today, all memory (3 GBytes) allocate by satnogs-client…
238,5 MBytes use after reboot.
I will take a closer look