Station can't generate waterfall

Hi,

I finally got Station 1316 rebuilt after cables were damaged by the rotator, and I’m trying to get it on the network again. Here’s what’s happening. Observations start and run fine, but at the end this happens:

May 01 16:56:54 satnogs-gs satnogs-client[27755]: satnogsclient.observer.worker - INFO - Tracking stopped.
May 01 16:56:54 satnogs-gs satnogs-client[27755]: satnogsclient.observer.observer - INFO - Observation Finished
May 01 16:56:54 satnogs-gs satnogs-client[27755]: satnogsclient.observer.observer - INFO - Executing post-observation script.
May 01 16:56:54 satnogs-gs satnogs-client[27755]: satnogsclient.observer.observer - INFO - Rename encoded files for uploading.
May 01 16:56:54 satnogs-gs satnogs-client[27755]: satnogsclient.observer.observer - INFO - Rename encoded file for uploading finished
May 01 16:56:54 satnogs-gs satnogs-client[27755]: satnogsclient.observer.observer - INFO - Rename data file for uploading finished
May 01 16:56:54 satnogs-gs satnogs-client[27755]: satnogsclient.observer.observer - INFO - Creating waterfall plot.
May 01 16:56:54 satnogs-gs satnogs-client[27755]: satnogsclient.observer.waterfall - INFO - Read waterfall file
May 01 16:56:54 satnogs-gs satnogs-client[27755]: apscheduler.executors.default - ERROR - Job “spawn_observer (trigger: date[2020-05-01 20:48:34 UTC], next run at: 2020-05-01 20:48:34 UTC)” raised an exception
May 01 16:56:54 satnogs-gs satnogs-client[27755]: Traceback (most recent call last):
May 01 16:56:54 satnogs-gs satnogs-client[27755]: File “/usr/lib/python3/dist-packages/apscheduler/executors/base.py”, line 125, in run_job
May 01 16:56:54 satnogs-gs satnogs-client[27755]: retval = job.func(*job.args, **job.kwargs)
May 01 16:56:54 satnogs-gs satnogs-client[27755]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py”, line 82, in spawn_observer
May 01 16:56:54 satnogs-gs satnogs-client[27755]: observer.observe()
May 01 16:56:54 satnogs-gs satnogs-client[27755]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 160, in observe
May 01 16:56:54 satnogs-gs satnogs-client[27755]: self.plot_waterfall()
May 01 16:56:54 satnogs-gs satnogs-client[27755]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 258, in plot_waterfall
May 01 16:56:54 satnogs-gs satnogs-client[27755]: waterfall_png=self.observation_waterfall_png)
May 01 16:56:54 satnogs-gs satnogs-client[27755]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/waterfall.py”, line 32, in plot_waterfall
May 01 16:56:54 satnogs-gs satnogs-client[27755]: tmin, tmax = np.min(t_idx), np.max(t_idx)
May 01 16:56:54 satnogs-gs satnogs-client[27755]: File “/usr/lib/python3/dist-packages/numpy/core/fromnumeric.py”, line 2618, in amin
May 01 16:56:54 satnogs-gs satnogs-client[27755]: initial=initial)
May 01 16:56:54 satnogs-gs satnogs-client[27755]: File “/usr/lib/python3/dist-packages/numpy/core/fromnumeric.py”, line 86, in _wrapreduction
May 01 16:56:54 satnogs-gs satnogs-client[27755]: return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
May 01 16:56:54 satnogs-gs satnogs-client[27755]: ValueError: zero-size array to reduction operation minimum which has no identity

I did see the .ogg file in /tmp/.satnogs/data, and but it didn’t seem to have any audio when I looked in audacity (it does look like it has a good header and stuff in xxd). I can upload this OGG somewhere if anyone wants to look at it.

I am running the satnogs-client in a VM on an x86 server (worked fine on an old satnogs-client installation, maybe something changed). I already had issues with upgrading from Debian Stretch to Buster, so I made a new VM and installed Satnogs via Ansible from scratch for a clean slate. Also, due to a hardware issue, I replaced our old USRP2 with a brand new USRP N210 with a new daughterboard, and I am confident this one does not have any issues.

Let me know if I can provide any additional information. Thanks.

Attaching screenshot of hex editor view of OGG and the waterfall dat. Looks to me like the flowgraph doesn’t create a good .ogg.

Here’s a little more log output - the only anomalous thing I see is the AGC warning, but I don’t think that is related. However I can’t figure out how to turn it off either.>

May 01 20:30:29 satnogs-gs satnogs-client[5961]: [INFO] [UHD] linux; GNU C++ version 8.3.0; Boost_106700; UHD_3.15.0.0-2
May 01 20:30:30 satnogs-gs satnogs-client[5961]: [INFO] [usrp2_impl.cpp:309] [USRP2] Opening a USRP2/N-Series device…
May 01 20:30:30 satnogs-gs satnogs-client[5961]: [INFO] [USRP2] Opening a USRP2/N-Series device…
May 01 20:30:30 satnogs-gs satnogs-client[5961]: [INFO] [usrp2_impl.cpp:346] [USRP2] Current recv frame size: 1472 bytes
May 01 20:30:30 satnogs-gs satnogs-client[5961]: [INFO] [USRP2] Current recv frame size: 1472 bytes
May 01 20:30:30 satnogs-gs satnogs-client[5961]: [INFO] [usrp2_impl.cpp:347] [USRP2] Current send frame size: 1472 bytes
May 01 20:30:30 satnogs-gs satnogs-client[5961]: [INFO] [USRP2] Current send frame size: 1472 bytes
May 01 20:30:30 satnogs-gs satnogs-client[5961]: [WARNING] [thread.cpp:19] [UHD] Unable to set the thread priority. Performance may be negatively affected.
May 01 20:30:30 satnogs-gs satnogs-client[5961]: Please see the general application notes in the manual for instructions.
May 01 20:30:30 satnogs-gs satnogs-client[5961]: EnvironmentError: OSError: error in pthread_setschedparam
May 01 20:30:30 satnogs-gs satnogs-client[5961]: [WARNING] [UHD] Unable to set the thread priority. Performance may be negatively affected.
May 01 20:30:30 satnogs-gs satnogs-client[5961]: Please see the general application notes in the manual for instructions.
May 01 20:30:30 satnogs-gs satnogs-client[5961]: EnvironmentError: OSError: error in pthread_setschedparam
May 01 20:30:30 satnogs-gs satnogs-client[5961]: [WARNING] [multi_usrp.cpp:1598] [MULTI_USRP] AGC is not available on this device.
May 01 20:30:30 satnogs-gs satnogs-client[5961]: [WARNING] [MULTI_USRP] AGC is not available on this device.
May 01 20:39:37 satnogs-gs satnogs-client[5961]: 3
May 01 20:39:37 satnogs-gs satnogs-client[5961]: apscheduler.executors.default - ERROR - Job “spawn_observer (trigger: date[2020-05-02 00:30:28 UTC], next run at: 2020-05-02 00:30:28 UTC)” raised an exception
May 01 20:39:37 satnogs-gs satnogs-client[5961]: Traceback (most recent call last):
May 01 20:39:37 satnogs-gs satnogs-client[5961]: File “/usr/lib/python3/dist-packages/apscheduler/executors/base.py”, line 125, in run_job
May 01 20:39:37 satnogs-gs satnogs-client[5961]: retval = job.func(*job.args, **job.kwargs)
May 01 20:39:37 satnogs-gs satnogs-client[5961]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py”, line 82, in spawn_observer
May 01 20:39:37 satnogs-gs satnogs-client[5961]: observer.observe()
May 01 20:39:37 satnogs-gs satnogs-client[5961]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 160, in observe
May 01 20:39:37 satnogs-gs satnogs-client[5961]: self.plot_waterfall()
May 01 20:39:37 satnogs-gs satnogs-client[5961]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 258, in plot_waterfall
May 01 20:39:37 satnogs-gs satnogs-client[5961]: waterfall_png=self.observation_waterfall_png)
May 01 20:39:37 satnogs-gs satnogs-client[5961]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/waterfall.py”, line 32, in plot_waterfall
May 01 20:39:37 satnogs-gs satnogs-client[5961]: tmin, tmax = np.min(t_idx), np.max(t_idx)
May 01 20:39:37 satnogs-gs satnogs-client[5961]: File “/usr/lib/python3/dist-packages/numpy/core/fromnumeric.py”, line 2618, in amin
May 01 20:39:37 satnogs-gs satnogs-client[5961]: initial=initial)
May 01 20:39:37 satnogs-gs satnogs-client[5961]: File “/usr/lib/python3/dist-packages/numpy/core/fromnumeric.py”, line 86, in _wrapreduction
May 01 20:39:37 satnogs-gs satnogs-client[5961]: return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
May 01 20:39:37 satnogs-gs satnogs-client[5961]: ValueError: zero-size array to reduction operation minimum which has no identity

Finally, I have noticed that during the pass, I can see network activity the entire time that is indicative of IQ samples streaming to GNURadio. However /tmp/iq.dat never grows beyond 0 bytes.

Can you please post the output of Advanced > Support from satnogs-setup?

{
“versions”: {
“satnogs-client”: “1.3.1”,
“satnogs-client-ansible”: “202004031132”,
“satnogs-flowgraphs”: “1.1.1-1”,
“gr-satnogs”: “2.1.2-1”,
“gr-soapy”: “2.1.2-1”,
“gnuradio”: “3.8.1.0~rc1-2”,
“satnogs-config”: “0.10.1”
},
“state”: {
“is-applied”: true,
“pending-tags”: null
},
“system”: {
“distribution”: {
“DESCRIPTION”: “Debian GNU/Linux 10 (buster)”,
“RELEASE”: “10”,
“CODENAME”: “buster”,
“ID”: “Debian”
},
“pending-updates”: false,
“platform”: {
“system”: “Linux”,
“node”: “satnogs-gs”,
“release”: “4.19.0-8-amd64”,
“version”: “#1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27)”,
“machine”: “x86_64”,
“processor”: “”
},
“memory”: {
“total”: 4138713088,
“available”: 3725340672,
“percent”: 10.0,
“used”: 173473792,
“free”: 3468562432,
“active”: 425635840,
“inactive”: 128278528,
“buffers”: 35524608,
“cached”: 461152256,
“shared”: 5668864,
“slab”: 71684096
},
“disk”: {
“total”: 63143981056,
“used”: 4306546688,
“free”: 55599484928,
“percent”: 7.2
}
},
“configuration”: {
“enable_iq_dump”: true,
“satnogs_antenna”: “TX/RX”,
“satnogs_api_token”: “[redacted]”,
“satnogs_gain_mode”: “Settings Field”,
“satnogs_other_settings”: “PGA0=12.5”,
“satnogs_remove_raw_files”: false,
“satnogs_rot_model”: “ROT_MODEL_NETROTCTL”,
“satnogs_rot_port”: “groundstation.localdomain:4533”,
“satnogs_rx_samp_rate”: “2000000”,
“satnogs_soapy_rx_device”: “driver=uhd,serial=31B4FAF”,
“satnogs_station_elev”: “220”,
“satnogs_station_id”: “1316”,
“satnogs_station_lat”: “39.3347”,
“satnogs_station_lon”: “-82.1198”
}
}

I just changed the gain settings to “Settings Field” with “PGA”, the issue happened before that too.

This afternoon, after updating my satnogs, I have no waterfall at any observation…

● satnogs-client.service - SatNOGS client
Loaded: loaded (/etc/systemd/system/satnogs-client.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-06-19 19:46:14 BST; 22min ago
Main PID: 249 (satnogs-client)
Memory: 135.2M
CGroup: /system.slice/satnogs-client.service
└─249 /var/lib/satnogs/bin/python3 /var/lib/satnogs/bin/satnogs-client

jun 19 19:57:12 raspberrypi satnogs-client[249]: observer.observe()
jun 19 19:57:12 raspberrypi satnogs-client[249]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 158, in obser
jun 19 19:57:12 raspberrypi satnogs-client[249]: self.plot_waterfall()
jun 19 19:57:12 raspberrypi satnogs-client[249]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 259, in plot_
jun 19 19:57:12 raspberrypi satnogs-client[249]: waterfall_png=self.observation_waterfall_png)
jun 19 19:57:12 raspberrypi satnogs-client[249]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/waterfall.py”, line 39, in plot_
jun 19 19:57:12 raspberrypi satnogs-client[249]: waterfall = read_waterfall(waterfall_file)
jun 19 19:57:12 raspberrypi satnogs-client[249]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/waterfall.py”, line 22, in read_
jun 19 19:57:12 raspberrypi satnogs-client[249]: ‘timestamp’: np.fromfile(wf_file, dtype=’|S32’, count=1)[0],
jun 19 19:57:12 raspberrypi satnogs-client[249]: IndexError: index 0 is out of bounds for axis 0 with size 0

Generating support report. Please wait…
------------[ copy here ]------------
{
“versions”: {
“satnogs-client”: “1.3.2”,
“satnogs-client-ansible”: “202006062329”,
“satnogs-flowgraphs”: “1.2.1-1”,
“gr-satnogs”: “2.1.2.1-1”,
“gr-soapy”: “2.1.3-1”,
“gnuradio”: “3.8.1.0~rc1-2”,
“satnogs-config”: “0.10.2”
},
“state”: {
“is-applied”: true,
“pending-tags”: null
},
“system”: {
“date”: “2020-06-19T19:12:31.724041+00:00”,
“distribution”: {
“DESCRIPTION”: “Raspbian GNU/Linux 10 (buster)”,
“RELEASE”: “10”,
“CODENAME”: “buster”,
“ID”: “Raspbian”
},
“pending-updates”: false,
“platform”: {
“system”: “Linux”,
“node”: “raspberrypi”,
“release”: “4.19.118+”,
“version”: “#1311 Mon Apr 27 14:16:15 BST 2020”,
“machine”: “armv6l”,
“processor”: “”
},
“memory”: {
“total”: 453414912,
“available”: 312930304,
“percent”: 31.0,
“used”: 84770816,
“free”: 101572608,
“active”: 96342016,
“inactive”: 220200960,
“buffers”: 21663744,
“cached”: 245407744,
“shared”: 3170304,
“slab”: 22986752
},
“disk”: {
“total”: 30293102592,
“used”: 4001869824,
“free”: 25013178368,
“percent”: 13.8
}
},
“configuration”: {
“satnogs_antenna”: “RX”,
“satnogs_api_token”: “[redacted]”,
“satnogs_rf_gain”: “43.9”,
“satnogs_rx_samp_rate”: “0.25e6”,
“satnogs_soapy_rx_device”: “driver=rtlsdr”,
“satnogs_station_elev”: “700”,
“satnogs_station_id”: “1589”,
“satnogs_station_lat”: “38.982”,
“satnogs_station_lon”: “-1.868”
}
}
------------[ copy end ]-------------

That sample rate looks very small… Are you sure that’s what you wanted?

Usually we recommend running the RTLSDRs at a sample rate of 2.048 MHz (2.048e6)

73
Mark VK5QI

1 Like

Seems that only works with a sample rate of 1.024e6…I’d like to try with 0.250e6 because with Cubic SDR works better with that sample rate.
With 1.024e6 waterfall is working but audio seems interrupted, that why I¡d like to try with other sample rates values

If you’re having issues with higher sample rates, it’s possible your host machine isn’t fast enough?

What model RPi is this running on? I’ve had no issues running 2.048 MHz sample rate on a RPi 3 and up…

73
Mark VK5QI

1 Like

Yes, you’re right… My Rpi is an old one. I ve to change it

Many thanks 73’s