ReadTimeout - Unable To Undertake Scheduled Observation

Hi all,

Having an issue getting my station up and running, and have spent substantial time reading and troubleshooting before this post. I’m guessing this is likely a PEBKAC issue (Problem Exists Between Keyboard And Chair), though could use some advice.

Thanks in advance.


Problem:
After installing and setting up my Pi and Satnogs client, I can schedule an observation successfully. When it comes time for the observation to run - it appears that little happens and that no observation is made.


What I Have Done

  • Reinstalled image from scratch
  • Tried another RPi
  • Ensured that Raspbian is up to date
  • Ensured that satnogs-config is updated
  • Installed Satnogs Monitor - I can see the jobs come up, but when the job should be active there is no waterfall or activity
  • rtl_test is successful
  • /tmp/.satnogs/data temporarily has the ‘receiving_waterfall…dat’ file of 0 size
  • /tmp/.satnogs/data/incomplete is empty
  • Installed SoapySDR and attempted a connection with CubicSDR on Win (crashes)
  • There are some kernel mismatch errors, but after investigating these appear unrelated and of no concern (common in current Raspbian)
  • APPLY - Interestingly, the Apply function ALWAYS stalls and I must Ctrl-C (this occurred on both Pi’s and with different installs) - This happens on 'Download UHD images…

systemctl status satnogs-client

pi@RPi1683:/tmp/.satnogs/data $ systemctl status satnogs-client
● satnogs-client.service - SatNOGS client
Loaded: loaded (/etc/systemd/system/satnogs-client.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2020-07-26 14:03:12 UTC; 36min ago
Main PID: 344 (satnogs-client)
Tasks: 22 (limit: 4915)
CGroup: /system.slice/satnogs-client.service
└─344 /var/lib/satnogs/bin/python3 /var/lib/satnogs/bin/satnogs-client

Jul 26 14:22:38 RPi1683 satnogs-client[344]: observer.observe()
Jul 26 14:22:38 RPi1683 satnogs-client[344]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 165, in
Jul 26 14:22:38 RPi1683 satnogs-client[344]: self.plot_waterfall()
Jul 26 14:22:38 RPi1683 satnogs-client[344]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 267, in
Jul 26 14:22:38 RPi1683 satnogs-client[344]: waterfall_png=self.observation_waterfall_png)
Jul 26 14:22:38 RPi1683 satnogs-client[344]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/waterfall.py”, line 39, in
Jul 26 14:22:38 RPi1683 satnogs-client[344]: waterfall = read_waterfall(waterfall_file)
Jul 26 14:22:38 RPi1683 satnogs-client[344]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/waterfall.py”, line 22, in
Jul 26 14:22:38 RPi1683 satnogs-client[344]: ‘timestamp’: np.fromfile(wf_file, dtype=’|S32’, count=1)[0],
Jul 26 14:22:38 RPi1683 satnogs-client[344]: IndexError: index 0 is out of bounds for axis 0 with size 0
lines 1-18/18 (END)


Support File

------------[ copy here ]------------
{
“versions”: {
“satnogs-client”: “1.3.4”,
“satnogs-client-ansible”: “202007160921”,
“satnogs-flowgraphs”: “1.2.2-1”,
“gr-satnogs”: “2.1.2.1-1”,
“gr-soapy”: “2.1.3-1”,
“gnuradio”: “3.8.1.0~rc1-2”,
“satnogs-config”: “0.10.2”
},
“state”: {
“is-applied”: false,
“pending-tags”: null
},
“system”: {
“date”: “2020-07-26T14:27:58.968153+00:00”,
“distribution”: {
“DESCRIPTION”: “Raspbian GNU/Linux 10 (buster)”,
“RELEASE”: “10”,
“CODENAME”: “buster”,
“ID”: “Raspbian”
},
“pending-updates”: false,
“platform”: {
“system”: “Linux”,
“node”: “RPi1683”,
“release”: “5.4.51-v7l+”,
“version”: “#1327 SMP Thu Jul 23 11:04:39 BST 2020”,
“machine”: “armv7l”,
“processor”: “”
},
“memory”: {
“total”: 4013887488,
“available”: 3766697984,
“percent”: 6.2,
“used”: 106831872,
“free”: 3558486016,
“active”: 262766592,
“inactive”: 135737344,
“buffers”: 27209728,
“cached”: 321359872,
“shared”: 8851456,
“slab”: 31088640
},
“disk”: {
“total”: 15351623680,
“used”: 3667795968,
“free”: 11012694016,
“percent”: 25.0
}
},
“configuration”: {
“satnogs_antenna”: “RX”,
“satnogs_api_token”: “[redacted]”,
“satnogs_rf_gain”: “32.8”,
“satnogs_rx_samp_rate”: “2.048e6”,
“satnogs_soapy_rx_device”: “driver=rtlsdr”,
“satnogs_station_elev”: “195”,
“satnogs_station_id”: “1683”,
“satnogs_station_lat”: “-35.051”,
“satnogs_station_lon”: “138.605”
}
}
------------[ copy end ]-------------

Some further information:
This was taken during an observation:

ls-lh on /tmp/.satnogs/data

pi@RPi1683:/tmp/.satnogs/data $ ls -lh
total 8.0K
drwxr-xr-x 2 satnogs satnogs 40 Jul 26 14:53 incomplete
-rw-r–r-- 1 satnogs satnogs 0 Jul 27 00:09 receiving_waterfall_2607981_2020-07-27T00-09-41.dat
-rw-r–r-- 1 satnogs satnogs 4.2K Jul 27 00:09 satnogs_2607981_2020-07-27T00-09-41.ogg

systemctl status satnogs-client.service

pi@RPi1683:/tmp/.satnogs/data $ systemctl status satnogs-client.service
● satnogs-client.service - SatNOGS client
Loaded: loaded (/etc/systemd/system/satnogs-client.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2020-07-26 14:53:20 UTC; 9h ago
Main PID: 356 (satnogs-client)
Tasks: 22 (limit: 4915)
CGroup: /system.slice/satnogs-client.service
└─356 /var/lib/satnogs/bin/python3 /var/lib/satnogs/bin/satnogs-client

Jul 27 00:09:56 RPi1683 satnogs-client[356]: observer.observe()
Jul 27 00:09:56 RPi1683 satnogs-client[356]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 165
Jul 27 00:09:56 RPi1683 satnogs-client[356]: self.plot_waterfall()
Jul 27 00:09:56 RPi1683 satnogs-client[356]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 267
Jul 27 00:09:56 RPi1683 satnogs-client[356]: waterfall_png=self.observation_waterfall_png)
Jul 27 00:09:56 RPi1683 satnogs-client[356]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/waterfall.py”, line 39
Jul 27 00:09:56 RPi1683 satnogs-client[356]: waterfall = read_waterfall(waterfall_file)
Jul 27 00:09:56 RPi1683 satnogs-client[356]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/waterfall.py”, line 22
Jul 27 00:09:56 RPi1683 satnogs-client[356]: ‘timestamp’: np.fromfile(wf_file, dtype=’|S32’, count=1)[0],
Jul 27 00:09:56 RPi1683 satnogs-client[356]: IndexError: index 0 is out of bounds for axis 0 with size 0

The .ogg file disappears before the observation is concluded. The .dat file at this time remains (observation still underway).

Edit:
Observation is now finished. Both above commands are still giving exactly the same output. There is no sign of a successful observation on the dashboard.

Further information:

HTTPSConnectionPool(host=‘network.satnogs.org’, port=443): Read timed out. (read timeout=45)

pi@RPi1683:/tmp/.satnogs/data $ systemctl status satnogs-client.service
● satnogs-client.service - SatNOGS client
Loaded: loaded (/etc/systemd/system/satnogs-client.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2020-07-26 14:53:20 UTC; 11h ago
Main PID: 356 (satnogs-client)
Tasks: 22 (limit: 4915)
CGroup: /system.slice/satnogs-client.service
└─356 /var/lib/satnogs/bin/python3 /var/lib/satnogs/bin/satnogs-client

Jul 27 02:25:09 RPi1683 satnogs-client[356]: return request(‘get’, url, params=params, **kwargs)
Jul 27 02:25:09 RPi1683 satnogs-client[356]: File “/usr/lib/python3/dist-packages/requests/api.py”, line 60, in request
Jul 27 02:25:09 RPi1683 satnogs-client[356]: return session.request(method=method, url=url, **kwargs)
Jul 27 02:25:09 RPi1683 satnogs-client[356]: File “/usr/lib/python3/dist-packages/requests/sessions.py”, line 533, in request
Jul 27 02:25:09 RPi1683 satnogs-client[356]: resp = self.send(prep, **send_kwargs)
Jul 27 02:25:09 RPi1683 satnogs-client[356]: File “/usr/lib/python3/dist-packages/requests/sessions.py”, line 646, in send
Jul 27 02:25:09 RPi1683 satnogs-client[356]: r = adapter.send(request, **kwargs)
Jul 27 02:25:09 RPi1683 satnogs-client[356]: File “/usr/lib/python3/dist-packages/requests/adapters.py”, line 529, in send
Jul 27 02:25:09 RPi1683 satnogs-client[356]: raise ReadTimeout(e, request=request)
Jul 27 02:25:09 RPi1683 satnogs-client[356]: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host=‘network.satnogs.org’, port=443): Read timed out. (read timeout=45)

Output from monitoring the service during observation.

journalctl -f -u satnogs-client.service

pi@RPi1683:~ $ journalctl -f -u satnogs-client.service
– Logs begin at Mon 2020-07-27 10:14:47 UTC. –
Jul 27 11:10:49 RPi1683 satnogs-client[342]: observer.observe()
Jul 27 11:10:49 RPi1683 satnogs-client[342]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py”, line 134, in observe
Jul 27 11:10:49 RPi1683 satnogs-client[342]: subprocess.call(pre_script)
Jul 27 11:10:49 RPi1683 satnogs-client[342]: File “/usr/lib/python3.7/subprocess.py”, line 323, in call
Jul 27 11:10:49 RPi1683 satnogs-client[342]: with Popen(*popenargs, **kwargs) as p:
Jul 27 11:10:49 RPi1683 satnogs-client[342]: File “/usr/lib/python3.7/subprocess.py”, line 775, in init
Jul 27 11:10:49 RPi1683 satnogs-client[342]: restore_signals, start_new_session)
Jul 27 11:10:49 RPi1683 satnogs-client[342]: File “/usr/lib/python3.7/subprocess.py”, line 1522, in _execute_child
Jul 27 11:10:49 RPi1683 satnogs-client[342]: raise child_exception_type(errno_num, err_msg, err_filename)
Jul 27 11:10:49 RPi1683 satnogs-client[342]: FileNotFoundError: [Errno 2] No such file or directory: ‘home/pi/rtl_biast/build/src/rtl_biast’: ‘home/pi/rtl_biast/build/src/rtl_biast’

That looks like you’re missing a leading / on the path to your pre(?)-observation script. Not sure if that would completely break an observation, but you’ll need to fix that if you want your bias-tee to work.

Thanks, I’ll fix that. I only just added that as added the hardware, but the issues were occurring beforehand.

Whilst the files seem to be created, if I watch on monitor, there is no waterfall or spectrum showing, which leads me to wonder if something else isn’t right. Do you have any other suggestions?

(Is the ‘Applied:False’ in regards to State an issue? The Apply always hangs at download UDP images and I have to Ctrl-C)

Ok, solution has been found - thought I would detail here in case it is of use for others. If anyone has any further understanding about what’s under the bonnet, to add, please do and I’ll amend.

It appears that the initial run of satnogs-setup failed due to hanging on ‘Download UDP images’. This means that some necessary components were never downloaded.

I initially had this occur on MULTIPLE RPi’s, over multiple installs. The clue to the actual problem was found in this error message:

satnogs-client[356]: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host=‘network.satnogs.org’, port=443): Read timed out. (read timeout=45)

As I was installing some other software on a separate unrelated RPi, I noticed that I had a similar error when installing via the terminal. Trying again fixed this. But I realised that I should consider my home network as an issue - and so maybe wireless connection contributed to this. So I installed on a different RPi, via Ethernet, and success.

Though with satnogs-setup via wifi on the original station, no matter the amount of times I tried to Apply, it never ceased hanging. So I connected it via Ethernet cable, and disabled Wifi in the config.txt.

  • sudo nano /boot/config.txt
  • Find the following line:
    # Additional overlays and parameters are documented /boot/overlays/README
  • And add these two lines under it :
    dtoverlay=disable-wifidtoverlay=disable-bt

I then went into satnogs-setup:
Reboot > Update > Reboot > Apply.

I saw the finalisation of the software install. I’ll shortly be testing, but I believe this will be resolved.