My station has only black observations (no waterfall, no audio)

1

My station is unable to upload anything. The observations are black with no waterfall and no audio. Restarting satnogs-client does not help.

Support info:

    {
        "versions": {
            "satnogs-client": "1.4",
            "satnogs-client-ansible": "202012231828",
            "satnogs-flowgraphs": "1.3-1",
            "gr-satnogs": "2.2.0.0-1",
            "gr-soapy": "2.1.3-1",
            "gnuradio": "3.8.2.0-2",
            "satnogs-config": "0.11"
        },
        "state": {
            "is-applied": true,
            "pending-tags": null
        },
        "system": {
            "date": "2021-01-27T12:26:09.425837+00:00",
            "distribution": {
                "DESCRIPTION": "Raspbian GNU/Linux 10 (buster)",
                "RELEASE": "10",
                "CODENAME": "buster",
                "ID": "Raspbian"
            },
            "pending-updates": true,
            "platform": {
                "system": "Linux",
                "node": "satnogs-fhab-1",
                "release": "5.4.83-v7l+",
                "version": "#1379 SMP Mon Dec 14 13:11:54 GMT 2020",
                "machine": "armv7l",
                "processor": ""
            },
            "memory": {
                "total": 4013871104,
                "available": 3679256576,
                "percent": 8.3,
                "used": 183197696,
                "free": 3302608896,
                "active": 411217920,
                "inactive": 216293376,
                "buffers": 99909632,
                "cached": 428154880,
                "shared": 8810496,
                "slab": 54046720
            },
            "disk": {
                "total": 30142779392,
                "used": 5676032000,
                "free": 23195947008,
                "percent": 19.7
            }
        },
        "configuration": {
            "enable_iq_dump": true,
            "hamlib_utils_rot_enabled": false,
            "iq_dump_filename": "/tmp/.satnogs/iq.raw",
            "satnogs_antenna": "RX",
            "satnogs_api_token": "[redacted]",
            "satnogs_post_observation_script": "/usr/local/bin/satnogs-post {{ID}} {{FREQ}} {{TLE}} {{TIMESTAMP}} {{BAUD}} {{SCRIPT_NAME}}",
            "satnogs_rf_gain": 48.0,
            "satnogs_rx_device": "rtlsdr",
            "satnogs_rx_samp_rate": "2.048e6",
            "satnogs_soapy_rx_device": "driver=rtlsdr",
            "satnogs_station_elev": 67,
            "satnogs_station_id": 1272,
            "satnogs_station_lat": 56.82,
            "satnogs_station_lon": 12.89
        }
    }

daemon.log:

Jan 26 11:39:24 satnogs-fhab-1 satnogs-client[374]: Found Rafael Micro R820T tuner
Jan 26 11:39:24 satnogs-fhab-1 satnogs-client[374]: [INFO] Opening Generic RTL2832U OEM :: 00000001...
Jan 26 11:39:24 satnogs-fhab-1 satnogs-client[374]: Found Rafael Micro R820T tuner
Jan 26 11:39:25 satnogs-fhab-1 satnogs-client[374]: [R82XX] PLL not locked!
Jan 26 11:39:25 satnogs-fhab-1 satnogs-client[374]: [INFO] Using format CF32.
Jan 26 11:39:25 satnogs-fhab-1 satnogs-client[374]: Allocating 15 zero-copy buffers
Jan 26 11:46:38 satnogs-fhab-1 satnogs-client[374]: Found Rafael Micro R820T tuner
Jan 26 11:46:39 satnogs-fhab-1 satnogs-client[374]: [INFO] Opening Generic RTL2832U OEM :: 00000001...
Jan 26 11:46:39 satnogs-fhab-1 satnogs-client[374]: Found Rafael Micro R820T tuner
Jan 26 11:46:39 satnogs-fhab-1 satnogs-client[374]: [R82XX] PLL not locked!
Jan 26 11:46:39 satnogs-fhab-1 satnogs-client[374]: [INFO] Using format CF32.
Jan 26 11:46:39 satnogs-fhab-1 satnogs-client[374]: Allocating 15 zero-copy buffers
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]: apscheduler.executors.default - ERROR - Job "get_jobs (trigger: interval[0:01:00], next run at: 2021-01-26 12:08:18 UTC)" raised an exception
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]: Traceback (most recent call last):
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     six.raise_from(e, None)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "<string>", line 3, in raise_from
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     httplib_response = conn.getresponse()
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/usr/lib/python3.7/http/client.py", line 1336, in getresponse
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     response.begin()
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/usr/lib/python3.7/http/client.py", line 306, in begin
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     version, status, reason = self._read_status()
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/usr/lib/python3.7/socket.py", line 589, in readinto
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     return self._sock.recv_into(b)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/usr/lib/python3.7/ssl.py", line 1052, in recv_into
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     return self.read(nbytes, buffer)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/usr/lib/python3.7/ssl.py", line 911, in read
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     return self._sslobj.read(len, buffer)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]: socket.timeout: The read operation timed out
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]: During handling of the above exception, another exception occurred:
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]: Traceback (most recent call last):
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     timeout=timeout
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     raise six.reraise(type(error), error, _stacktrace)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     raise value
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     chunked=chunked,
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 428, in _make_request
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     self, url, "Read timed out. (read timeout=%s)" % timeout_value
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]: urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='network.satnogs.org', port=443): Read timed out. (read timeout=45)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]: During handling of the above exception, another exception occurred:
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]: Traceback (most recent call last):
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/apscheduler/executors/base.py", line 125, in run_job
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     retval = job.func(*job.args, **job.kwargs)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 171, in get_jobs
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     timeout=45)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/api.py", line 76, in get
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     return request('get', url, params=params, **kwargs)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/api.py", line 61, in request
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     return session.request(method=method, url=url, **kwargs)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     resp = self.send(prep, **send_kwargs)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     r = adapter.send(request, **kwargs)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]:     raise ReadTimeout(e, request=request)
Jan 26 12:08:14 satnogs-fhab-1 satnogs-client[374]: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='network.satnogs.org', port=443): Read timed out. (read timeout=45)
Jan 26 12:56:09 satnogs-fhab-1 satnogs-client[374]: Found Rafael Micro R820T tuner
Jan 26 12:56:09 satnogs-fhab-1 satnogs-client[374]: [INFO] Opening Generic RTL2832U OEM :: 00000001...
Jan 26 12:56:09 satnogs-fhab-1 satnogs-client[374]: Found Rafael Micro R820T tuner
Jan 26 12:56:10 satnogs-fhab-1 satnogs-client[374]: [R82XX] PLL not locked!
Jan 26 12:56:10 satnogs-fhab-1 satnogs-client[374]: [INFO] Using format CF32.
Jan 26 12:56:10 satnogs-fhab-1 satnogs-client[374]: Allocating 15 zero-copy buffers
Jan 26 13:23:54 satnogs-fhab-1 satnogs-client[374]: Found Rafael Micro R820T tuner
Jan 26 13:23:54 satnogs-fhab-1 satnogs-client[374]: [INFO] Opening Generic RTL2832U OEM :: 00000001...
Jan 26 13:23:54 satnogs-fhab-1 satnogs-client[374]: Found Rafael Micro R820T tuner
Jan 26 13:23:55 satnogs-fhab-1 satnogs-client[374]: [R82XX] PLL not locked!
Jan 26 13:23:55 satnogs-fhab-1 satnogs-client[374]: [INFO] Using format CF32.
Jan 26 13:23:55 satnogs-fhab-1 satnogs-client[374]: Allocating 15 zero-copy buffers
Jan 26 14:02:57 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 14:17:21 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 14:29:32 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 14:35:52 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 15:16:12 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 16:16:05 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 16:32:31 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 17:11:27 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 17:39:33 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 18:44:04 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 19:08:28 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 19:27:00 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 19:46:12 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 20:13:32 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 20:42:52 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 20:49:22 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 20:57:12 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 21:16:41 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 22:00:54 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 26 22:31:44 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 00:09:06 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 01:12:57 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 02:26:19 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 02:58:38 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 03:24:54 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 03:43:23 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 04:19:12 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 04:35:37 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 05:03:02 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 05:12:45 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 06:23:58 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 06:50:24 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 07:02:39 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 07:08:05 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 07:32:21 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 08:33:03 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 08:42:15 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 08:48:11 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 08:57:43 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 09:03:24 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 09:19:06 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 09:42:45 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 10:12:12 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 10:20:27 satnogs-fhab-1 satnogs-client[374]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 10:25:46 satnogs-fhab-1 systemd[1]: satnogs-client.service: Main process exited, code=killed, status=15/TERM
Jan 27 10:27:32 satnogs-fhab-1 systemd[1]: satnogs-client.service: State 'stop-final-sigterm' timed out. Killing.
Jan 27 10:27:32 satnogs-fhab-1 systemd[1]: satnogs-client.service: Killing process 10016 (satnogs_fsk.py) with signal SIGKILL.
Jan 27 10:27:32 satnogs-fhab-1 systemd[1]: satnogs-client.service: Failed with result 'timeout'.
Jan 27 10:56:51 satnogs-fhab-1 satnogs-client[13977]: rot_init called
Jan 27 10:56:51 satnogs-fhab-1 satnogs-client[13977]: dummy: _init called
Jan 27 10:56:51 satnogs-fhab-1 satnogs-client[13977]: rot_register (1)
Jan 27 10:56:51 satnogs-fhab-1 satnogs-client[13977]: rot_register (2)
Jan 27 10:56:51 satnogs-fhab-1 satnogs-client[13977]: dummy_rot_init called
Jan 27 10:56:51 satnogs-fhab-1 satnogs-client[13977]: rot_open called
Jan 27 10:56:51 satnogs-fhab-1 satnogs-client[13977]: dummy_rot_open called
Jan 27 10:56:51 satnogs-fhab-1 satnogs-client[13977]: rot_get_position called
Jan 27 10:56:51 satnogs-fhab-1 satnogs-client[13977]: dummy_rot_get_position called
Jan 27 10:56:51 satnogs-fhab-1 satnogs-client[13977]: rot_set_position called
Jan 27 10:56:51 satnogs-fhab-1 satnogs-client[13977]: dummy_rot_set_position called: 157.72 20.30
Jan 27 10:56:53 satnogs-fhab-1 satnogs-client[13977]: Found Rafael Micro R820T tuner
Jan 27 10:56:54 satnogs-fhab-1 satnogs-client[13977]: [INFO] Opening Generic RTL2832U OEM :: 00000001...
Jan 27 10:56:54 satnogs-fhab-1 satnogs-client[13977]: Found Rafael Micro R820T tuner
Jan 27 10:56:54 satnogs-fhab-1 satnogs-client[13977]: [R82XX] PLL not locked!
Jan 27 10:56:54 satnogs-fhab-1 satnogs-client[13977]: [INFO] Using format CF32.
Jan 27 10:56:54 satnogs-fhab-1 satnogs-client[13977]: Allocating 15 zero-copy buffers
Jan 27 11:14:41 satnogs-fhab-1 satnogs-client[13977]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 11:34:53 satnogs-fhab-1 satnogs-client[13977]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 11:44:19 satnogs-fhab-1 satnogs-client[13977]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]: apscheduler.executors.default - ERROR - Job "get_jobs (trigger: interval[0:01:00], next run at: 2021-01-27 12:07:34 UTC)" raised an exception
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]: Traceback (most recent call last):
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     six.raise_from(e, None)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "<string>", line 3, in raise_from
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     httplib_response = conn.getresponse()
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/usr/lib/python3.7/http/client.py", line 1336, in getresponse
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     response.begin()
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/usr/lib/python3.7/http/client.py", line 306, in begin
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     version, status, reason = self._read_status()
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/usr/lib/python3.7/socket.py", line 589, in readinto
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     return self._sock.recv_into(b)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/usr/lib/python3.7/ssl.py", line 1052, in recv_into
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     return self.read(nbytes, buffer)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/usr/lib/python3.7/ssl.py", line 911, in read
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     return self._sslobj.read(len, buffer)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]: socket.timeout: The read operation timed out
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]: During handling of the above exception, another exception occurred:
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]: Traceback (most recent call last):
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     timeout=timeout
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     raise six.reraise(type(error), error, _stacktrace)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     raise value
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     chunked=chunked,
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 428, in _make_request
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     self, url, "Read timed out. (read timeout=%s)" % timeout_value
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]: urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='network.satnogs.org', port=443): Read timed out. (read timeout=45)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]: During handling of the above exception, another exception occurred:
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]: Traceback (most recent call last):
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/apscheduler/executors/base.py", line 125, in run_job
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     retval = job.func(*job.args, **job.kwargs)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 171, in get_jobs
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     timeout=45)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/api.py", line 76, in get
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     return request('get', url, params=params, **kwargs)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/api.py", line 61, in request
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     return session.request(method=method, url=url, **kwargs)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     resp = self.send(prep, **send_kwargs)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     r = adapter.send(request, **kwargs)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]:     raise ReadTimeout(e, request=request)
Jan 27 12:07:25 satnogs-fhab-1 satnogs-client[13977]: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='network.satnogs.org', port=443): Read timed out. (read timeout=45)

Network connectivity is ok

    pi@satnogs-fhab-1:~ $ ping network.satnogs.org
    PING srv01.libre.space (94.130.162.100) 56(84) bytes of data.
    64 bytes from srv01.libre.space (94.130.162.100): icmp_seq=1 ttl=54 time=25.10 ms
    64 bytes from srv01.libre.space (94.130.162.100): icmp_seq=2 ttl=54 time=23.7 ms
    64 bytes from srv01.libre.space (94.130.162.100): icmp_seq=3 ttl=54 time=24.4 ms
    ^C
    --- srv01.libre.space ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 5ms
    rtt min/avg/max/mdev = 23.699/24.706/25.977/0.948 ms
    pi@satnogs-fhab-1:~ $ wget -O /dev/null 'https://network.satnogs.org'
    --2021-01-27 12:31:11--  https://network.satnogs.org/
    Resolving network.satnogs.org (network.satnogs.org)... 94.130.162.100, 2a01:4f8:13b:29db::1
    Connecting to network.satnogs.org (network.satnogs.org)|94.130.162.100|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 6054 (5.9K) [text/html]
    Saving to: ‘/dev/null’

    /dev/null                                                   100%[===========================================================================================================================================>]   5.91K  --.-KB/s    in 0s

    2021-01-27 12:31:11 (121 MB/s) - ‘/dev/null’ saved [6054/6054]
2

oh, and the code formatting button in the forum seems broken. It inserts spaces at the start of each line, which apparently doesn’t work. I had to manually type triple backtick instead.

3

I restarted the pi and it started working again.

4

Well, that only lasted about 10 hours. Apparently, my station has been failing all observations since 2021-01-29 08:40.

So the last 7 days, my station has made about 240 black observations (out of total 300).

5

Tried unbinding usb (echo "usb1" > /sys/bus/usb/drivers/usb/unbind; sleep 1;echo "usb1" > /sys/bus/usb/drivers/usb/bind) but that didn’t help either.

Restarting satnogs-client didn’t help. But stopping the service and then starting it again did help. My station works again, at least for the moment.

Maybe I should add a reboot of the Pi at the end of my post observation script?

6

I just come to say that my stations 771 and 812 also show this problem. Rebooting is a workaround as restarting satnogs-client sometimes works, sometimes it does not.

7

Any script experts able to craft a simple post observation script that reboots the pi once every user chosen interval AND also check that no observation is occurring in the next 5 minutes (to allow for the full boot sequence)?

–Roy
K3RLD

8

Have you checked that you have NTP activated, it keeps the clock in sync https://raspberrytips.com/time-sync-raspberry-pi/

9

Yes NTP has been activated all the time.

10

This is a known bug of RTLSDR that we investigate in https://gitlab.com/librespacefoundation/satnogs/satnogs-client/-/issues/415. Unfortunately currently the only workaround is reboot.

1 Like
11

I use an Airspy and since some days most of the observations are black here too.I did an update 3 days ago. Only change is that the Auto Scheduler is inactive (I did not activate Auto Scheduler! That happened after an update a few weeks ago). I can vet the failed observations “good” and they stay “good”, but if I vet them “bad” they get back to “failed” even if there is a seemingly good waterfall with correct signal and audio. See https://network.satnogs.org/observations/3606311/

12

Yes I’m getting failed observations which have signals in them, looks like the bug Fredy has mentioned.

13

I think we confuse two separate cases here, please correct me if I’m wrong.

  1. Observations have no artifacts (audio or waterfall or data) uploaded and in the client logs (sudo journalctl -f -u satnogs-client.service) there is this message: ERROR - Observer job lock acquiring timed out.. This case is related with the rtlsdr issue that I linked previously.

  2. Observations have artifacts (at least audio) and are marked as failed. This case is result of the automated check of audio file duration against the scheduled duration. If the difference between them is more than 1 minute then the observation is marked failed, which may be changed if waterfall is uploaded and vetted. The difference between scheduled and audio durations may come from the initialization of the sdr or other issues like lost samples, we need to investigate more these cases and adapt the automated checks, also the vetting process is still under discussion and hopefully soon we are going to move forward with it too.

2 Likes
14

Thank you @fredy yes my case is the second one, I have waterfall and audio with failed status that can be updated to green if there’s a signal but when I update to red (as the observation has not failed) the failed black status remains.

1 Like
15

A workaround for this is to hit u to make it unknown and then ‘b’ for bad.

2 Likes