Station fails to upload

After awhile my station fails to upload any more data to satnogs. Sometimes after a few hours it starts working again, other times a restart of the satnogs client is required, and other times it fills up /tmp and needs a full reboot. Anyone got any ideas on how to troubleshoot this best. It’s really frustrating…

satnogs-client-ansible-202002272333, satnogs-client-1.1, gr-satnogs-2.0.1-1

Mar 25 22:22:16 70 systemd[1]: Started SatNOGS client.
Mar 25 22:24:31 70 satnogs-client[23125]: 2020-03-25 22:24:31,439 - satnogsclient.scheduler.tasks - ERROR - Bad status code: 400
Mar 25 22:25:18 70 satnogs-client[23125]: 2020-03-25 22:25:18,087 - apscheduler.executors.default - ERROR - Job "post_data (trigger: interval[0:02:00], next run at: 2020-03-25 22:26:19 UTC)" raised an exception
Mar 25 22:25:18 70 satnogs-client[23125]: Traceback (most recent call last):
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
Mar 25 22:25:18 70 satnogs-client[23125]:     chunked=chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
Mar 25 22:25:18 70 satnogs-client[23125]:     conn.request(method, url, **httplib_request_kw)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1244, in request
Mar 25 22:25:18 70 satnogs-client[23125]:     self._send_request(method, url, body, headers, encode_chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1290, in _send_request
Mar 25 22:25:18 70 satnogs-client[23125]:     self.endheaders(body, encode_chunked=encode_chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1239, in endheaders
Mar 25 22:25:18 70 satnogs-client[23125]:     self._send_output(message_body, encode_chunked=encode_chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1065, in _send_output
Mar 25 22:25:18 70 satnogs-client[23125]:     self.send(chunk)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 987, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     self.sock.sendall(data)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/ssl.py", line 1015, in sendall
Mar 25 22:25:18 70 satnogs-client[23125]:     v = self.send(byte_view[count:])
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/ssl.py", line 984, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     return self._sslobj.write(data)
Mar 25 22:25:18 70 satnogs-client[23125]: socket.timeout: The write operation timed out
Mar 25 22:25:18 70 satnogs-client[23125]: During handling of the above exception, another exception occurred:
Mar 25 22:25:18 70 satnogs-client[23125]: Traceback (most recent call last):
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     timeout=timeout
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
Mar 25 22:25:18 70 satnogs-client[23125]:     _stacktrace=sys.exc_info()[2])
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 367, in increment
Mar 25 22:25:18 70 satnogs-client[23125]:     raise six.reraise(type(error), error, _stacktrace)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/six.py", line 692, in reraise
Mar 25 22:25:18 70 satnogs-client[23125]:     raise value.with_traceback(tb)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
Mar 25 22:25:18 70 satnogs-client[23125]:     chunked=chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
Mar 25 22:25:18 70 satnogs-client[23125]:     conn.request(method, url, **httplib_request_kw)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1244, in request
Mar 25 22:25:18 70 satnogs-client[23125]:     self._send_request(method, url, body, headers, encode_chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1290, in _send_request
Mar 25 22:25:18 70 satnogs-client[23125]:     self.endheaders(body, encode_chunked=encode_chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1239, in endheaders
Mar 25 22:25:18 70 satnogs-client[23125]:     self._send_output(message_body, encode_chunked=encode_chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1065, in _send_output
Mar 25 22:25:18 70 satnogs-client[23125]:     self.send(chunk)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 987, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     self.sock.sendall(data)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/ssl.py", line 1015, in sendall
Mar 25 22:25:18 70 satnogs-client[23125]:     v = self.send(byte_view[count:])
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/ssl.py", line 984, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     return self._sslobj.write(data)
Mar 25 22:25:18 70 satnogs-client[23125]: urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out'))
Mar 25 22:25:18 70 satnogs-client[23125]: During handling of the above exception, another exception occurred:
Mar 25 22:25:18 70 satnogs-client[23125]: Traceback (most recent call last):
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/apscheduler/executors/base.py", line 125, in run_job
Mar 25 22:25:18 70 satnogs-client[23125]:     retval = job.func(*job.args, **job.kwargs)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 126, in post_data
Mar 25 22:25:18 70 satnogs-client[23125]:     timeout=45)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 131, in put
Mar 25 22:25:18 70 satnogs-client[23125]:     return request('put', url, data=data, **kwargs)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
Mar 25 22:25:18 70 satnogs-client[23125]:     return session.request(method=method, url=url, **kwargs)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
Mar 25 22:25:18 70 satnogs-client[23125]:     resp = self.send(prep, **send_kwargs)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     r = adapter.send(request, **kwargs)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     raise ConnectionError(err, request=request)
Mar 25 22:25:18 70 satnogs-client[23125]: requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out'))

Please update to the latest version and let us know if you still observe this issue.

Updated to latest and I’m getting bad request now from the API. There is about 10 observations that are in the tmp file waiting to uploaded.

Mar 26 04:10:49 70 satnogs-client[31680]: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://network.satnogs.org/api/observations/1921740/
Mar 26 04:10:49 70 satnogs-client[31680]: During handling of the above exception, another exception occurred:
Mar 26 04:10:49 70 satnogs-client[31680]: Traceback (most recent call last):
Mar 26 04:10:49 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 1034, in emit
Mar 26 04:10:49 70 satnogs-client[31680]:     msg = self.format(record)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 880, in format
Mar 26 04:10:50 70 satnogs-client[31680]:     return fmt.format(record)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 619, in format
Mar 26 04:10:50 70 satnogs-client[31680]:     record.message = record.getMessage()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 380, in getMessage
Mar 26 04:10:50 70 satnogs-client[31680]:     msg = msg % self.args
Mar 26 04:10:50 70 satnogs-client[31680]: TypeError: %i format: a number is required, not str
Mar 26 04:10:50 70 satnogs-client[31680]: Call stack:
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
Mar 26 04:10:50 70 satnogs-client[31680]:     self._bootstrap_inner()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
Mar 26 04:10:50 70 satnogs-client[31680]:     self.run()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/threading.py", line 865, in run
Mar 26 04:10:50 70 satnogs-client[31680]:     self._target(*self._args, **self._kwargs)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/concurrent/futures/thread.py", line 80, in _worker
Mar 26 04:10:50 70 satnogs-client[31680]:     work_item.run()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
Mar 26 04:10:50 70 satnogs-client[31680]:     result = self.fn(*self.args, **self.kwargs)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3/dist-packages/apscheduler/executors/base.py", line 125, in run_job
Mar 26 04:10:50 70 satnogs-client[31680]:     retval = job.func(*job.args, **job.kwargs)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 161, in post_data
Mar 26 04:10:50 70 satnogs-client[31680]:     'response status code: %s', fil, observation_id, response.status_code)
Mar 26 04:10:50 70 satnogs-client[31680]: Message: 'Upload of %s for observation %i failed, response status code: %s'
Mar 26 04:10:50 70 satnogs-client[31680]: Arguments: ('waterfall_1921740_2020-03-25T07-06-39.png', '1921740', 400)
Mar 26 04:10:50 70 satnogs-client[31680]: --- Logging error ---
Mar 26 04:10:50 70 satnogs-client[31680]: Traceback (most recent call last):
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 138, in post_data
Mar 26 04:10:50 70 satnogs-client[31680]:     response.raise_for_status()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
Mar 26 04:10:50 70 satnogs-client[31680]:     raise HTTPError(http_error_msg, response=self)
Mar 26 04:10:50 70 satnogs-client[31680]: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://network.satnogs.org/api/observations/1921740/
Mar 26 04:10:50 70 satnogs-client[31680]: During handling of the above exception, another exception occurred:
Mar 26 04:10:50 70 satnogs-client[31680]: Traceback (most recent call last):
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 1034, in emit
Mar 26 04:10:50 70 satnogs-client[31680]:     msg = self.format(record)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 880, in format
Mar 26 04:10:50 70 satnogs-client[31680]:     return fmt.format(record)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 619, in format
Mar 26 04:10:50 70 satnogs-client[31680]:     record.message = record.getMessage()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 380, in getMessage
Mar 26 04:10:50 70 satnogs-client[31680]:     msg = msg % self.args
Mar 26 04:10:50 70 satnogs-client[31680]: TypeError: %i format: a number is required, not str
Mar 26 04:10:50 70 satnogs-client[31680]: Call stack:
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
Mar 26 04:10:50 70 satnogs-client[31680]:     self._bootstrap_inner()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
Mar 26 04:10:50 70 satnogs-client[31680]:     self.run()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/threading.py", line 865, in run
Mar 26 04:10:50 70 satnogs-client[31680]:     self._target(*self._args, **self._kwargs)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/concurrent/futures/thread.py", line 80, in _worker
Mar 26 04:10:50 70 satnogs-client[31680]:     work_item.run()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
Mar 26 04:10:50 70 satnogs-client[31680]:     result = self.fn(*self.args, **self.kwargs)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3/dist-packages/apscheduler/executors/base.py", line 125, in run_job
Mar 26 04:10:50 70 satnogs-client[31680]:     retval = job.func(*job.args, **job.kwargs)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 161, in post_data
Mar 26 04:10:50 70 satnogs-client[31680]:     'response status code: %s', fil, observation_id, response.status_code)
Mar 26 04:10:50 70 satnogs-client[31680]: Message: 'Upload of %s for observation %i failed, response status code: %s'
Mar 26 04:10:50 70 satnogs-client[31680]: Arguments: ('waterfall_1921740_2020-03-25T07-06-39.png', '1921740', 400)
Mar 26 04:10:50 70 satnogs-client[31680]: --- Logging error ---
Mar 26 04:10:50 70 satnogs-client[31680]: Traceback (most recent call last):
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 138, in post_data
Mar 26 04:10:50 70 satnogs-client[31680]:     response.raise_for_status()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
Mar 26 04:10:50 70 satnogs-client[31680]:     raise HTTPError(http_error_msg, response=self)
Mar 26 04:10:50 70 satnogs-client[31680]: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://network.satnogs.org/api/observations/1921740/
Mar 26 04:10:50 70 satnogs-client[31680]: During handling of the above exception, another exception occurred:
Mar 26 04:10:50 70 satnogs-client[31680]: Traceback (most recent call last):
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 1034, in emit
Mar 26 04:10:50 70 satnogs-client[31680]:     msg = self.format(record)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 880, in format
Mar 26 04:10:50 70 satnogs-client[31680]:     return fmt.format(record)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 619, in format
Mar 26 04:10:50 70 satnogs-client[31680]:     record.message = record.getMessage()

I could reboot it and loose the data?

I moved the data to the sd card and restarted the client. I can upload it somewhere if someone needs it.

if possible can you share these files. After moving the files and reboot does it work fine?