Station fails to upload

SatNOGS Software
1

After awhile my station fails to upload any more data to satnogs. Sometimes after a few hours it starts working again, other times a restart of the satnogs client is required, and other times it fills up /tmp and needs a full reboot. Anyone got any ideas on how to troubleshoot this best. It’s really frustrating…

satnogs-client-ansible-202002272333, satnogs-client-1.1, gr-satnogs-2.0.1-1

Mar 25 22:22:16 70 systemd[1]: Started SatNOGS client.
Mar 25 22:24:31 70 satnogs-client[23125]: 2020-03-25 22:24:31,439 - satnogsclient.scheduler.tasks - ERROR - Bad status code: 400
Mar 25 22:25:18 70 satnogs-client[23125]: 2020-03-25 22:25:18,087 - apscheduler.executors.default - ERROR - Job "post_data (trigger: interval[0:02:00], next run at: 2020-03-25 22:26:19 UTC)" raised an exception
Mar 25 22:25:18 70 satnogs-client[23125]: Traceback (most recent call last):
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
Mar 25 22:25:18 70 satnogs-client[23125]:     chunked=chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
Mar 25 22:25:18 70 satnogs-client[23125]:     conn.request(method, url, **httplib_request_kw)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1244, in request
Mar 25 22:25:18 70 satnogs-client[23125]:     self._send_request(method, url, body, headers, encode_chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1290, in _send_request
Mar 25 22:25:18 70 satnogs-client[23125]:     self.endheaders(body, encode_chunked=encode_chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1239, in endheaders
Mar 25 22:25:18 70 satnogs-client[23125]:     self._send_output(message_body, encode_chunked=encode_chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1065, in _send_output
Mar 25 22:25:18 70 satnogs-client[23125]:     self.send(chunk)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 987, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     self.sock.sendall(data)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/ssl.py", line 1015, in sendall
Mar 25 22:25:18 70 satnogs-client[23125]:     v = self.send(byte_view[count:])
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/ssl.py", line 984, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     return self._sslobj.write(data)
Mar 25 22:25:18 70 satnogs-client[23125]: socket.timeout: The write operation timed out
Mar 25 22:25:18 70 satnogs-client[23125]: During handling of the above exception, another exception occurred:
Mar 25 22:25:18 70 satnogs-client[23125]: Traceback (most recent call last):
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     timeout=timeout
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
Mar 25 22:25:18 70 satnogs-client[23125]:     _stacktrace=sys.exc_info()[2])
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 367, in increment
Mar 25 22:25:18 70 satnogs-client[23125]:     raise six.reraise(type(error), error, _stacktrace)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/six.py", line 692, in reraise
Mar 25 22:25:18 70 satnogs-client[23125]:     raise value.with_traceback(tb)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
Mar 25 22:25:18 70 satnogs-client[23125]:     chunked=chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
Mar 25 22:25:18 70 satnogs-client[23125]:     conn.request(method, url, **httplib_request_kw)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1244, in request
Mar 25 22:25:18 70 satnogs-client[23125]:     self._send_request(method, url, body, headers, encode_chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1290, in _send_request
Mar 25 22:25:18 70 satnogs-client[23125]:     self.endheaders(body, encode_chunked=encode_chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1239, in endheaders
Mar 25 22:25:18 70 satnogs-client[23125]:     self._send_output(message_body, encode_chunked=encode_chunked)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 1065, in _send_output
Mar 25 22:25:18 70 satnogs-client[23125]:     self.send(chunk)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/http/client.py", line 987, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     self.sock.sendall(data)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/ssl.py", line 1015, in sendall
Mar 25 22:25:18 70 satnogs-client[23125]:     v = self.send(byte_view[count:])
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3.7/ssl.py", line 984, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     return self._sslobj.write(data)
Mar 25 22:25:18 70 satnogs-client[23125]: urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out'))
Mar 25 22:25:18 70 satnogs-client[23125]: During handling of the above exception, another exception occurred:
Mar 25 22:25:18 70 satnogs-client[23125]: Traceback (most recent call last):
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/apscheduler/executors/base.py", line 125, in run_job
Mar 25 22:25:18 70 satnogs-client[23125]:     retval = job.func(*job.args, **job.kwargs)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 126, in post_data
Mar 25 22:25:18 70 satnogs-client[23125]:     timeout=45)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 131, in put
Mar 25 22:25:18 70 satnogs-client[23125]:     return request('put', url, data=data, **kwargs)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
Mar 25 22:25:18 70 satnogs-client[23125]:     return session.request(method=method, url=url, **kwargs)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
Mar 25 22:25:18 70 satnogs-client[23125]:     resp = self.send(prep, **send_kwargs)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     r = adapter.send(request, **kwargs)
Mar 25 22:25:18 70 satnogs-client[23125]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
Mar 25 22:25:18 70 satnogs-client[23125]:     raise ConnectionError(err, request=request)
Mar 25 22:25:18 70 satnogs-client[23125]: requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out'))
2

Please update to the latest version and let us know if you still observe this issue.

3

Updated to latest and I’m getting bad request now from the API. There is about 10 observations that are in the tmp file waiting to uploaded.

Mar 26 04:10:49 70 satnogs-client[31680]: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://network.satnogs.org/api/observations/1921740/
Mar 26 04:10:49 70 satnogs-client[31680]: During handling of the above exception, another exception occurred:
Mar 26 04:10:49 70 satnogs-client[31680]: Traceback (most recent call last):
Mar 26 04:10:49 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 1034, in emit
Mar 26 04:10:49 70 satnogs-client[31680]:     msg = self.format(record)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 880, in format
Mar 26 04:10:50 70 satnogs-client[31680]:     return fmt.format(record)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 619, in format
Mar 26 04:10:50 70 satnogs-client[31680]:     record.message = record.getMessage()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 380, in getMessage
Mar 26 04:10:50 70 satnogs-client[31680]:     msg = msg % self.args
Mar 26 04:10:50 70 satnogs-client[31680]: TypeError: %i format: a number is required, not str
Mar 26 04:10:50 70 satnogs-client[31680]: Call stack:
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
Mar 26 04:10:50 70 satnogs-client[31680]:     self._bootstrap_inner()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
Mar 26 04:10:50 70 satnogs-client[31680]:     self.run()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/threading.py", line 865, in run
Mar 26 04:10:50 70 satnogs-client[31680]:     self._target(*self._args, **self._kwargs)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/concurrent/futures/thread.py", line 80, in _worker
Mar 26 04:10:50 70 satnogs-client[31680]:     work_item.run()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
Mar 26 04:10:50 70 satnogs-client[31680]:     result = self.fn(*self.args, **self.kwargs)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3/dist-packages/apscheduler/executors/base.py", line 125, in run_job
Mar 26 04:10:50 70 satnogs-client[31680]:     retval = job.func(*job.args, **job.kwargs)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 161, in post_data
Mar 26 04:10:50 70 satnogs-client[31680]:     'response status code: %s', fil, observation_id, response.status_code)
Mar 26 04:10:50 70 satnogs-client[31680]: Message: 'Upload of %s for observation %i failed, response status code: %s'
Mar 26 04:10:50 70 satnogs-client[31680]: Arguments: ('waterfall_1921740_2020-03-25T07-06-39.png', '1921740', 400)
Mar 26 04:10:50 70 satnogs-client[31680]: --- Logging error ---
Mar 26 04:10:50 70 satnogs-client[31680]: Traceback (most recent call last):
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 138, in post_data
Mar 26 04:10:50 70 satnogs-client[31680]:     response.raise_for_status()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
Mar 26 04:10:50 70 satnogs-client[31680]:     raise HTTPError(http_error_msg, response=self)
Mar 26 04:10:50 70 satnogs-client[31680]: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://network.satnogs.org/api/observations/1921740/
Mar 26 04:10:50 70 satnogs-client[31680]: During handling of the above exception, another exception occurred:
Mar 26 04:10:50 70 satnogs-client[31680]: Traceback (most recent call last):
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 1034, in emit
Mar 26 04:10:50 70 satnogs-client[31680]:     msg = self.format(record)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 880, in format
Mar 26 04:10:50 70 satnogs-client[31680]:     return fmt.format(record)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 619, in format
Mar 26 04:10:50 70 satnogs-client[31680]:     record.message = record.getMessage()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 380, in getMessage
Mar 26 04:10:50 70 satnogs-client[31680]:     msg = msg % self.args
Mar 26 04:10:50 70 satnogs-client[31680]: TypeError: %i format: a number is required, not str
Mar 26 04:10:50 70 satnogs-client[31680]: Call stack:
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
Mar 26 04:10:50 70 satnogs-client[31680]:     self._bootstrap_inner()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
Mar 26 04:10:50 70 satnogs-client[31680]:     self.run()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/threading.py", line 865, in run
Mar 26 04:10:50 70 satnogs-client[31680]:     self._target(*self._args, **self._kwargs)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/concurrent/futures/thread.py", line 80, in _worker
Mar 26 04:10:50 70 satnogs-client[31680]:     work_item.run()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
Mar 26 04:10:50 70 satnogs-client[31680]:     result = self.fn(*self.args, **self.kwargs)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3/dist-packages/apscheduler/executors/base.py", line 125, in run_job
Mar 26 04:10:50 70 satnogs-client[31680]:     retval = job.func(*job.args, **job.kwargs)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 161, in post_data
Mar 26 04:10:50 70 satnogs-client[31680]:     'response status code: %s', fil, observation_id, response.status_code)
Mar 26 04:10:50 70 satnogs-client[31680]: Message: 'Upload of %s for observation %i failed, response status code: %s'
Mar 26 04:10:50 70 satnogs-client[31680]: Arguments: ('waterfall_1921740_2020-03-25T07-06-39.png', '1921740', 400)
Mar 26 04:10:50 70 satnogs-client[31680]: --- Logging error ---
Mar 26 04:10:50 70 satnogs-client[31680]: Traceback (most recent call last):
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 138, in post_data
Mar 26 04:10:50 70 satnogs-client[31680]:     response.raise_for_status()
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
Mar 26 04:10:50 70 satnogs-client[31680]:     raise HTTPError(http_error_msg, response=self)
Mar 26 04:10:50 70 satnogs-client[31680]: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://network.satnogs.org/api/observations/1921740/
Mar 26 04:10:50 70 satnogs-client[31680]: During handling of the above exception, another exception occurred:
Mar 26 04:10:50 70 satnogs-client[31680]: Traceback (most recent call last):
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 1034, in emit
Mar 26 04:10:50 70 satnogs-client[31680]:     msg = self.format(record)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 880, in format
Mar 26 04:10:50 70 satnogs-client[31680]:     return fmt.format(record)
Mar 26 04:10:50 70 satnogs-client[31680]:   File "/usr/lib/python3.7/logging/__init__.py", line 619, in format
Mar 26 04:10:50 70 satnogs-client[31680]:     record.message = record.getMessage()

I could reboot it and loose the data?

4

I moved the data to the sd card and restarted the client. I can upload it somewhere if someone needs it.

5

if possible can you share these files. After moving the files and reboot does it work fine?

6

Hi, Did you ever resolve this issue? It’s happening to my station now. I would appreciate any help.

1 Like
7

i am using pacsat on windows to upload telemetry automatic to satnogs. i do have rx well mirsat 1 telemetry with arrl lindenblad …server upload pacsat to satnogs on ,can not find my telemetry on satnogs??? problems …what to do ,do not use pi , dont have it
it suppose to work automatic from pacsat gnd station

8

Not sure if your problem is now solved. However I can see in DB stats by filtering with your callsign that there are uploaded data.

You can also check them in these links (just make sure you have logged in before in db.satnogs.org):
https://db.satnogs.org/api/telemetry/?observer=on7ndr-JO11sf
https://db.satnogs.org/api/telemetry/?observer=ON7NDR-JO11sj

9

@on7ndr ping me if you are still having issues as I am the author of the Pacsat Ground Station. Make sure you are on the latest version 0.44. An earlier version did not retry sending telemetry if there was an issue with the internet connection.

Chris
G0KLA

10

@chrisethompson chris , i have decoded mirsat telemetry with a homemade lindenblad arrl antenna and hsdr software
since i have no luck with uploads with pacsat , i installed satnogs client on raspberry pi and that is working . but pacsat data from 8100 kissserver gets in log a error ?? so no upload to satnogs … pacsat software is configurated to “upload to sever” kisslog on, etc my lattitude is correct and locator also. maybe the station must be active via pi also ? i don’t know. i can only tell you , that i had telemetry on the pacsat groundstation with voltage antenna parameters and so on , but my data has never reach satnogs db … will try again via hssdr , rtlsdr , vb cable,and hs_soundmodem set to g3ruh 9600 fsk , confirm that the soundmodem was decoding the telemetry and callsign from 3b8mir…
in attachment the confirmed log from mirsat and NO upload to satnogs …see log 73…many thanks
PacSatGround20210726.txt (31.6 KB)

11

@on7ndr Nico,

Thanks for the log. It looks like you modified the upload URL. Probably when you were trying to get things working. You are connecting fine but getting return code 301 from the Satnogs DB. Your upload URL is https://db.satnogs.org/api/telemetry/on7ndr. It should be just https://db.satnogs.org/api/telemetry/

You get reset this on the Spacecraft > Mir-sat-1 screen where it says “Telem Server”.

73
Chris
G0KLA

12

@chrisethompson .
Thanks for your gently reply on my problem.
But i have never changed the server URL chris , this was automatic generated i think by the program?
Sorry if i’m wrong. did i realy change’d that adress.
it is actually the first time with your note’s above note that i see this adress appear.
i had to search :slight_smile:

i have only given my call and coordinates , for the rest i do have not changed server settings from db satnogs.

i have done reset the adress chris " telem server" whithout on7ndr…
gl 73 de on7ndr nico

1 Like
13

Great @on7ndr Nico. Let me know if that does not work.

73
Chris