Observation 4386874: DEKART (46493) and apscheduler issue

Hello!
Regarding Observation 4386874

and sudo journalctl -f -u satnogs-client.service generate this message

juil. 07 14:50:41 cceres_team satnogs-client[516]: apscheduler.executors.default - ERROR - Job “get_jobs (trigger: interval[0:01:00], next run at: 2021-07-07 12:50:55 UTC)” raised an exception
juil. 07 14:50:41 cceres_team satnogs-client[516]: Traceback (most recent call last):
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 426, in _make_request
juil. 07 14:50:41 cceres_team satnogs-client[516]: six.raise_from(e, None)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “”, line 3, in raise_from
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 421, in _make_request
juil. 07 14:50:41 cceres_team satnogs-client[516]: httplib_response = conn.getresponse()
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/stdlib.py”, line 102, in getresponse
juil. 07 14:50:41 cceres_team satnogs-client[516]: rv = real_getresponse(self, *args, **kwargs)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/lib/python3.7/http/client.py”, line 1352, in getresponse
juil. 07 14:50:41 cceres_team satnogs-client[516]: response.begin()
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/lib/python3.7/http/client.py”, line 310, in begin
juil. 07 14:50:41 cceres_team satnogs-client[516]: version, status, reason = self._read_status()
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/lib/python3.7/http/client.py”, line 271, in _read_status
juil. 07 14:50:41 cceres_team satnogs-client[516]: line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/lib/python3.7/socket.py”, line 589, in readinto
juil. 07 14:50:41 cceres_team satnogs-client[516]: return self._sock.recv_into(b)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/lib/python3.7/ssl.py”, line 1052, in recv_into
juil. 07 14:50:41 cceres_team satnogs-client[516]: return self.read(nbytes, buffer)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/lib/python3.7/ssl.py”, line 911, in read
juil. 07 14:50:41 cceres_team satnogs-client[516]: return self._sslobj.read(len, buffer)
juil. 07 14:50:41 cceres_team satnogs-client[516]: socket.timeout: The read operation timed out
juil. 07 14:50:41 cceres_team satnogs-client[516]: During handling of the above exception, another exception occurred:
juil. 07 14:50:41 cceres_team satnogs-client[516]: Traceback (most recent call last):
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/local/lib/python3.7/dist-packages/requests/adapters.py”, line 449, in send
juil. 07 14:50:41 cceres_team satnogs-client[516]: timeout=timeout
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 727, in urlopen
juil. 07 14:50:41 cceres_team satnogs-client[516]: method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/var/lib/satnogs/lib/python3.7/site-packages/urllib3/util/retry.py”, line 410, in increment
juil. 07 14:50:41 cceres_team satnogs-client[516]: raise six.reraise(type(error), error, _stacktrace)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/var/lib/satnogs/lib/python3.7/site-packages/urllib3/packages/six.py”, line 735, in reraise
juil. 07 14:50:41 cceres_team satnogs-client[516]: raise value
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 677, in urlopen
juil. 07 14:50:41 cceres_team satnogs-client[516]: chunked=chunked,
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 428, in _make_request
juil. 07 14:50:41 cceres_team satnogs-client[516]: self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 336, in _raise_timeout
juil. 07 14:50:41 cceres_team satnogs-client[516]: self, url, “Read timed out. (read timeout=%s)” % timeout_value
juil. 07 14:50:41 cceres_team satnogs-client[516]: urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host=‘network.satnogs.org’, port=443): Read timed out. (read timeout=45)
juil. 07 14:50:41 cceres_team satnogs-client[516]: During handling of the above exception, another exception occurred:
juil. 07 14:50:41 cceres_team satnogs-client[516]: Traceback (most recent call last):
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/local/lib/python3.7/dist-packages/apscheduler/executors/base.py”, line 125, in run_job
juil. 07 14:50:41 cceres_team satnogs-client[516]: retval = job.func(*job.args, **job.kwargs)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py”, line 171, in get_jobs
juil. 07 14:50:41 cceres_team satnogs-client[516]: timeout=45)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/local/lib/python3.7/dist-packages/requests/api.py”, line 76, in get
juil. 07 14:50:41 cceres_team satnogs-client[516]: return request(‘get’, url, params=params, **kwargs)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/local/lib/python3.7/dist-packages/requests/api.py”, line 61, in request
juil. 07 14:50:41 ccere_team satnogs-client[516]: return session.request(method=method, url=url, **kwargs)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/local/lib/python3.7/dist-packages/requests/sessions.py”, line 530, in request
juil. 07 14:50:41 cceres_team satnogs-client[516]: resp = self.send(prep, **send_kwargs)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/local/lib/python3.7/dist-packages/requests/sessions.py”, line 643, in send
juil. 07 14:50:41 cceres_team satnogs-client[516]: r = adapter.send(request, **kwargs)
juil. 07 14:50:41 cceres_team satnogs-client[516]: File “/usr/local/lib/python3.7/dist-packages/requests/adapters.py”, line 529, in send
juil. 07 14:50:41 cceres_team satnogs-client[516]: raise ReadTimeout(e, request=request)
juil. 07 14:50:41 cceres_team satnogs-client[516]: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host=‘network.satnogs.org’, port=443): Read timed out. (read timeout=45)
juil. 07 14:50:55 cceres_team satnogs-client[516]: satnogsclient.scheduler.tasks - INFO - Trying to GET observation jobs from the network
juil. 07 14:50:55 cceres_team satnogs-client[516]: satnogsclient.scheduler.tasks - INFO - Post data started
juil. 07 14:51:37 cceres_team satnogs-client[516]: satnogsclient.observer.observer - INFO - Tracking stopped.
juil. 07 14:51:37 cceres_team satnogs-client[516]: satnogsclient.observer.worker - INFO - Tracking stopped.
juil. 07 14:51:40 cceres_team satnogs-client[516]: satnogsclient.observer.worker - INFO - Tracking stopped.

Is that mean satellite tracking time is son long?
Someone can help to understand this.

cheers

Or the satnogs network have had tempory connection issue?

Hi @cceres_team,

Yes, this error is caused by network connection problems when the satnogs-client can’t fetch jobs / upcoming observations. It automatically retries after some time, so as long as your observations turn out to be successful it can be ignored. In the upcoming version of satnogs-client the error logging in this case got improved, see satnogs-client@5930d1b8.

There might be several reasons for this error, e.g. bad internet connection or SatNOGS network being temporarily unavailable.

The availability status of SatNOGS Network can be checked at https://status.libre.space/, currently everything is operational.

Sincerely,
Fabian

Thanks Fabian for clarifying this kind of issue.

Sincerely,
Silla