Satnogs scheduler error

Anyone else have a scheduler error which look like “Temporary failure in name resolution” for ‘network.satnogs.org’.

Started happening 7 hours ago for me on both my stations.

Logs:

@raspberrypi:~ $ /home/pi/satnogs-auto-scheduler/env/bin/python /home/pi/satnogs-auto-scheduler/schedule_single_station.py -s 1522 -d 1.2 -r 40 -T
2020-05-14 07:25:12,600 - root - INFO - Requesting information for ground station 1522
2020-05-14 07:25:14,067 - root - INFO - Ground station information retrieved!
2020-05-14 07:25:14,069 - root - INFO - Updating transmitters and TLEs for station
2020-05-14 07:25:14,074 - root - INFO - Fetching transmitter information from DB.
2020-05-14 07:25:22,729 - root - INFO - Transmitters received!
2020-05-14 07:25:22,802 - root - INFO - Transmitters filtered based on ground station capability.
2020-05-14 07:25:22,804 - root - INFO - Fetching transmitter information from DB.
2020-05-14 07:25:30,393 - root - INFO - Transmitters received!
2020-05-14 07:25:30,464 - root - INFO - Transmitters filtered based on ground station capability.
2020-05-14 07:25:30,467 - root - INFO - Fetching satellite information from DB.
2020-05-14 07:25:33,749 - root - INFO - Satellites received!
2020-05-14 07:25:33,792 - root - INFO - Requesting transmitter success rates.
Traceback (most recent call last):
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/urllib3/connection.py”, line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/urllib3/util/connection.py”, line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File “/usr/lib/python3.7/socket.py”, line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 677, in urlopen
chunked=chunked,
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 381, in _make_request
self._validate_conn(conn)
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 976, in _validate_conn
conn.connect()
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/urllib3/connection.py”, line 308, in connect
conn = self._new_conn()
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/urllib3/connection.py”, line 172, in _new_conn
self, “Failed to establish a new connection: %s” % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x75992430>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/requests/adapters.py”, line 449, in send
timeout=timeout
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 725, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/urllib3/util/retry.py”, line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=‘network.satnogs.org’, port=443): Max retries exceeded with url: /api/transmitters/?page=6 (Caused by NewConnectionError(’<urllib3.connection.HTTPSConnection object at 0x75992430>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/pi/satnogs-auto-scheduler/schedule_single_station.py”, line 319, in
main()
File “/home/pi/satnogs-auto-scheduler/schedule_single_station.py”, line 172, in main
cache.update()
File “/home/pi/satnogs-auto-scheduler/cache.py”, line 84, in update
transmitters_stats = get_transmitter_stats()
File “/home/pi/satnogs-auto-scheduler/satnogs_client.py”, line 64, in get_transmitter_stats
transmitters = get_paginated_endpoint(’{}/api/transmitters/’.format(settings.NETWORK_BASE_URL))
File “/home/pi/satnogs-auto-scheduler/satnogs_client.py”, line 18, in get_paginated_endpoint
r = requests.get(url=next_page_url)
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/requests/api.py”, line 76, in get
return request(‘get’, url, params=params, **kwargs)
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/requests/api.py”, line 61, in request
return session.request(method=method, url=url, **kwargs)
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/requests/sessions.py”, line 530, in request
resp = self.send(prep, **send_kwargs)
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/requests/sessions.py”, line 665, in send
history = [resp for resp in gen] if allow_redirects else []
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/requests/sessions.py”, line 665, in
history = [resp for resp in gen] if allow_redirects else []
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/requests/sessions.py”, line 245, in resolve_redirects
**adapter_kwargs
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/requests/sessions.py”, line 643, in send
r = adapter.send(request, **kwargs)
File “/home/pi/satnogs-auto-scheduler/env/lib/python3.7/site-packages/requests/adapters.py”, line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host=‘network.satnogs.org’, port=443): Max retries exceeded with url: /api/transmitters/?page=6 (Caused by NewConnectionError(’<urllib3.connection.HTTPSConnection object at 0x75992430>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’))

1 Like

Yep. Seen it before. Most recent:

May 14 17:23:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Post data started
May 14 17:24:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Trying to GET observation jobs from the network
May 14 17:25:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Trying to GET observation jobs from the network
May 14 17:25:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Post data started
May 14 17:26:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Trying to GET observation jobs from the network
May 14 17:27:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Trying to GET observation jobs from the network
May 14 17:27:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Post data started
May 14 17:28:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Trying to GET observation jobs from the network
May 14 17:29:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Trying to GET observation jobs from the network
May 14 17:29:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Post data started
May 14 17:29:09 pinog satnogs-client[312]: apscheduler.executors.default - ERROR - Job "get_jobs (trigger: interval[0:01:00], next run at: 2020-05-14 17:30:09 UTC)" raised an exception
May 14 17:29:09 pinog satnogs-client[312]: Traceback (most recent call last):
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
May 14 17:29:09 pinog satnogs-client[312]:     (self._dns_host, self.port), self.timeout, **extra_kw)
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 57, in create_connection
May 14 17:29:09 pinog satnogs-client[312]:     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
May 14 17:29:09 pinog satnogs-client[312]:     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
May 14 17:29:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 14 17:29:09 pinog satnogs-client[312]: During handling of the above exception, another exception occurred:
May 14 17:29:09 pinog satnogs-client[312]: Traceback (most recent call last):
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
May 14 17:29:09 pinog satnogs-client[312]:     chunked=chunked)
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
May 14 17:29:09 pinog satnogs-client[312]:     self._validate_conn(conn)
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
May 14 17:29:09 pinog satnogs-client[312]:     conn.connect()
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect
May 14 17:29:09 pinog satnogs-client[312]:     conn = self._new_conn()
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
May 14 17:29:09 pinog satnogs-client[312]:     self, "Failed to establish a new connection: %s" % e)
May 14 17:29:09 pinog satnogs-client[312]: urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0xa2ac4d50>: Failed to establish a new connection: [E
rrno -3] Temporary failure in name resolution
May 14 17:29:09 pinog satnogs-client[312]: During handling of the above exception, another exception occurred:
May 14 17:29:09 pinog satnogs-client[312]: Traceback (most recent call last):
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
May 14 17:29:09 pinog satnogs-client[312]:     timeout=timeout
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
May 14 17:29:09 pinog satnogs-client[312]:     _stacktrace=sys.exc_info()[2])
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
May 14 17:29:09 pinog satnogs-client[312]:     raise MaxRetryError(_pool, url, error or ResponseError(cause))
May 14 17:29:09 pinog satnogs-client[312]: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='network.satnogs.org', port=443): Max retries exceeded with url: /api/jobs/?ground_stat
ion=1450&lat=52.212&lon=-0.904&alt=112 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0xa2ac4d50>: Failed to establish a new connection: [Errno -3] Tem
porary failure in name resolution'))
May 14 17:29:09 pinog satnogs-client[312]: During handling of the above exception, another exception occurred:
May 14 17:29:09 pinog satnogs-client[312]: Traceback (most recent call last):
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/apscheduler/executors/base.py", line 125, in run_job
May 14 17:29:09 pinog satnogs-client[312]:     retval = job.func(*job.args, **job.kwargs)
May 14 17:29:09 pinog satnogs-client[312]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 160, in get_jobs
May 14 17:29:09 pinog satnogs-client[312]:     timeout=45)
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
May 14 17:29:09 pinog satnogs-client[312]:     return request('get', url, params=params, **kwargs)
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
May 14 17:29:09 pinog satnogs-client[312]:     return session.request(method=method, url=url, **kwargs)
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
May 14 17:29:09 pinog satnogs-client[312]:     resp = self.send(prep, **send_kwargs)
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
May 14 17:29:09 pinog satnogs-client[312]:     r = adapter.send(request, **kwargs)
May 14 17:29:09 pinog satnogs-client[312]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
May 14 17:29:09 pinog satnogs-client[312]:     raise ConnectionError(e, request=request)
May 14 17:29:09 pinog satnogs-client[312]: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='network.satnogs.org', port=443): Max retries exceeded with url: /api/jobs/?ground_station=1450&lat=52.212&lon=-0.904&alt=112 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0xa2ac4d50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
May 14 17:30:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Trying to GET observation jobs from the network
May 14 17:31:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Trying to GET observation jobs from the network
May 14 17:31:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Post data started
May 14 17:32:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Trying to GET observation jobs from the network
May 14 17:33:09 pinog satnogs-client[312]: satnogsclient.scheduler.tasks - INFO - Trying to GET observation jobs from the network

Interesting pattern of times, but I assume this is down to when the request is done rather than when it fails:

pi@pinog:~ $ journalctl -u satnogs-client.service|grep 'socket.gaierror: \[Errno -3\] Temporary failure in name resolution'
May 09 12:32:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 09 13:36:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 09 13:38:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 09 16:15:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 09 18:44:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 11 09:30:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 11 15:11:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 11 16:16:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 11 19:23:14 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 11 23:16:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 12 00:45:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 12 08:27:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 12 09:38:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 12 12:04:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 12 15:18:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 12 16:17:14 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 13 01:05:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 13 04:04:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 13 05:41:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 13 05:43:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 13 05:44:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 13 06:17:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 13 06:31:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 13 16:44:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 13 19:27:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 13 21:32:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 13 22:45:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 13 22:52:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 14 03:24:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 14 03:26:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 14 03:27:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 14 03:30:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 14 03:31:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 14 03:50:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 14 04:12:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 14 08:27:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution
May 14 17:29:09 pinog satnogs-client[312]: socket.gaierror: [Errno -3] Temporary failure in name resolution

It seemed to fix itself after a day. Working fine for a while now.
Maybe a network issue.

I have the same issue and only after reboot it is fixed - but it doesn’t seem to be satnogs specific, because when I curl network.satnogs.org it also has issue with name resolution.

Is it possible to add some mitigation for this in /satnogsclient/scheduler/tasks.py in get_jobs method? So when the connection error occurs it will try to cycle the internet interface or sth?