SatNOGS Network Downtime 2020-09-23

SatNOGS Software
1

Today, 2020-09-23, there will be downtime of SatNOGS Network between 07:00 - 11:00 UTC due to maintenance reasons. Downtime duration is estimated to be less than one hour.

Keep watching this thread for more updates.

Thank you for your patience and sorry for any inconvenience.

3 Likes
2

Due to some unexpected events, downtime will be move between 10:30 - 14:30 UTC. Still is not expected to be longer than 1 hour.

1 Like
3

Unfortunately it took us a little more to fix some other issues, so downtime is going to start in the next couple of minutes.

1 Like
4

Downtime is over, again thank you for your patience.

Please report here if you notice anything out the ordinary.

1 Like
5

Hi Fredy,

it may not be related but my last successful upload was https://network.satnogs.org/observations/2884426/ (16:38 UTC) since then (even a couple of minutes ago) all my records show no data. I rebooted my station just now.

Thanks
Ansi

1 Like
6

Hi!
I also confirm the abnormality, from https://network.satnogs.org/observations/2884559/ (16:59 UTC). After the changes, it does not load data and automatically marks them as failed.
Greetingsā€¦

2 Likes
7

All my obs fail as well since over an hour. Station is up to date and has just been rebooted.

ep 23 21:44:43 satpi satnogs-client[332]: apscheduler.executors.default - ERROR - Job "spawn_observer (trigger: date[2020-09-23 20:44:43 UTC], next run at: 2020-09-23 20:44:43 UTC)" raised an exception
Sep 23 21:44:43 satpi satnogs-client[332]: Traceback (most recent call last):
Sep 23 21:44:43 satpi satnogs-client[332]:   File "/usr/lib/python3/dist-packages/apscheduler/executors/base.py", line 125, in run_job
Sep 23 21:44:43 satpi satnogs-client[332]:     retval = job.func(*job.args, **job.kwargs)
Sep 23 21:44:43 satpi satnogs-client[332]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 26, in spawn_observer
Sep 23 21:44:43 satpi satnogs-client[332]:     tle = {'tle0': obj['tle0'], 'tle1': obj['tle1'], 'tle2': obj['tle2']}
Sep 23 21:44:43 satpi satnogs-client[332]: KeyError: 'tle0'

Seems to affect many or all stations: https://network.satnogs.org/observations/?future=0&norad=&observer=&station=&start=&end=&page=3

1 Like
8

Could it be that a field returned from the server has been renamed from tle0 to tle_line_0 ?

Sep 23 22:29:30 satpi satnogs-client[332]: satnogsclient.scheduler.tasks - DEBUG - Observation obj: {'id': 2881129, 'start': '2020-09-23T21:31:47Z', 'end': '2020-09-23T21:37:31Z', 'ground_station': 1207, 'tle_line_0': '0 AISTECHSAT-3', 'tle_line_1': '1 44103U 19018AB  20267.45394285  .00000494  00000-0  23773-4 0  9991', 'tle_line_2': '2 44103  97.3983 327.3305 0012785 196.2725 290.0558 15.23301986 82356', 'frequency': 436730000, 'mode': 'FSK AX.100 Mode 5', 'transmitter': 'TGP8nqLVquNWuYYytpbQ7A', 'baud': 9600.0}
2 Likes
9

Thanks for the reportsā€¦ a fix is on the way.

4 Likes
10

Great, thanks for your quick reaction !

2 Likes
11

The fix will be deployed in the next minutes, you can check the deployment progress in https://gitlab.com/librespacefoundation/satnogs/satnogs-network/-/pipelines.

Iā€™ve opened also an issue for adding a test about Jobs API, in order to avoid such regression in the future.

2 Likes
12

Fix has been deployed, please confirm that everything works fine now. Thank you!

5 Likes
13

Thanks, yes jobs succeed again.

2 Likes
14

It works again,
https://network.satnogs.org/observations/2885647/

Thank you very much for the very quick response.

Ansi

3 Likes