My remote Station is OFFLINE after upgrading system packages

All of mine experienced the same issue. Did an upgrade, more by habit (than necessity) for testing units, and all three collapsed on themselves. After kicking myself for breaking the first golden rule of software (don’t update unless it’s broken), I’m finding that even fresh installs from the last released IMG are seeing issues if left to update (normally) on their own (described at the bottom)

Also, for me, the client-update started squawking about expected kernel updates which I think has more to do with something upstream with Raspbian than the SATNOGS load, but I can’t be positive on that. (32bit vs 64bit expected kernels ?)

results from uname -a:

Linux satnogspi2 4.19.102-v7+ #1295 SMP Thu Feb 6 15:43:59 GMT 2020 armv7l GNU/Linux

The “warning” speaks of expecting …-v8+ which I thought was only for 64-bit (still researching)

Newer kernel available
The currently running kernel version is 4.19.102-v7+ which is not the expected kernel version 4.19.102-v8+.
Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting.

Obviously, the recommended restart didn’t “solve” anything. Still boots back into kernel7l as it should.

looking in /boot for the images

satnogspi2 :: ~ » ls -al /boot/*.img
-rwxr-xr-x 1 root root 5432400 Feb 16 07:57 /boot/kernel7.img
-rwxr-xr-x 1 root root 5766672 Feb 16 07:57 /boot/kernel7l.img
-rwxr-xr-x 1 root root 13521408 Feb 16 07:57 /boot/kernel8.img
-rwxr-xr-x 1 root root 5152576 Feb 16 07:57 /boot/kernel.img

which of course shows all the bootloaders… This is a RasPi 3Bv1.2 (reports as armv7l under 32bit OS, actually an ARMv8 chip though), (Rev, a22082) btw (from cpuinfo)

Hardware : BCM2835
Revision : a22082
Serial : 00000000########
Model : Raspberry Pi 3 Model B Rev 1.2

Another data point:
'cause it’s the weekend, and American Football is over… I took yet another “fresh” SATNOGS install from it’s initial image (Tag version ‘2019091100’), all the way through a standard update & setup for the client.

  • In addition to having the ca-certificates issue listed here, and fixing that with the solution here,
  • it also failed in starting the satnogs-client. (failed after 10 repeats as per the journalctl logs)

snipped last entry from that log

Feb 16 08:51:56 satnogspi2 systemd[1]: Started SatNOGS client.
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: Traceback (most recent call last):
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: File “/var/lib/satnogs/bin/satnogs-client”, line 5, in
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: from satnogsclient import main
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/init.py”, line 13, in <mod
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: from satnogsclient.locator import locator
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/locator/locator.py”, line 5, i
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: import gps
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: ImportError: No module named gps
Feb 16 08:51:58 satnogspi2 systemd[1]: satnogs-client.service: Main process exited, code=exited, status=1/FAILURE
Feb 16 08:51:58 satnogspi2 systemd[1]: satnogs-client.service: Failed with result ‘exit-code’.
Feb 16 08:51:58 satnogspi2 systemd[1]: satnogs-client.service: Service RestartSec=100ms expired, scheduling restart.
Feb 16 08:51:58 satnogspi2 systemd[1]: satnogs-client.service: Scheduled restart job, restart counter is at 10.
Feb 16 08:51:58 satnogspi2 systemd[1]: Stopped SatNOGS client.
Feb 16 08:51:58 satnogspi2 systemd[1]: satnogs-client.service: Start request repeated too quickly.
Feb 16 08:51:58 satnogspi2 systemd[1]: satnogs-client.service: Failed with result ‘exit-code’.
Feb 16 08:51:58 satnogspi2 systemd[1]: Failed to start SatNOGS client.

So, I did the suggested addition of the python-gps package as mentioned above, and that seems to have addressed one of the most critical issues. It’s online now, but the logs still make me wonder if it’s actually “happy”…:

Feb 16 19:10:46 satnogspi2 systemd[1]: Started SatNOGS client.
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: 2020-02-16 19:20:56,923 - apscheduler.executors.default - ERROR - Job "get_jobs (trigger: interval[0:0
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: Traceback (most recent call last):
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/usr/lib/python2.7/dist-packages/apscheduler/executors/base.py”, line 125, in run_job
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: retval = job.func(*job.args, **job.kwargs)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/scheduler/tasks.py”, line 163
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: timeout=45)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/usr/lib/python2.7/dist-packages/requests/api.py”, line 75, in get
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: return request(‘get’, url, params=params, **kwargs)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/usr/lib/python2.7/dist-packages/requests/api.py”, line 60, in request
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: return session.request(method=method, url=url, **kwargs)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/usr/lib/python2.7/dist-packages/requests/sessions.py”, line 533, in request
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: resp = self.send(prep, **send_kwargs)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/usr/lib/python2.7/dist-packages/requests/sessions.py”, line 646, in send
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: r = adapter.send(request, **kwargs)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/usr/lib/python2.7/dist-packages/requests/adapters.py”, line 516, in send
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: raise ConnectionError(e, request=request)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: ConnectionError: HTTPSConnectionPool(host=‘network.satnogs.org’, port=443): Max retries exceeded with

So… it’s online, and “fixed” as per the issue raised by the OP, but is it actually “fixed” ?

The kernel issue is vexing, because I’m seeing this on every new image I’m provisioning, and I’m not sure what to attribute it to…
also, when actually installing the package “python-gps”, it of course removes the “gpsd-clients” which existed prior to all this happening, and I’m not exactly sure whether that’s a good thing.

pi@satnogspi3:~ $ sudo apt install python-gps
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages were automatically installed and are no longer required:
graphviz libcdt5 libcgraph6 libcomedi0 libgnuradio-analog3.7.13 libgnuradio-atsc3.7.13 libgnuradio-channels3.7.13 libgnuradio-comedi3.7.13
libgnuradio-digital3.7.13 libgnuradio-dtv3.7.13 libgnuradio-fec3.7.13 libgnuradio-fft3.7.13 libgnuradio-filter3.7.13 libgnuradio-noaa3.7.13
libgnuradio-pager3.7.13 libgnuradio-qtgui3.7.13 libgnuradio-trellis3.7.13 libgnuradio-video-sdl3.7.13 libgnuradio-vocoder3.7.13
libgnuradio-wavelet3.7.13 libgnuradio-wxgui3.7.13 libgnuradio-zeromq3.7.13 libgts-0.7-5 libgts-bin libgvc6 libgvpr2 liblab-gamut1 libpathplan4
libxaw7 libxdot4 libxmu6 python-bs4 python-cairo python-cheetah python-gdal python-gtk2 python-html5lib python-lxml python-networkx python-opengl
python-pygraphviz python-pyqt5 python-soupsieve python-webencodings python-zmq python3-gps python3-serial
Use ‘sudo apt autoremove’ to remove them.
The following packages will be REMOVED:
gpsd-clients
The following NEW packages will be installed:
python-gps
0 upgraded, 1 newly installed, 1 to remove and 2 not upgraded.
Need to get 0 B/98.6 kB of archives.
After this operation, 32.5 MB disk space will be freed.
Do you want to continue? [Y/n]

This is from one of the “straight from stock” Pi’s I provisioned to be a guinea pig for these recent issues. In addition to having updated a bunch of GR stuff, and now recommending that I remove the cruft, it’s still holding a few things back (pinned).

So, while I’m currently online and able to schedule some observations (to test some new antennas), I’m curious if others are seeing these issues as well…

Anyone ?

Cheers,
-Lucky