This afternoon I’ve made an “Upgrade System Packages” using sudo Satnogs-Setup.
Since then, my station is Off-line. I reboot it several times without success. How can I check what’s happening using SSH?
Thanks
73’s from EA5WA Juan Carlos
1 Like
– Logs begin at Sun 2020-02-16 16:26:49 GMT. –
Feb 16 16:27:01 raspberrypi satnogs-client[419]: import gps
Feb 16 16:27:01 raspberrypi satnogs-client[419]: ImportError: No module named gp s
Feb 16 16:27:01 raspberrypi systemd[1]: satnogs-client.service: Main process exi ted, code=exited, status=1/FAILURE
Feb 16 16:27:01 raspberrypi systemd[1]: satnogs-client.service: Failed with resu lt ‘exit-code’.
Feb 16 16:27:02 raspberrypi systemd[1]: satnogs-client.service: Service RestartS ec=100ms expired, scheduling restart.
Feb 16 16:27:02 raspberrypi systemd[1]: satnogs-client.service: Scheduled restar t job, restart counter is at 5.
Feb 16 16:27:02 raspberrypi systemd[1]: Stopped SatNOGS client.
Feb 16 16:27:02 raspberrypi systemd[1]: satnogs-client.service: Start request re peated too quickly.
Feb 16 16:27:02 raspberrypi systemd[1]: satnogs-client.service: Failed with resu lt ‘exit-code’.
Feb 16 16:27:02 raspberrypi systemd[1]: Failed to start SatNOGS client.
1 Like
2 Likes
Many thanks…Now is ONLINE
1 Like
All of mine experienced the same issue. Did an upgrade, more by habit (than necessity) for testing units, and all three collapsed on themselves. After kicking myself for breaking the first golden rule of software (don’t update unless it’s broken), I’m finding that even fresh installs from the last released IMG are seeing issues if left to update (normally) on their own (described at the bottom)
Also, for me, the client-update started squawking about expected kernel updates which I think has more to do with something upstream with Raspbian than the SATNOGS load, but I can’t be positive on that. (32bit vs 64bit expected kernels ?)
results from uname -a:
Linux satnogspi2 4.19.102-v7+ #1295 SMP Thu Feb 6 15:43:59 GMT 2020 armv7l GNU/Linux
The “warning” speaks of expecting …-v8+ which I thought was only for 64-bit (still researching)
Newer kernel available
The currently running kernel version is 4.19.102-v7+ which is not the expected kernel version 4.19.102-v8+.
Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting.
Obviously, the recommended restart didn’t “solve” anything. Still boots back into kernel7l as it should.
looking in /boot for the images
satnogspi2 :: ~ » ls -al /boot/*.img
-rwxr-xr-x 1 root root 5432400 Feb 16 07:57 /boot/kernel7.img
-rwxr-xr-x 1 root root 5766672 Feb 16 07:57 /boot/kernel7l.img
-rwxr-xr-x 1 root root 13521408 Feb 16 07:57 /boot/kernel8.img
-rwxr-xr-x 1 root root 5152576 Feb 16 07:57 /boot/kernel.img
which of course shows all the bootloaders… This is a RasPi 3Bv1.2 (reports as armv7l under 32bit OS, actually an ARMv8 chip though), (Rev, a22082) btw (from cpuinfo)
Hardware : BCM2835
Revision : a22082
Serial : 00000000########
Model : Raspberry Pi 3 Model B Rev 1.2
Another data point:
'cause it’s the weekend, and American Football is over… I took yet another “fresh” SATNOGS install from it’s initial image (Tag version ‘2019091100’), all the way through a standard update & setup for the client.
- In addition to having the ca-certificates issue listed here, and fixing that with the solution here,
- it also failed in starting the satnogs-client. (failed after 10 repeats as per the journalctl logs)
snipped last entry from that log
Feb 16 08:51:56 satnogspi2 systemd[1]: Started SatNOGS client.
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: Traceback (most recent call last):
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: File “/var/lib/satnogs/bin/satnogs-client”, line 5, in
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: from satnogsclient import main
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/init.py”, line 13, in <mod
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: from satnogsclient.locator import locator
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/locator/locator.py”, line 5, i
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: import gps
Feb 16 08:51:58 satnogspi2 satnogs-client[563]: ImportError: No module named gps
Feb 16 08:51:58 satnogspi2 systemd[1]: satnogs-client.service: Main process exited, code=exited, status=1/FAILURE
Feb 16 08:51:58 satnogspi2 systemd[1]: satnogs-client.service: Failed with result ‘exit-code’.
Feb 16 08:51:58 satnogspi2 systemd[1]: satnogs-client.service: Service RestartSec=100ms expired, scheduling restart.
Feb 16 08:51:58 satnogspi2 systemd[1]: satnogs-client.service: Scheduled restart job, restart counter is at 10.
Feb 16 08:51:58 satnogspi2 systemd[1]: Stopped SatNOGS client.
Feb 16 08:51:58 satnogspi2 systemd[1]: satnogs-client.service: Start request repeated too quickly.
Feb 16 08:51:58 satnogspi2 systemd[1]: satnogs-client.service: Failed with result ‘exit-code’.
Feb 16 08:51:58 satnogspi2 systemd[1]: Failed to start SatNOGS client.
So, I did the suggested addition of the python-gps package as mentioned above, and that seems to have addressed one of the most critical issues. It’s online now, but the logs still make me wonder if it’s actually “happy”…:
Feb 16 19:10:46 satnogspi2 systemd[1]: Started SatNOGS client.
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: 2020-02-16 19:20:56,923 - apscheduler.executors.default - ERROR - Job "get_jobs (trigger: interval[0:0
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: Traceback (most recent call last):
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/usr/lib/python2.7/dist-packages/apscheduler/executors/base.py”, line 125, in run_job
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: retval = job.func(*job.args, **job.kwargs)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/scheduler/tasks.py”, line 163
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: timeout=45)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/usr/lib/python2.7/dist-packages/requests/api.py”, line 75, in get
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: return request(‘get’, url, params=params, **kwargs)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/usr/lib/python2.7/dist-packages/requests/api.py”, line 60, in request
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: return session.request(method=method, url=url, **kwargs)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/usr/lib/python2.7/dist-packages/requests/sessions.py”, line 533, in request
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: resp = self.send(prep, **send_kwargs)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/usr/lib/python2.7/dist-packages/requests/sessions.py”, line 646, in send
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: r = adapter.send(request, **kwargs)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: File “/usr/lib/python2.7/dist-packages/requests/adapters.py”, line 516, in send
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: raise ConnectionError(e, request=request)
Feb 16 19:20:56 satnogspi2 satnogs-client[2610]: ConnectionError: HTTPSConnectionPool(host=‘network.satnogs.org’, port=443): Max retries exceeded with
So… it’s online, and “fixed” as per the issue raised by the OP, but is it actually “fixed” ?
The kernel issue is vexing, because I’m seeing this on every new image I’m provisioning, and I’m not sure what to attribute it to…
also, when actually installing the package “python-gps”, it of course removes the “gpsd-clients” which existed prior to all this happening, and I’m not exactly sure whether that’s a good thing.
pi@satnogspi3:~ $ sudo apt install python-gps
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages were automatically installed and are no longer required:
graphviz libcdt5 libcgraph6 libcomedi0 libgnuradio-analog3.7.13 libgnuradio-atsc3.7.13 libgnuradio-channels3.7.13 libgnuradio-comedi3.7.13
libgnuradio-digital3.7.13 libgnuradio-dtv3.7.13 libgnuradio-fec3.7.13 libgnuradio-fft3.7.13 libgnuradio-filter3.7.13 libgnuradio-noaa3.7.13
libgnuradio-pager3.7.13 libgnuradio-qtgui3.7.13 libgnuradio-trellis3.7.13 libgnuradio-video-sdl3.7.13 libgnuradio-vocoder3.7.13
libgnuradio-wavelet3.7.13 libgnuradio-wxgui3.7.13 libgnuradio-zeromq3.7.13 libgts-0.7-5 libgts-bin libgvc6 libgvpr2 liblab-gamut1 libpathplan4
libxaw7 libxdot4 libxmu6 python-bs4 python-cairo python-cheetah python-gdal python-gtk2 python-html5lib python-lxml python-networkx python-opengl
python-pygraphviz python-pyqt5 python-soupsieve python-webencodings python-zmq python3-gps python3-serial
Use ‘sudo apt autoremove’ to remove them.
The following packages will be REMOVED:
gpsd-clients
The following NEW packages will be installed:
python-gps
0 upgraded, 1 newly installed, 1 to remove and 2 not upgraded.
Need to get 0 B/98.6 kB of archives.
After this operation, 32.5 MB disk space will be freed.
Do you want to continue? [Y/n]
This is from one of the “straight from stock” Pi’s I provisioned to be a guinea pig for these recent issues. In addition to having updated a bunch of GR stuff, and now recommending that I remove the cruft, it’s still holding a few things back (pinned).
So, while I’m currently online and able to schedule some observations (to test some new antennas), I’m curious if others are seeing these issues as well…
Anyone ?
Cheers,
-Lucky
I have both installed ok using SatNOGS Raspberry Pi image:
pi@cruftpi3:~ $ dpkg -l|grep gps
ii gpsd 3.17-7 armhf Global Positioning System - daemon
ii gpsd-clients 3.17-7 armhf Global Positioning System - clients
ii libgps23:armhf 3.17-7 armhf Global Positioning System - library
ii python-gps 3.17-7 armhf Global Positioning System - Python libraries
Edit: I see if I do an upgrade, it will remove python-gps:
root@cruftpi3:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
libboost-atomic1.62.0 libboost-atomic1.67-dev libboost-chrono1.62.0 libboost-chrono1.67-dev libboost-date-time-dev libboost-date-time1.62.0
libboost-date-time1.67-dev libboost-filesystem-dev libboost-filesystem1.67-dev libboost-program-options-dev libboost-program-options1.67-dev
libboost-serialization1.67-dev libboost-system-dev libboost-system1.67-dev libboost-test-dev libboost-test1.67-dev libboost-thread-dev
libboost-thread1.67-dev libboost1.67-dev libcppunit-1.14-0 libcppunit-dev libdrm-etnaviv1 libexpat1-dev libglfw3 libgnuradio-fosphor3.7.12
libgps23 libhiredis0.14 libjemalloc2 libllvm6.0 libllvm8 liblog4cpp5-dev liblua5.1-0 libpython-dev libpython2-dev libpython2.7-dev lua-bitop
lua-cjson ocl-icd-libopencl1 python2-dev python2.7-dev redis-tools
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
python-gps
The following NEW packages will be installed:
gir1.2-atk-1.0 gir1.2-freedesktop gir1.2-gdkpixbuf-2.0 gir1.2-glib-2.0 gir1.2-gtk-3.0 gir1.2-pango-1.0 libgirepository-1.0-1 libgps25 libllvm9
libpangoxft-1.0-0 libuhd3.14.1 python3-cairo python3-gi python3-gi-cairo python3-gps python3-serial python3-tk
The following packages have been kept back:
gnuradio
The following packages will be upgraded:
ansible base-files e2fsprogs gpsd gpsd-clients libavcodec58 libavresample4 libavutil56 libboost-atomic1.67-dev libboost-atomic1.67.0
libboost-chrono1.67-dev libboost-chrono1.67.0 libboost-date-time1.67-dev libboost-date-time1.67.0 libboost-filesystem1.67-dev
libboost-filesystem1.67.0 libboost-program-options1.67-dev libboost-program-options1.67.0 libboost-regex1.67.0 libboost-serialization1.67-dev
libboost-serialization1.67.0 libboost-system1.67-dev libboost-system1.67.0 libboost-test1.67-dev libboost-test1.67.0 libboost-thread1.67-dev
libboost-thread1.67.0 libboost-timer1.67.0 libboost1.67-dev libcom-err2 libcups2 libdrm-amdgpu1 libdrm-common libdrm-etnaviv1 libdrm-nouveau2
libdrm-radeon1 libdrm2 libegl-mesa0 libegl1 libext2fs2 libgbm1 libgl1 libgl1-mesa-dri libglapi-mesa libgles2 libglvnd0 libglx-mesa0 libglx0
libgnutls30 libidn2-0 libllvm7 libmariadb3 libopengl0 libopenjp2-7 libpq5 libpython3.7 libpython3.7-minimal libpython3.7-stdlib libqt5core5a
libqt5dbus5 libqt5gui5 libqt5network5 libqt5opengl5 libqt5printsupport5 libqt5sql5 libqt5sql5-sqlite libqt5test5 libqt5widgets5 libqt5xml5
libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc libraspberrypi0 libsasl2-2 libsasl2-modules-db libss2 libssl1.1 libswresample3
libtiff5 mariadb-common mesa-va-drivers mesa-vdpau-drivers openssh-client openssh-server openssh-sftp-server openssl python-apt
python-apt-common python3-apt python3.7 python3.7-minimal qt5-gtk-platformtheme raspberrypi-bootloader raspberrypi-kernel raspi-config
rpi-eeprom rpi-eeprom-images ssh sudo uhd-host
100 upgraded, 17 newly installed, 1 to remove and 1 not upgraded.
Need to get 0 B/213 MB of archives.
After this operation, 151 MB of additional disk space will be used.
Do you want to continue? [Y/n] ^C
…and removing python-gps “breaks” something needed by the client, or at least I’m assuming that because it works when python-gps is installed.
oh boy… conflicting system-wide packages… my least favorite kind of troubleshooting. 
Admittedly not sure what’s needed by what at the moment… it’s quite a bit of code and I’ve only now started digging into it 
One thing I did notice was that many Pythonic dependencies for the client were isolated into a VirtEnv, so as to keep things isolated from system-wide shenanigans like this, so I’m curious (for anyone who knows)… where’s that line for the client where some things are needed system-wide, and others are “local” ?
Can someone point me to document(s) which describe the layout and architecture of the client ? besides the source code of course
Cheers,
-Lucky
Just use satnogs-setup Update options. It will upgrade system packages and also restore the correct dependencies of python-gps after selecting Apply.
2 Likes
Unfortunately, we are in a transitioning period from stretch to buster, GNU Radio 3.7 to 3.8, gr-osmosdr to gr-soapy which may create some dependency issues.
This specific issue was caused because we prepared the repositories for the new stack but at the last moment an outstanding bug was discovered which prevented it. The new gpsd bindings, which are Python 3 only are removed so the issue should be fixed now if you follow the procedure I posted on my previous post.
2 Likes
@Acinonyx Absolutely appreciate the challenges in migrating from 3 (or more) different legacy versions to a bright new future “stable”. Been there a few times myself, and given the complexity of the system that’s been built here I have full respect and appreciation for how hard it must be.
That said, the “Stable” branch of the “old” stuff isn’t coming back as per your advice above.
after jumping through hoops to get the client service to start, and stay started, etc… I’m NOT getting successful observations. Not a RF signal thing, the system won’t even kick things into gear, much less process anything or output data (waterfall, etc…)
latest error, from a cold reboot, watching journalctrl -f -u… I’m getting the following:
– Logs begin at Mon 2020-02-17 04:00:58 UTC. –
Feb 17 04:01:02 satnogspi systemd[1]: Started SatNOGS client.
Feb 17 04:13:24 satnogspi satnogs-client[354]: rot_init called
Feb 17 04:13:24 satnogspi satnogs-client[354]: dummy: _init called
Feb 17 04:13:24 satnogspi satnogs-client[354]: rot_register (1)
Feb 17 04:13:24 satnogspi satnogs-client[354]: rot_register (2)
Feb 17 04:13:24 satnogspi satnogs-client[354]: dummy_rot_init called
Feb 17 04:13:24 satnogspi satnogs-client[354]: rot_open called
Feb 17 04:13:24 satnogspi satnogs-client[354]: dummy_rot_open called
Feb 17 04:13:24 satnogspi satnogs-client[354]: rot_get_position called
Feb 17 04:13:24 satnogspi satnogs-client[354]: dummy_rot_get_position called
Feb 17 04:13:24 satnogspi satnogs-client[354]: rot_set_position called
Feb 17 04:13:25 satnogspi satnogs-client[354]: 2020-02-17 04:13:25,019 - apscheduler.executors.default - ERROR - Job “spawn_observer (trigger: date[2020-02-17 04:13:24 UTC], next run at: 2020-02-17 04:13:24 UTC)” raised an exception
Feb 17 04:13:25 satnogspi satnogs-client[354]: Traceback (most recent call last):
Feb 17 04:13:25 satnogspi satnogs-client[354]: File “/usr/lib/python2.7/dist-packages/apscheduler/executors/base.py”, line 125, in run_job
Feb 17 04:13:25 satnogspi satnogs-client[354]: retval = job.func(*job.args, **job.kwargs)
Feb 17 04:13:25 satnogspi satnogs-client[354]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/scheduler/tasks.py”, line 80, in spawn_observer
Feb 17 04:13:25 satnogspi satnogs-client[354]: observer.observe()
Feb 17 04:13:25 satnogspi satnogs-client[354]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/observer.py”, line 152, in observe
Feb 17 04:13:25 satnogspi satnogs-client[354]: self.observation_decoded_data)
Feb 17 04:13:25 satnogspi satnogs-client[354]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/upsat/gnuradio_handler.py”, line 52, in exec_gnuradio
Feb 17 04:13:25 satnogspi satnogs-client[354]: script_name, ‘–soapy-rx-device=’ + device, ‘–samp-rate-rx=’ + str(samp_rate),
Feb 17 04:13:25 satnogspi satnogs-client[354]: TypeError: cannot concatenate ‘str’ and ‘NoneType’ objects
So, from this I know a few things are working. The task scheduler IS reaching out, or being told what to do because this popped up before a scheduled test task. However, it looks like something is trying to kick off a soapy-based Rx unit, which it wasn’t doing in the old version (gr-osmosdr).
So a few things of the new didn’t revert back to the old, and I’m in this limbo zone of “running”, but not being able to do anything.
second scheduled observation, minutes later, pretty much spewed the same…
Feb 17 04:18:57 satnogspi satnogs-client[354]: 2020-02-17 04:18:57,018 - apscheduler.executors.default - ERROR - Job “spawn_observer (trigger: date[2020-02-17 04:18:56 UTC], next run at: 2020-02-17 04:18:56 UTC)” raised an exception
Feb 17 04:18:57 satnogspi satnogs-client[354]: Traceback (most recent call last):
Feb 17 04:18:57 satnogspi satnogs-client[354]: File “/usr/lib/python2.7/dist-packages/apscheduler/executors/base.py”, line 125, in run_job
Feb 17 04:18:57 satnogspi satnogs-client[354]: retval = job.func(*job.args, **job.kwargs)
Feb 17 04:18:57 satnogspi satnogs-client[354]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/scheduler/tasks.py”, line 80, in spawn_observer
Feb 17 04:18:57 satnogspi satnogs-client[354]: observer.observe()
Feb 17 04:18:57 satnogspi satnogs-client[354]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/observer.py”, line 152, in observe
Feb 17 04:18:57 satnogspi satnogs-client[354]: self.observation_decoded_data)
Feb 17 04:18:57 satnogspi satnogs-client[354]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/upsat/gnuradio_handler.py”, line 52, in exec_gnuradio
Feb 17 04:18:57 satnogspi satnogs-client[354]: script_name, ‘–soapy-rx-device=’ + device, ‘–samp-rate-rx=’ + str(samp_rate),
Feb 17 04:18:57 satnogspi satnogs-client[354]: TypeError: cannot concatenate ‘str’ and ‘NoneType’ objects
Capture from the “Generate support info” lifeline feature 
------------[ copy here ]------------
{
“versions”: {
“satnogs-client”: “1.0+29.g2b7011a”,
“satnogs-client-ansible”: “202002121234”,
“gr-satnogs”: “1.5.1-1”,
“satnogs-config”: “0.6.3”
},
“state”: {
“is-applied”: true,
“pending-tags”: null
},
“system”: {
“distribution”: {
“DESCRIPTION”: “Raspbian GNU/Linux 10 (buster)”,
“RELEASE”: “10”,
“CODENAME”: “buster”,
“ID”: “Raspbian”
},
“pending-updates”: false,
“platform”: {
“system”: “Linux”,
“node”: “satnogspi”,
“release”: “4.19.97-v7+”,
“version”: “#1294 SMP Thu Jan 30 13:15:58 GMT 2020”,
“machine”: “armv7l”,
“processor”: “”
},
“memory”: {
“total”: 971055104,
“available”: 747638784,
“percent”: 23.0,
“used”: 159817728,
“free”: 598999040,
“active”: 224284672,
“inactive”: 78426112,
“buffers”: 28065792,
“cached”: 184172544,
“shared”: 7503872,
“slab”: 46481408
},
“disk”: {
“total”: 15348867072,
“used”: 4110381056,
“free”: 10568568832,
“percent”: 28.0
}
},
“configuration”: {
“satnogs_api_token”: “[reducted]”,
“satnogs_network_api_url”: “SatNOGS Network API”,
“satnogs_rf_gain”: “7.7”,
“satnogs_rx_device”: “rtlsdr”,
“satnogs_station_elev”: “199”,
“satnogs_station_id”: “1291”,
“satnogs_station_lat”: “34.635”,
“satnogs_station_lon”: “-86.499”
}
}
------------[ copy end ]-------------
as you can see, I’m not trying to specify a soapy-version of rtlsdr dongle. just the standard one.
Thoughts ?
-Lucky
Thanks for the support report. It helps a lot.
For some reason you are using the latest version of satnogs-client which is not compatible with gr-satnogs-1.5.1-1. Did you ever enable the experimental option in satnogs-setup or overriden the SATNOGS_CLIENT_VERSION?
If you Update and Apply again and it should revert to the correct version. The issue with python-gps should be fixed now. If not please reopen issue #67.
4 Likes