Raspbian SatNOGS client image issues

Yes, it was the Raspbian image. That step not included.
I believe the images uses systemd.
Some cleanup would be handy and increase /tmp/.satnogs. Did find that can fill rather quick. Some waterfall files have been >50Mb

Interesting how .ogg files virtually always get uploaded, not so for waterfall. The latter helps to adjust gain.

As a trial, I’ve gone back to a manual build, but upgraded to latest gr-satnogs and satnogs-client.

Thanks for your feedback! There is already a pending merge request to clean up after completed jobs.

There can also be leftover data from incomplete jobs. We could either clean this up in the client, via a cron job or leave it as a manual maintenance task.

Today I tried again with the latest raspbian build.

I get the following error. No files generated in data folder.
Sep 26 20:22:19 raspberrypi satnogs-client[528]: 2017-09-26 20:22:19,950 - satnogsclient - DEBUG - Received message: RPRT 0
Sep 26 20:22:20 raspberrypi satnogs-client[528]: 2017-09-26 20:22:20,064 - satnogsclient - DEBUG - Sending message: F 145983278
Exception in thread Thread-22:
Sep 26 20:21:31 raspberrypi satnogs-client[528]: Traceback (most recent call last):
Sep 26 20:21:31 raspberrypi satnogs-client[528]: File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
Sep 26 20:21:31 raspberrypi satnogs-client[528]: self.run()
Sep 26 20:21:31 raspberrypi satnogs-client[528]: File “/usr/lib/python2.7/threading.py”, line 754, in run
Sep 26 20:21:31 raspberrypi satnogs-client[528]: self.__target(*self.__args, **self.__kwargs)
Sep 26 20:21:31 raspberrypi satnogs-client[528]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 13
Sep 26 20:21:31 raspberrypi satnogs-client[528]: self.send_to_socket(p, sock)
Sep 26 20:21:31 raspberrypi satnogs-client[528]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 18
Sep 26 20:21:31 raspberrypi satnogs-client[528]: position = sock.send(“p\n”).split(’\n’)
Sep 26 20:21:31 raspberrypi satnogs-client[528]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/commsocket.py”, lin
Sep 26 20:21:31 raspberrypi satnogs-client[528]: self.s.send(message)
Sep 26 20:21:31 raspberrypi satnogs-client[528]: error: [Errno 32] Broken pipe

Any one seen this?

also here
Sep 26 21:33:02 raspberrypi satnogs-client[806]: 2017-09-26 21:33:02,994 - satnogsclient - DEBUG - Received message: RPRT 0
Sep 26 21:33:03 raspberrypi satnogs-client[806]: Exception in thread Thread-24:
Sep 26 21:33:03 raspberrypi satnogs-client[806]: Traceback (most recent call last):
Sep 26 21:33:03 raspberrypi satnogs-client[806]: File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
Sep 26 21:33:03 raspberrypi satnogs-client[806]: self.run()
Sep 26 21:33:03 raspberrypi satnogs-client[806]: File “/usr/lib/python2.7/threading.py”, line 754, in run
Sep 26 21:33:03 raspberrypi satnogs-client[806]: self.__target(*self.__args, **self.__kwargs)
Sep 26 21:33:03 raspberrypi satnogs-client[806]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 127, in _communicate_tracking_info
Sep 26 21:33:03 raspberrypi satnogs-client[806]: self.check_observation_end_reached()
Sep 26 21:33:03 raspberrypi satnogs-client[806]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 175, in check_observation_end_reached
Sep 26 21:33:03 raspberrypi satnogs-client[806]: self.trackstop()
Sep 26 21:33:03 raspberrypi satnogs-client[806]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 168, in trackstop
Sep 26 21:33:03 raspberrypi satnogs-client[806]: os.killpg(os.getpgid(self._gnu_proc.pid), signal.SIGKILL)
Sep 26 21:33:03 raspberrypi satnogs-client[806]: OSError: [Errno 3] No such process

It is possibly the same issue with the missing cleanup of completed jobs in the client. It could be that the system ran out of memory and started killing processes randomly. In the first case it may be rotctld and in the second GNU Radio.

This issue will be fixed with the release of SatNOGS client version 0.5 hopefully in a few days.

More Information regarding above logs: I have also tried building satnogs-client 0 branch locally, with similar results. Tried Fedora and Rasbian on raspberry pi 3. Regarding cleanup, the logs were on fresh system. I have tried this with multiple observations. I do-not have any rotor so did not set any related settings in the BASIC or ADVANCED settings page for them.

@Jferns, try the following:

HAMLIB_UTILS_ROT_ENABLED -> Yes
HAMLIB_UTILS_ROT_OPTS -> -m 1

This will enable the hamlib dummy rotator for the client.

I just tried on latest version of client, using False for the HAMLIB_UTILSROT_ENABLED (without a rotator) and it worked out just fine.

Figured out.
@Acinonyx @pierros, Can “no”, “No”, “False” be used interchangeably here?

I think it should be False. Did you find something else?

Well, I’ve gone and broken my systems :frowning:

Had client 0.4 running on my VHF system, reasonably well.
(jessie upgraded to stretch with gr-satnogs and satnogs-client manually installed)

Upgraded to 0.5 using:
sudo systemctl stop satnogs-client.service
sudo pip install satnogsclient --upgrade
sudo reboot

When I looked at the confg page on port 5000, I found SATNOGS_COMPLETE_OUTPUT_PATH was blank and no directory was created for /tmp/.satnogs/data/complete I edited the .env to set the paramater.

I’m not getting any files created in /tmp/.satnogs/data/ and sub-directories. Of course nothing being uploaded to dev-network either.

What would have caused the upgrade from 0.4 to 0.5 to cause this? Do I need to upgrade gr-satnogs as well?

Gone from working system to nothing, except the jobs are still being scheduled and I think still atempting to collect data.

Of course I did the UHF system as well, but 0.3 to 0.5. It broke as well.
*Mental note to self: Only do 1 at a time!!

Hey @Zathras! Sorry for your troubles :frowning:
Did you check the https://wiki.satnogs.org/Compatibility page?

About SATNOGS_COMPLETE_OUTPUT_PATH, it is changed on v0.5 to be blank as the new default behaviour is after successful uploading to delete files (ogg, waterfall and data) and not moving them into the complete directory.

Edit: The rest seems to be compatibility issues between client and gr-satnogs.

Thanks guys.

I’ve changed the SATNOGS_COMPLETE_OUTPUT_PATH to blank.

Never knew about the Compatiblity page. Ok, so I need the latest gr-satnogs.
Now, not being a linux guru, how do I make sure I download and install the correct version? How do I check what version I have installed currently?
I tried git clone https://github.com/satnogs/gr-satnogs.git and recompile, but still not getting any data. Where is the data stored while and after processing. Couldn’t find ogg or waterfall files anywhere?

Sample of the log just after a METEOR-M 2 pass:
Oct 04 10:15:20 satnogs-vhf bash[700]: 2017-10-04 10:15:20,846 - satnogsclient - DEBUG - Sending message: F 137897423
Oct 04 10:15:20 satnogs-vhf bash[700]: 2017-10-04 10:15:20,849 - satnogsclient - DEBUG - Received message: RPRT 0
Oct 04 10:15:20 satnogs-vhf bash[700]: 2017-10-04 10:15:20,973 - satnogsclient - DEBUG - Sending message: F 137897422
Oct 04 10:15:20 satnogs-vhf bash[700]: 2017-10-04 10:15:20,978 - satnogsclient - DEBUG - Received message: RPRT 0
Oct 04 10:15:21 satnogs-vhf bash[700]: Exception in thread Thread-101:
Oct 04 10:15:21 satnogs-vhf bash[700]: Traceback (most recent call last):
Oct 04 10:15:21 satnogs-vhf bash[700]: File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
Oct 04 10:15:21 satnogs-vhf bash[700]: self.run()
Oct 04 10:15:21 satnogs-vhf bash[700]: File “/usr/lib/python2.7/threading.py”, line 754, in run
Oct 04 10:15:21 satnogs-vhf bash[700]: self.__target(*self.__args, **self.__kwargs)
Oct 04 10:15:21 satnogs-vhf bash[700]: File “/usr/local/lib/python2.7/dist-packages/satnogsclient/observer/worker.py”, line 127, in _communicate_tracking_info
Oct 04 10:15:21 satnogs-vhf bash[700]: self.check_observation_end_reached()
Oct 04 10:15:21 satnogs-vhf bash[700]: File “/usr/local/lib/python2.7/dist-packages/satnogsclient/observer/worker.py”, line 175, in check_observation_end_reached
Oct 04 10:15:21 satnogs-vhf bash[700]: self.trackstop()
Oct 04 10:15:21 satnogs-vhf bash[700]: File “/usr/local/lib/python2.7/dist-packages/satnogsclient/observer/worker.py”, line 168, in trackstop
Oct 04 10:15:21 satnogs-vhf bash[700]: os.killpg(os.getpgid(self._gnu_proc.pid), signal.SIGKILL)
Oct 04 10:15:21 satnogs-vhf bash[700]: OSError: [Errno 3] No such process
Oct 04 10:15:33 satnogs-vhf bash[700]: 2017-10-04 10:15:33,989 - apscheduler.executors.default - INFO - Running job “get_jobs (trigger: interval[0:01:00], next run at: 2017-10-04 02:16:33 UTC)” (scheduled at 2017-10-04 02:15:33.964267+00:00)
Oct 04 10:15:36 satnogs-vhf bash[700]: 2017-10-04 10:15:36,019 - satnogsclient - DEBUG - Opening TCP socket: 127.0.0.1:5011

Hi @Zathras – I’m new to SatNOGS, but since you mentioned having the Raspbian image up above:

  • You can check which version of gr-satnogs is present by running dpkg -l gr-satnogs:
pi@raspberrypi:~/.ssh $ dpkg -l gr-satnogs
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                  Version                 Architecture            Description
+++-=====================================-=======================-=======================-===============================================================================
ii  gr-satnogs                            20171001.ae1e331-1      armhf                   SatNOGS GNU Radio Out-Of-Tree Module

Here, you can see that my version is “20171001.ae1e331-1” – meaning it was built October 1 2017, and it’s about a week old (looking for “ae1e331” on that page, you can see I’m a couple of updates behind.)

@Acinonyx, have I got those last two points right? Is there an intended way to fetch/apply the latest versions of packages (perhaps just sudo satnogs-setup without the -n argument)?

Hi saintaardvark,

Nope, that did not work for me. I’m not using the SatNOGS Raspbian image, just what created myself. The instructions no longer exist though :frowning: Which is a pity because I did refer to them all the time.
I had several issues with the SatNOGS Raspbian image, which is why I went back to my version.

This should not have happened. @fredy @comzeradd @Acinonyx did that happen on the Docathon move? Can we restore them somehow on the wiki?

Instructions for Raspbian exist on wiki. Are you talking about other instructions?

For rasbian, these are the instructions but for fedora we have removed them without moving them to wiki.

For now we can find the fedora process here.

I think we should move it back either on wiki or docs repo until all issues with the new rasbian image are solved and it is stable like the fedora one.

@Zathras If you have the rasbian image you used can you check if rotctld process was running? ps -ax or check if port 4533 is listening with netstat -nap

I got the same error on the fresh raspbian image and I think satnogs-client requires rotctld process running, which does not happen if HAMLIB_UTILS_ROT_ENABLED is set to False(No in the configuration screen). As @Acinonyx pointed above if there is no rotator then below setting are required.

HAMLIB_UTILS_ROT_ENABLED -> Yes
HAMLIB_UTILS_ROT_OPTS -> -m 1

I had to set the above setting to get rid of the above error. Correct me if I am wrong. The raspbian wiki might need to be updated or satnogs-client fixed.

Another issue. Also yesterday night I was testing raspbian image again and got lot of TLS certificate errors and PI stopped responding to ssh commands and communicating with website. This started just when a CW observation started. I could not even log back in to PI. I could reproduce it three times. So frustrating.

This is peculiar. Can you elaborate on the issue? Only on CW observations or any observations? Did it become responsive again once the observations finished?