Im on Station 702 and just got 6 missing waterfalls.
Very odd and rare.
How do I go about troubleshooting from my end?
Edit. At the moment, the only thing I have done is to try and reapply the settings in satnogs-setup.
Running the latest client version on a 686 Debian PC. Been working fine for about a week.
I don’t know what causes this, either - but I’ve experienced it. Sudo Reboot seems to fix it. Interested to learn what others have to offer.
This post here suggest not to reboot;
what does
df -h
and
ps aux | grep -e "satnogs_"
tell?
df -h
Filesystem Size Used Avail Use% Mounted on
udev 865M 0 865M 0% /dev
tmpfs 176M 19M 157M 11% /run
/dev/sda1 233G 2.9G 218G 2% /
tmpfs 876M 0 876M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 876M 0 876M 0% /sys/fs/cgroup
tmpfs 876M 29M 848M 4% /tmp/.satnogs
tmpfs 176M 0 176M 0% /run/user/1000
ps aux | grep -e “satnogs_”
root 23268 0.0 0.0 12780 1020 pts/0 S+ 11:07 0:00 grep -e satnogs_
ls -al /tmp/.satnogs/data
?
total 28904
drwxr-xr-x 4 satnogs satnogs 280 Aug 9 07:32 .
drwxrwxrwt 3 root root 80 Aug 9 11:09 …
drwxr-xr-x 8 satnogs satnogs 160 Aug 5 17:35 files
drwxr-xr-x 2 satnogs satnogs 40 Aug 5 17:35 incomplete
-rw-r–r-- 1 satnogs satnogs 5056433 Aug 7 17:06 receiving_satnogs_899542_2019-08-07T23-56-53.out
-rw-r–r-- 1 satnogs satnogs 3833856 Aug 7 17:36 receiving_satnogs_899561_2019-08-08T00-28-59.out
-rw-r–r-- 1 satnogs satnogs 692224 Aug 7 19:54 receiving_satnogs_899578_2019-08-08T02-52-53.out
-rw-r–r-- 1 satnogs satnogs 1773568 Aug 8 20:16 receiving_satnogs_905255_2019-08-09T03-13-18.out
-rw-r–r-- 1 satnogs satnogs 1306624 Aug 9 07:40 receiving_satnogs_906164_2019-08-09T14-32-29.out
-rw-r–r-- 1 satnogs satnogs 5330000 Aug 7 17:06 receiving_waterfall_899542_2019-08-07T23-56-53.dat
-rw-r–r-- 1 satnogs satnogs 4038500 Aug 7 17:36 receiving_waterfall_899561_2019-08-08T00-28-59.dat
-rw-r–r-- 1 satnogs satnogs 803600 Aug 7 19:54 receiving_waterfall_899578_2019-08-08T02-52-53.dat
-rw-r–r-- 1 satnogs satnogs 2050000 Aug 8 20:16 receiving_waterfall_905255_2019-08-09T03-13-18.dat
-rw-r–r-- 1 satnogs satnogs 4698600 Aug 9 07:40 receiving_waterfall_906164_2019-08-09T14-32-29.dat
Everything but the temporary storage looks good. Strange that it does not convert the raw data into .ogg and .png. What does
journalctl -u satnogs-client.service
tell you for the observations at the specific times the files in /tmp/.satnogs/data/ are written?
An example of the last observation (906164) would be sufficient in here!
That file is HUGE. Is there a way to jump to the end?
Yes, type the > character at the prompt 
Sorry for the wall of errors.
Might have to cut paste this into a text editor;
Note. I could not find the exact observation number, so this is one of many repeating patterns between obs.
Aug 09 11:20:32 satnogs-acer satnogs-client[22099]: 2019-08-09 11:20:32,541 - apscheduler.executors.default - INFO - Job “get_jobs (trigger: interval[0:01:00], next run at: 2019-08-09 18:21:30 UTC)” executed successfully
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: Exception in thread Thread-55:
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: Traceback (most recent call last):
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: self.run()
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/threading.py”, line 754, in run
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: self.__target(*self.__args, **self.__kwargs)
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 123, in _communicate_tracking_info
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: self.check_observation_end_reached()
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 168, in check_observation_end_reached
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: self.trackstop()
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 164, in trackstop
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: os.killpg(os.getpgid(self._gnu_proc.pid), signal.SIGINT)
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: OSError: [Errno 3] No such process
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: 2019-08-09 11:21:14,026 - apscheduler.executors.default - INFO - Running job “spawn_observer (trigger: date[2019-08-09 18:21:14 UTC], next run at: 2019-08-09 18:21:14 UTC)” (scheduled at 2019-08-09 18:21:14+00:00)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: 2019-08-09 11:21:14,090 - satnogsclient - ERROR - Cannot connect to socket 127.0.0.1:4533
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: Exception in thread Thread-57:
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: Traceback (most recent call last):
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.run()
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/threading.py”, line 754, in run
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.__target(*self.__args, **self.__kwargs)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 142, in _status_interface
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: sock.bind()
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/commsocket.py”, line 104, in bind
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.s.bind((self._TCP_IP, self._TCP_PORT))
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/socket.py”, line 228, in meth
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: return getattr(self._sock,name)(*args)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: error: [Errno 98] Address already in use
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: linux; GNU C++ version 6.3.0 20170221; Boost_106200; UHD_003.009.005-0-unknown
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: 2019-08-09 11:21:14,141 - satnogsclient - ERROR - Cannot connect to socket 127.0.0.1:4533
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: Exception in thread Thread-56:
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: Traceback (most recent call last):
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.run()
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/threading.py”, line 754, in run
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.__target(*self.__args, **self.__kwargs)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 134, in _communicate_tracking_info
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.send_to_socket(p, sock)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 180, in send_to_socket
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: position = sock.send(“p\n”).split(’\n’)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/commsocket.py”, line 76, in send
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.s.send(message)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: error: [Errno 32] Broken pipe
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: linux; GNU C++ version 6.3.0 20170221; Boost_106200; UHD_003.009.005-0-unknown
Aug 09 11:21:16 satnogs-acer satnogs-client[22099]: Usage: satnogs_bpsk_ax25.py: [options]
Aug 09 11:21:16 satnogs-acer satnogs-client[22099]: satnogs_bpsk_ax25.py: error: option --rf-gain: invalid engineering notation value: ‘7.7.’
Aug 09 11:21:30 satnogs-acer satnogs-client[22099]: 2019-08-09 11:21:30,906 - apscheduler.executors.default - INFO - Running job “get_jobs (trigger: interval[0:01:00], next run at: 2019-08-09 18:22:30 UTC)” (scheduled at 2019-08-09 18:21:30.852326+00:00)
Aug 09 11:21:32 satnogs-acer satnogs-client[22099]: 2019-08-09 11:21:32,508 - apscheduler.executors.default - INFO - Job “get_jobs (trigger: interval[0:01:00], next run at: 2019-08-09 18:22:30 UTC)” executed successfully
Aug 09 11:21:45 satnogs-acer satnogs-client[22099]: 2019-08-09 11:21:45,620 - apscheduler.executors.default - INFO - Job “spawn_observer (trigger: date[2019-08-09 18:21:14 UTC], next run at: 2019-08-09 18:21:14 UTC)” executed successfully
Aug 09 11:21:16 satnogs-acer satnogs-client[22099]: satnogs_bpsk_ax25.py: error: option --rf-gain: invalid engineering notation value: ‘7.7.’
Check the RF gain. Seems to have an extra dot in the number.
Yup. Just noticed that.
Wow. Never expected a fat finger to cause so much grief.
Thanks for your help.
Sorry for the noise.
Always learning…
No worries!
You may also want to check out the SatNOGS IRC channel for even quicker feedback.
Note to self and others, if your station starts to operate outside of its usual behaviour, put it in testing mode.
I just got an email advising me as such… lesson learned, put in testing mode, then troubleshoot.
@thebaldgeek if you have some time please add your case in this wiki page to help others in the future.
Having empty waterfall issues with another station today. #908 which I put in testing mode as soon as I got the email letting me know that it was returning abnormal results.
Looked at my own notes from the wiki and can see that I have enough hard drive.
I don’t have any rouge satnog clients running.
Looking the log file, I see this in a pattern;
Aug 15 19:58:46 raspberrypi satnogs-client[18169]: 2019-08-15 19:58:46,830 - apscheduler.executors.default - INFO - Job "get_jobs (trigger: interval[0:01:00], next run at: 2019-08-15 19:59:45 UTC)" executed successfully
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: 2019-08-15 19:59:26,018 - apscheduler.executors.default - INFO - Running job "spawn_observer (trigger: date[2019-08-15 19:59:26 UTC], next run at: 2019-08-15 19:59:26 UTC)" (scheduled at
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: 2019-08-15 19:59:26,108 - satnogsclient - ERROR - Cannot connect to socket 127.0.0.1:4533
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: Exception in thread Thread-384:
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: Traceback (most recent call last):
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: self.run()
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: File "/usr/lib/python2.7/threading.py", line 754, in run
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: self.__target(*self.__args, **self.__kwargs)
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py", line 142, in _status_interface
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: sock.bind()
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/commsocket.py", line 104, in bind
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: self.s.bind((self._TCP_IP, self._TCP_PORT))
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: File "/usr/lib/python2.7/socket.py", line 228, in meth
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: return getattr(self._sock,name)(*args)
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: error: [Errno 98] Address already in use
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: 2019-08-15 19:59:26,149 - satnogsclient - ERROR - Cannot connect to socket 127.0.0.1:4533
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: Exception in thread Thread-383:
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: Traceback (most recent call last):
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: self.run()
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: File "/usr/lib/python2.7/threading.py", line 754, in run
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: self.__target(*self.__args, **self.__kwargs)
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py", line 134, in _communicate_tracking_info
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: self.send_to_socket(p, sock)
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py", line 180, in send_to_socket
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: position = sock.send("p\n").split('\n')
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/commsocket.py", line 76, in send
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: self.s.send(message)
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: error: [Errno 32] Broken pipe
Aug 15 19:59:27 raspberrypi satnogs-client[18169]: 2019-08-15 19:59:27,064 - apscheduler.executors.default - INFO - Job "spawn_observer (trigger: date[2019-08-15 19:59:26 UTC], next run at: 2019-08-15 19:59:26 UTC)" executed successfully
Aug 15 19:59:45 raspberrypi satnogs-client[18169]: 2019-08-15 19:59:45,909 - apscheduler.executors.default - INFO - Running job "get_jobs (trigger: interval[0:01:00], next run at: 2019-08-15 20:00:45 UTC)" (scheduled at 2019-08-15 19:59:
A
And I see this as well;
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: Exception in thread Thread-385:
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: Traceback (most recent call last):
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: self.run()
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: File "/usr/lib/python2.7/threading.py", line 754, in run
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: self.__target(*self.__args, **self.__kwargs)
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py", line 123, in _communicate_tracking_info
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: self.check_observation_end_reached()
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py", line 168, in check_observation_end_reached
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: self.trackstop()
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py", line 164, in trackstop
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: os.killpg(os.getpgid(self._gnu_proc.pid), signal.SIGINT)
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: OSError: [Errno 3] No such process
I have not rebooted as I don’t want to lose the passes that I have if I can.
Thanks for your help once again.
I experienced those same errors as well. The troubleshooting page lists “Cannot connect to socket 127.0.0.1:4533” as an roctld related error.
This error only shows in the satnog-client logs when a scheduled sat operation takes place and tries to connect for data transfer. You can see rotctld has a brief socket while running sudo netstat -atnp. Systemctl status shows it’s loaded, but exited, with status disabled, vendor preset enabled.
local webserver at 127.0.0.1:5000 config shows values:
SATNOGS_ROT_FLIP False
SATNOGS_ROT_FLIP_ANGLE 75
SATNOGS_ROT_IP 127.0.0.1
SATNOGS_ROT_PORT 4533
SATNOGS_ROT_THRESHOLD 4
I never had rotctld turned on in the satnog-setup. ever. So I changed the port number from 4533 to port 4444, stopped, and started the service, and all of the ‘connection failed, and numbered python errors’ disappeared completely. This workaround tells me that somewhere in the code, it is being told to behave as if you have an antenna control, but the values in satnog-setup arent having any effect on turning off this feature. I can rewrite this for clarity later if requested.
ADDENDUM: After installing python-pip, which is required for gevent and gevent-websocket, the connection error has returned under port 4444 now, and the python errors have different numbers.
I have experienced recurrent errors with Ubuntu stations and a RTLSDR
(after about a week or so of good working). The client was working nicely but I got no waterfall or sound.
I made the test rtl_sdr to see if it returned something as in:
rtl_sdr -f 144.8M -g 48 -s 250000 -n 1000000 iq.dat
As it returned nothing, I deduced the problem was near the USB port
(however lsusb was returning correct results).
So I rebooted the whole USB system (using /sys/bus/pci/drivers/…) and all the problems were gone.