How do I troubleshoot missing waterfalls?

Im on Station 702 and just got 6 missing waterfalls.
Very odd and rare.
How do I go about troubleshooting from my end?

Edit. At the moment, the only thing I have done is to try and reapply the settings in satnogs-setup.

Running the latest client version on a 686 Debian PC. Been working fine for about a week.

I don’t know what causes this, either - but I’ve experienced it. Sudo Reboot seems to fix it. Interested to learn what others have to offer.

This post here suggest not to reboot;

what does

df -h

and

ps aux | grep -e "satnogs_"

tell?

df -h
Filesystem Size Used Avail Use% Mounted on
udev 865M 0 865M 0% /dev
tmpfs 176M 19M 157M 11% /run
/dev/sda1 233G 2.9G 218G 2% /
tmpfs 876M 0 876M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 876M 0 876M 0% /sys/fs/cgroup
tmpfs 876M 29M 848M 4% /tmp/.satnogs
tmpfs 176M 0 176M 0% /run/user/1000

ps aux | grep -e “satnogs_”
root 23268 0.0 0.0 12780 1020 pts/0 S+ 11:07 0:00 grep -e satnogs_

ls -al /tmp/.satnogs/data

?

total 28904
drwxr-xr-x 4 satnogs satnogs 280 Aug 9 07:32 .
drwxrwxrwt 3 root root 80 Aug 9 11:09 …
drwxr-xr-x 8 satnogs satnogs 160 Aug 5 17:35 files
drwxr-xr-x 2 satnogs satnogs 40 Aug 5 17:35 incomplete
-rw-r–r-- 1 satnogs satnogs 5056433 Aug 7 17:06 receiving_satnogs_899542_2019-08-07T23-56-53.out
-rw-r–r-- 1 satnogs satnogs 3833856 Aug 7 17:36 receiving_satnogs_899561_2019-08-08T00-28-59.out
-rw-r–r-- 1 satnogs satnogs 692224 Aug 7 19:54 receiving_satnogs_899578_2019-08-08T02-52-53.out
-rw-r–r-- 1 satnogs satnogs 1773568 Aug 8 20:16 receiving_satnogs_905255_2019-08-09T03-13-18.out
-rw-r–r-- 1 satnogs satnogs 1306624 Aug 9 07:40 receiving_satnogs_906164_2019-08-09T14-32-29.out
-rw-r–r-- 1 satnogs satnogs 5330000 Aug 7 17:06 receiving_waterfall_899542_2019-08-07T23-56-53.dat
-rw-r–r-- 1 satnogs satnogs 4038500 Aug 7 17:36 receiving_waterfall_899561_2019-08-08T00-28-59.dat
-rw-r–r-- 1 satnogs satnogs 803600 Aug 7 19:54 receiving_waterfall_899578_2019-08-08T02-52-53.dat
-rw-r–r-- 1 satnogs satnogs 2050000 Aug 8 20:16 receiving_waterfall_905255_2019-08-09T03-13-18.dat
-rw-r–r-- 1 satnogs satnogs 4698600 Aug 9 07:40 receiving_waterfall_906164_2019-08-09T14-32-29.dat

Everything but the temporary storage looks good. Strange that it does not convert the raw data into .ogg and .png. What does

journalctl -u satnogs-client.service

tell you for the observations at the specific times the files in /tmp/.satnogs/data/ are written?

An example of the last observation (906164) would be sufficient in here!

1 Like

That file is HUGE. Is there a way to jump to the end?

Yes, type the > character at the prompt :wink:

1 Like

Sorry for the wall of errors.
Might have to cut paste this into a text editor;
Note. I could not find the exact observation number, so this is one of many repeating patterns between obs.

Aug 09 11:20:32 satnogs-acer satnogs-client[22099]: 2019-08-09 11:20:32,541 - apscheduler.executors.default - INFO - Job “get_jobs (trigger: interval[0:01:00], next run at: 2019-08-09 18:21:30 UTC)” executed successfully
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: Exception in thread Thread-55:
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: Traceback (most recent call last):
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: self.run()
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/threading.py”, line 754, in run
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: self.__target(*self.__args, **self.__kwargs)
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 123, in _communicate_tracking_info
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: self.check_observation_end_reached()
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 168, in check_observation_end_reached
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: self.trackstop()
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 164, in trackstop
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: os.killpg(os.getpgid(self._gnu_proc.pid), signal.SIGINT)
Aug 09 11:20:44 satnogs-acer satnogs-client[22099]: OSError: [Errno 3] No such process
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: 2019-08-09 11:21:14,026 - apscheduler.executors.default - INFO - Running job “spawn_observer (trigger: date[2019-08-09 18:21:14 UTC], next run at: 2019-08-09 18:21:14 UTC)” (scheduled at 2019-08-09 18:21:14+00:00)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: 2019-08-09 11:21:14,090 - satnogsclient - ERROR - Cannot connect to socket 127.0.0.1:4533
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: Exception in thread Thread-57:
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: Traceback (most recent call last):
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.run()
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/threading.py”, line 754, in run
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.__target(*self.__args, **self.__kwargs)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 142, in _status_interface
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: sock.bind()
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/commsocket.py”, line 104, in bind
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.s.bind((self._TCP_IP, self._TCP_PORT))
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/socket.py”, line 228, in meth
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: return getattr(self._sock,name)(*args)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: error: [Errno 98] Address already in use
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: linux; GNU C++ version 6.3.0 20170221; Boost_106200; UHD_003.009.005-0-unknown
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: 2019-08-09 11:21:14,141 - satnogsclient - ERROR - Cannot connect to socket 127.0.0.1:4533
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: Exception in thread Thread-56:
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: Traceback (most recent call last):
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.run()
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/usr/lib/python2.7/threading.py”, line 754, in run
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.__target(*self.__args, **self.__kwargs)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 134, in _communicate_tracking_info
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.send_to_socket(p, sock)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py”, line 180, in send_to_socket
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: position = sock.send(“p\n”).split(’\n’)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/commsocket.py”, line 76, in send
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: self.s.send(message)
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: error: [Errno 32] Broken pipe
Aug 09 11:21:14 satnogs-acer satnogs-client[22099]: linux; GNU C++ version 6.3.0 20170221; Boost_106200; UHD_003.009.005-0-unknown
Aug 09 11:21:16 satnogs-acer satnogs-client[22099]: Usage: satnogs_bpsk_ax25.py: [options]
Aug 09 11:21:16 satnogs-acer satnogs-client[22099]: satnogs_bpsk_ax25.py: error: option --rf-gain: invalid engineering notation value: ‘7.7.’
Aug 09 11:21:30 satnogs-acer satnogs-client[22099]: 2019-08-09 11:21:30,906 - apscheduler.executors.default - INFO - Running job “get_jobs (trigger: interval[0:01:00], next run at: 2019-08-09 18:22:30 UTC)” (scheduled at 2019-08-09 18:21:30.852326+00:00)
Aug 09 11:21:32 satnogs-acer satnogs-client[22099]: 2019-08-09 11:21:32,508 - apscheduler.executors.default - INFO - Job “get_jobs (trigger: interval[0:01:00], next run at: 2019-08-09 18:22:30 UTC)” executed successfully
Aug 09 11:21:45 satnogs-acer satnogs-client[22099]: 2019-08-09 11:21:45,620 - apscheduler.executors.default - INFO - Job “spawn_observer (trigger: date[2019-08-09 18:21:14 UTC], next run at: 2019-08-09 18:21:14 UTC)” executed successfully

Aug 09 11:21:16 satnogs-acer satnogs-client[22099]: satnogs_bpsk_ax25.py: error: option --rf-gain: invalid engineering notation value: ‘7.7.’

Check the RF gain. Seems to have an extra dot in the number.

2 Likes

Yup. Just noticed that.
Wow. Never expected a fat finger to cause so much grief.

Thanks for your help.
Sorry for the noise.
Always learning…

2 Likes

No worries!

You may also want to check out the SatNOGS IRC channel for even quicker feedback.

2 Likes

Note to self and others, if your station starts to operate outside of its usual behaviour, put it in testing mode.
I just got an email advising me as such… lesson learned, put in testing mode, then troubleshoot.

2 Likes

@thebaldgeek if you have some time please add your case in this wiki page to help others in the future. :wink:

1 Like

Having empty waterfall issues with another station today. #908 which I put in testing mode as soon as I got the email letting me know that it was returning abnormal results.

Looked at my own notes from the wiki and can see that I have enough hard drive.
I don’t have any rouge satnog clients running.

Looking the log file, I see this in a pattern;

Aug 15 19:58:46 raspberrypi satnogs-client[18169]: 2019-08-15 19:58:46,830 - apscheduler.executors.default - INFO - Job "get_jobs (trigger: interval[0:01:00], next run at: 2019-08-15 19:59:45 UTC)" executed successfully
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: 2019-08-15 19:59:26,018 - apscheduler.executors.default - INFO - Running job "spawn_observer (trigger: date[2019-08-15 19:59:26 UTC], next run at: 2019-08-15 19:59:26 UTC)" (scheduled at
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: 2019-08-15 19:59:26,108 - satnogsclient - ERROR - Cannot connect to socket 127.0.0.1:4533
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: Exception in thread Thread-384:
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: Traceback (most recent call last):
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:     self.run()
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:   File "/usr/lib/python2.7/threading.py", line 754, in run
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:     self.__target(*self.__args, **self.__kwargs)
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:   File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py", line 142, in _status_interface
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:     sock.bind()
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:   File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/commsocket.py", line 104, in bind
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:     self.s.bind((self._TCP_IP, self._TCP_PORT))
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:   File "/usr/lib/python2.7/socket.py", line 228, in meth
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:     return getattr(self._sock,name)(*args)
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: error: [Errno 98] Address already in use
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: 2019-08-15 19:59:26,149 - satnogsclient - ERROR - Cannot connect to socket 127.0.0.1:4533
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: Exception in thread Thread-383:
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: Traceback (most recent call last):
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:     self.run()
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:   File "/usr/lib/python2.7/threading.py", line 754, in run
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:     self.__target(*self.__args, **self.__kwargs)
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:   File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py", line 134, in _communicate_tracking_info
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:     self.send_to_socket(p, sock)
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:   File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py", line 180, in send_to_socket
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:     position = sock.send("p\n").split('\n')
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:   File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/commsocket.py", line 76, in send
Aug 15 19:59:26 raspberrypi satnogs-client[18169]:     self.s.send(message)
Aug 15 19:59:26 raspberrypi satnogs-client[18169]: error: [Errno 32] Broken pipe
Aug 15 19:59:27 raspberrypi satnogs-client[18169]: 2019-08-15 19:59:27,064 - apscheduler.executors.default - INFO - Job "spawn_observer (trigger: date[2019-08-15 19:59:26 UTC], next run at: 2019-08-15 19:59:26 UTC)" executed successfully
Aug 15 19:59:45 raspberrypi satnogs-client[18169]: 2019-08-15 19:59:45,909 - apscheduler.executors.default - INFO - Running job "get_jobs (trigger: interval[0:01:00], next run at: 2019-08-15 20:00:45 UTC)" (scheduled at 2019-08-15 19:59:
A

And I see this as well;

Aug 15 20:09:42 raspberrypi satnogs-client[18169]: Exception in thread Thread-385:
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: Traceback (most recent call last):
Aug 15 20:09:42 raspberrypi satnogs-client[18169]:   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
Aug 15 20:09:42 raspberrypi satnogs-client[18169]:     self.run()
Aug 15 20:09:42 raspberrypi satnogs-client[18169]:   File "/usr/lib/python2.7/threading.py", line 754, in run
Aug 15 20:09:42 raspberrypi satnogs-client[18169]:     self.__target(*self.__args, **self.__kwargs)
Aug 15 20:09:42 raspberrypi satnogs-client[18169]:   File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py", line 123, in _communicate_tracking_info
Aug 15 20:09:42 raspberrypi satnogs-client[18169]:     self.check_observation_end_reached()
Aug 15 20:09:42 raspberrypi satnogs-client[18169]:   File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py", line 168, in check_observation_end_reached
Aug 15 20:09:42 raspberrypi satnogs-client[18169]:     self.trackstop()
Aug 15 20:09:42 raspberrypi satnogs-client[18169]:   File "/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/observer/worker.py", line 164, in trackstop
Aug 15 20:09:42 raspberrypi satnogs-client[18169]:     os.killpg(os.getpgid(self._gnu_proc.pid), signal.SIGINT)
Aug 15 20:09:42 raspberrypi satnogs-client[18169]: OSError: [Errno 3] No such process

I have not rebooted as I don’t want to lose the passes that I have if I can.

Thanks for your help once again.