Station not uploading some artifacts, ran out of space

dereksgc · August 15, 2022, 4:41am

Hello, yesterday I set up my station for testing outside, after solving some gain issues it appeared to be working well, but now after I let it run overnight there are numerous failed observations or observations that only have audio uploaded. Looking into the log showed that the satnogs client restarted in some instances, and now it seems to have ran out of space on the device, which shouldn’t have happened considering it has a 64 GB SD card available (see the log excerpt below).

https://network.satnogs.org/observations/?future=0&norad=&observer=&station=2571&start=2022-08-14+00%3A00&end=&transmitter_mode=

Any ideas how to fix this?

Aug 15 02:29:25 raspberrypi satnogs-client[3657]: rot_init called
Aug 15 02:29:25 raspberrypi satnogs-client[3657]: dummy: _init called
Aug 15 02:29:25 raspberrypi satnogs-client[3657]: rot_register (1)
Aug 15 02:29:25 raspberrypi satnogs-client[3657]: rot_register (2)
Aug 15 02:29:25 raspberrypi satnogs-client[3657]: dummy_rot_init called
Aug 15 02:29:25 raspberrypi satnogs-client[3657]: rot_open called
Aug 15 02:29:25 raspberrypi satnogs-client[3657]: dummy_rot_open called
Aug 15 02:29:25 raspberrypi satnogs-client[3657]: rot_get_position called
Aug 15 02:29:25 raspberrypi satnogs-client[3657]: dummy_rot_get_position called
Aug 15 02:29:25 raspberrypi satnogs-client[3657]: rot_set_position called
Aug 15 02:29:25 raspberrypi satnogs-client[3657]: dummy_rot_set_position called: 4.65 0.98
Aug 15 02:29:32 raspberrypi satnogs-client[3657]: Found Rafael Micro R820T tuner
Aug 15 02:29:32 raspberrypi satnogs-client[3657]: [INFO] Opening Generic RTL2832U OEM :: 00000001...
Aug 15 02:29:32 raspberrypi satnogs-client[3657]: Found Rafael Micro R820T tuner
Aug 15 02:29:32 raspberrypi satnogs-client[3657]: [R82XX] PLL not locked!
Aug 15 02:29:32 raspberrypi satnogs-client[3657]: [INFO] Using format CF32.
Aug 15 02:29:33 raspberrypi satnogs-client[3657]: Allocating 15 zero-copy buffers
Aug 15 02:41:42 raspberrypi satnogs-client[3657]: Found Rafael Micro R820T tuner
Aug 15 02:41:42 raspberrypi satnogs-client[3657]: [INFO] Opening Generic RTL2832U OEM :: 00000001...
Aug 15 02:41:43 raspberrypi satnogs-client[3657]: Found Rafael Micro R820T tuner
Aug 15 02:41:43 raspberrypi satnogs-client[3657]: [R82XX] PLL not locked!
Aug 15 02:41:43 raspberrypi satnogs-client[3657]: [INFO] Using format CF32.
Aug 15 02:41:43 raspberrypi satnogs-client[3657]: Allocating 15 zero-copy buffers
Aug 15 03:02:04 raspberrypi satnogs-client[3657]: Found Rafael Micro R820T tuner
Aug 15 03:02:05 raspberrypi satnogs-client[3657]: [INFO] Opening Generic RTL2832U OEM :: 00000001...
Aug 15 03:02:05 raspberrypi satnogs-client[3657]: Found Rafael Micro R820T tuner
Aug 15 03:02:05 raspberrypi satnogs-client[3657]: [R82XX] PLL not locked!
Aug 15 03:02:05 raspberrypi satnogs-client[3657]: [INFO] Using format CF32.
Aug 15 03:02:07 raspberrypi satnogs-client[3657]: Allocating 15 zero-copy buffers
Aug 15 03:13:19 raspberrypi systemd[1]: satnogs-client.service: Main process exited, code=killed, status=9/KILL
Aug 15 03:13:19 raspberrypi systemd[1]: satnogs-client.service: Failed with result 'signal'.
Aug 15 03:13:20 raspberrypi systemd[1]: satnogs-client.service: Service RestartSec=100ms expired, scheduling restart.
Aug 15 03:13:20 raspberrypi systemd[1]: satnogs-client.service: Scheduled restart job, restart counter is at 5.
Aug 15 03:13:20 raspberrypi systemd[1]: Stopped SatNOGS client.
Aug 15 03:13:20 raspberrypi systemd[1]: Started SatNOGS client.
Aug 15 03:51:35 raspberrypi satnogs-client[3848]: rot_init called
Aug 15 03:51:35 raspberrypi satnogs-client[3848]: dummy: _init called
Aug 15 03:51:35 raspberrypi satnogs-client[3848]: rot_register (1)
Aug 15 03:51:35 raspberrypi satnogs-client[3848]: rot_register (2)
Aug 15 03:51:35 raspberrypi satnogs-client[3848]: dummy_rot_init called
Aug 15 03:51:35 raspberrypi satnogs-client[3848]: rot_open called
Aug 15 03:51:35 raspberrypi satnogs-client[3848]: dummy_rot_open called
Aug 15 03:51:35 raspberrypi satnogs-client[3848]: rot_get_position called
Aug 15 03:51:35 raspberrypi satnogs-client[3848]: dummy_rot_get_position called
Aug 15 03:51:35 raspberrypi satnogs-client[3848]: rot_set_position called
Aug 15 03:51:42 raspberrypi satnogs-client[3848]: Found Rafael Micro R820T tuner
Aug 15 03:51:42 raspberrypi satnogs-client[3848]: [INFO] Opening Generic RTL2832U OEM :: 00000001...
Aug 15 03:51:42 raspberrypi satnogs-client[3848]: Found Rafael Micro R820T tuner
Aug 15 03:51:42 raspberrypi satnogs-client[3848]: [R82XX] PLL not locked!
Aug 15 03:51:42 raspberrypi satnogs-client[3848]: [INFO] Using format CF32.
Aug 15 03:51:43 raspberrypi satnogs-client[3848]: Allocating 15 zero-copy buffers
Aug 15 04:05:15 raspberrypi satnogs-client[3848]: apscheduler.executors.default - ERROR - Job "spawn_observer (trigger: date[2022-08-15 02:51:35 UTC], next run at: 2022-08-15 02:51:35 UTC)" raised an exception
Aug 15 04:05:15 raspberrypi satnogs-client[3848]: Traceback (most recent call last):
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/PIL/ImageFile.py", line 510, in _save
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     fh = fp.fileno()
Aug 15 04:05:15 raspberrypi satnogs-client[3848]: AttributeError: '_idat' object has no attribute 'fileno'
Aug 15 04:05:15 raspberrypi satnogs-client[3848]: During handling of the above exception, another exception occurred:
Aug 15 04:05:15 raspberrypi satnogs-client[3848]: Traceback (most recent call last):
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/PIL/Image.py", line 2151, in save
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     save_handler(self, fp, filename)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/PIL/PngImagePlugin.py", line 1339, in _save
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     ImageFile._save(im, _idat(fp, chunk), [("zip", (0, 0) + im.size, 0, rawmode)])
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/PIL/ImageFile.py", line 525, in _save
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     fp.write(d)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/PIL/PngImagePlugin.py", line 1032, in write
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     self.chunk(self.fp, b"IDAT", data)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/PIL/PngImagePlugin.py", line 1019, in putchunk
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     fp.write(data)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]: OSError: [Errno 28] No space left on device
Aug 15 04:05:15 raspberrypi satnogs-client[3848]: During handling of the above exception, another exception occurred:
Aug 15 04:05:15 raspberrypi satnogs-client[3848]: Traceback (most recent call last):
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/apscheduler/executors/base.py", line 125, in run_job
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     retval = job.func(*job.args, **job.kwargs)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 65, in spawn_observer
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     observer.observe()
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py", line 210, in observe
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     self.plot_waterfall(waterfall)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/observer/observer.py", line 324, in plot_waterfall
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     waterfall.plot(self.observation_waterfall_png, vmin, vmax)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/waterfall.py", line 142, in plot
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     plt.savefig(figure_path, bbox_inches='tight')
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/matplotlib/pyplot.py", line 859, in savefig
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     res = fig.savefig(*args, **kwargs)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/matplotlib/figure.py", line 2311, in savefig
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     self.canvas.print_figure(fname, **kwargs)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 2217, in print_figure
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     **kwargs)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 1639, in wrapper
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     return func(*args, **kwargs)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py", line 512, in print_png
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     dpi=self.figure.dpi, metadata=metadata, pil_kwargs=pil_kwargs)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/matplotlib/image.py", line 1605, in imsave
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     image.save(fname, **pil_kwargs)
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:   File "/var/lib/satnogs/lib/python3.7/site-packages/PIL/Image.py", line 2155, in save
Aug 15 04:05:15 raspberrypi satnogs-client[3848]:     fp.close()
Aug 15 04:05:15 raspberrypi satnogs-client[3848]: OSError: [Errno 28] No space left on device
Aug 15 04:23:23 raspberrypi satnogs-client[3848]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
Aug 15 04:50:06 raspberrypi satnogs-client[3848]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.

jebba · August 15, 2022, 5:15am

It is probably writing to /tmp/.satnogs and it isn’t getting cleared. A reboot will clear /tmp, which will temporarily fix the problem. Then keep an eye on that directory.

dereksgc · August 15, 2022, 5:23am

I rebooted it and then looked in the logs, and now I’m getting an error immediately after boot by the looks of it;

-- Logs begin at Mon 2022-08-15 06:17:56 BST, end at Mon 2022-08-15 06:22:09 BST. --
Aug 15 06:18:00 raspberrypi systemd[1]: Started SatNOGS client.
Aug 15 06:18:09 raspberrypi satnogs-client[306]: satnogsclient.scheduler.tasks - ERROR - An error occurred trying to GET observation jobs from network
Aug 15 06:18:09 raspberrypi satnogs-client[306]: Traceback (most recent call last):
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     (self._dns_host, self.port), self.timeout, **extra_kw
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
Aug 15 06:18:09 raspberrypi satnogs-client[306]: socket.gaierror: [Errno -3] Temporary failure in name resolution
Aug 15 06:18:09 raspberrypi satnogs-client[306]: During handling of the above exception, another exception occurred:
Aug 15 06:18:09 raspberrypi satnogs-client[306]: Traceback (most recent call last):
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     chunked=chunked,
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     self._validate_conn(conn)
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     conn.connect()
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connection.py", line 309, in connect
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     conn = self._new_conn()
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connection.py", line 172, in _new_conn
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     self, "Failed to establish a new connection: %s" % e
Aug 15 06:18:09 raspberrypi satnogs-client[306]: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x721d7c30>: Failed to establish a new connection: [Errno -3] Temporary failu
Aug 15 06:18:09 raspberrypi satnogs-client[306]: During handling of the above exception, another exception occurred:
Aug 15 06:18:09 raspberrypi satnogs-client[306]: Traceback (most recent call last):
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     timeout=timeout
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/urllib3/util/retry.py", line 446, in increment
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     raise MaxRetryError(_pool, url, error or ResponseError(cause))
Aug 15 06:18:09 raspberrypi satnogs-client[306]: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='network.satnogs.org', port=443): Max retries exceeded with url: /api/jobs/?ground_station=2571&lat=49.
Aug 15 06:18:09 raspberrypi satnogs-client[306]: During handling of the above exception, another exception occurred:
Aug 15 06:18:09 raspberrypi satnogs-client[306]: Traceback (most recent call last):
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 172, in get_jobs
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     timeout=45)
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/api.py", line 76, in get
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     return request('get', url, params=params, **kwargs)
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/api.py", line 61, in request
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     return session.request(method=method, url=url, **kwargs)
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     resp = self.send(prep, **send_kwargs)
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     r = adapter.send(request, **kwargs)
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     raise ConnectionError(e, request=request)
Aug 15 06:18:09 raspberrypi satnogs-client[306]: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='network.satnogs.org', port=443): Max retries exceeded with url: /api/jobs/?ground_station=2571&lat=
Aug 15 06:18:09 raspberrypi satnogs-client[306]: apscheduler.executors.default - ERROR - Job "get_jobs (trigger: interval[0:01:00], next run at: 2022-08-15 05:19:09 UTC)" raised an exception
Aug 15 06:18:09 raspberrypi satnogs-client[306]: Traceback (most recent call last):
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/apscheduler/executors/base.py", line 125, in run_job
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     retval = job.func(*job.args, **job.kwargs)
Aug 15 06:18:09 raspberrypi satnogs-client[306]:   File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 177, in get_jobs
Aug 15 06:18:09 raspberrypi satnogs-client[306]:     response.raise_for_status()
Aug 15 06:18:09 raspberrypi satnogs-client[306]: UnboundLocalError: local variable 'response' referenced before assignment
lines 3-55/55 (END)

mfalkvidd · August 15, 2022, 5:35am

That error is common just after startup. The software tries to connect to the satnogs servers before the network is ready. It can be ignored.

dereksgc · August 15, 2022, 9:07am

Thanks for the info. It seems to be running better now after I cleared the temp folder manually and rebooted, but it already got another observation with audio only; SatNOGS Network - Observation 6334791
I can’t check the logs now, I will post it once I get back home

mfalkvidd · August 15, 2022, 9:31am

When you’re home, checking disk usage might be useful.
This is what my system looks like:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G  6.0G   21G  23% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  153M  1.8G   8% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
tmpfs           1.9G   12K  1.9G   1% /tmp/.satnogs
/dev/mmcblk0p1  253M   49M  204M  20% /boot
tmpfs           384M     0  384M   0% /run/user/1000
$ sudo du -ax /tmp | sort -nr | head
156     /tmp
116     /tmp/cache
72      /tmp/cache/tles_1272.txt
36      /tmp/cache/transmitters_1272.txt
8       /tmp/systemd-private-68...8-vnstat.service-93mILe
8       /tmp/systemd-private-68...8-systemd-timesyncd.service-276GRb
4       /tmp/.XIM-unix
4       /tmp/.X11-unix
4       /tmp/.Test-unix
4       /tmp/systemd-private-68...8-vnstat.service-93mILe/tmp

The default size limit for each tmpfs filesystem is half the memory size. I have a 4GB Raspberry Pi 4, so my tmpfs max size is about 2GB.

dereksgc · August 15, 2022, 10:49am

So that seems to be the root cause of the issue, the temp file directory is already at 37% use since a few hours ago, I guess it makes sense why it was saying it cant fit more on the device, it wasn’t filling up the SD card but rather just this significantly smaller location. This is a Pi Zero W so it is only limited to 214 MB, still that should be plenty for VHF satellites. Maybe I could add a post-obs script that always deletes the contents of the folder?

pi@raspberrypi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        59G  3.4G   53G   7% /
devtmpfs        182M     0  182M   0% /dev
tmpfs           214M  1.4M  213M   1% /dev/shm
tmpfs           214M  3.1M  211M   2% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           214M     0  214M   0% /sys/fs/cgroup
tmpfs           214M   79M  136M  37% /tmp/.satnogs
/dev/mmcblk0p1  253M   49M  204M  20% /boot
tmpfs            43M     0   43M   0% /run/user/1000

jebba · August 15, 2022, 10:56am

Fwiw, I see on my system it now isn’t deleting files in two out of three systems and /tmp is growing too.

mfalkvidd · August 15, 2022, 1:16pm

My station seems to use about 30MB for a 5 minute observation (excluding the 120MB raw iq, which isn’t saved by default). So the Raspberry Pi Zero’s 214 MB should be sufficient for 8-9 minutes long observations (depending a bit on the bandwidth and the amount of decoded data on the observed transmitter).

On my station I see no growth; /tmp/.satnogs/ is cleared after each observation.

dereksgc · August 15, 2022, 1:34pm

I got a suggestion from a friend to shortcut the temp directory from RAM to the SD card, it probably isn’t the best for the sake of the card’s health but I am hoping the Pi Zero setup is only temporary until I can get a bigger SBC and offload that directory to a USB drive. It seems like it is clearing the directory now, in my last reply it was 37%, now I checked again and it’s down to 34%, so I imagine the problems were caused by a particularly long satellite pass.
My original plan was to set up the station to receive HRPT, which ranges from 3 to 6 MHz in bandwidth and has a data rate 10x - 100x higher than APT, so I will have to reconsider that for the time being…

mfalkvidd · August 15, 2022, 1:50pm

If you comment out this entry in /etc/fstab and reboot, satnogs will use /tmp which (at least on my machine) resides on the sd card:

tmpfs /tmp/.satnogs tmpfs rw,nosuid,noexec,nodev,noatime 0 0

jebba · August 15, 2022, 5:53pm

I’m not sure if something has changed, but two of my systems are no longer deleting the waterfalls/oggs. Maybe I set a config wrong, idk.

kerel · August 15, 2022, 11:05pm

To confirm what has been said previously, /tmp/.satnogs/ should be clear after the observation eventually and this was not changed.

If the upload of waterfall/ogg files fails repeatedly, e.g. due to network issues or a reduced availability of SatNOGS Network then those files will not get deleted and create a backlog. This will eventually lead to tmp running out of space.

Ideally such a situation is detected before we run out of space, files are dropped and a warning is shown to the user (e.g. via the log). Unless such a process is implemented, this case needs manual detection & intervention.

@jebba: Did your stations stop deleting waterfalls/oggs despite successful uploads or are those results missing from the Observations in SatNOGS Network? From this thread only some stations seem to be affected, maybe based on the connection quality to SatNOGS Network. I’d like to get an idea of problem location: upload failing or upload succeeding but deletion being skipped

(OT: In the medium/long run the waterfall/ogg transport might get replaced by some message queue,see gr-satnogs#237, but congestion would still be possible and needs detection/handling).

jebba · August 16, 2022, 1:15am

The waterfalls are getting uploaded, e.g. observation 6340506.

The files are getting saved in /tmp/.satnogs/data/complete/. So maybe I told it to keep them there, but I don’t see where I enabled that. I should also note, I did upgrade to 1.7 and install gr-satellites recently, but I don’t think that’s related (?).

PE0SAT · August 16, 2022, 2:09pm

I had a similar problem some weeks ago, after a reboot this was solved.
My idea is, that one of the distribution updates was responsible.

But this is an assumption.

dereksgc · August 16, 2022, 3:30pm

Thank you all very much for the replies, indeed it seems like a reboot has solved the issue, and in the future I will just move the temp dir to SD card so it can fit longer passes. For now I have taken the station down again until I finish its power cable so it can be put back up permanently and hopefully taken out of testing

jebba · August 16, 2022, 11:42pm

I compared the stations. On the ones that were still growing disk size, I had apparently set this in satnogs-setup under Paths.

SATNOGS_COMPLETE_OUTPUT_PATH   *Define completed data path [/tmp/.satnogs/data/complete]

The default is just blank. I removed that.

kerel · August 17, 2022, 10:30am

The limit to 8-9 minutes of observation when using a Raspberry Pi Zero is good to know, I’m just not sure where to document it.

Thank you for checking your station @jebba, good to see this second problem solved.

I just added a note on the SATNOGS_COMPLETE_OUTPUT_PATH docs on why it’s default to empty is recommended and listed the observed grow rate by @mfalkvidd (see satnogs-client!599).