Hi all,
for some reason my station shows offline but the processes on the pi are all present as they should.
log gives no special indication. a reboot brings service back.
webinterface at port 5000 is responsive and shows network mode…
to dig deeper for a causing reason someone has an idea to look additional.
cos reboot will fix it for the moment and i’d like to find the root cause.
asking for some hints,
br Robert
hey Robert!
Which versions are you running? (gr-satnogs and satnogs-client) how did you set them up?
Can you post here the configurations? Minus the API key of course 
Are you on the dev website or production? I’m just getting started, I tried setting up with DEV and wouldn’t work, created an account on production site and works just fine?
hi @all
Short Info:
found the reason, the temp space ran out of space. Question if there is a chance to make it bigger on a pi?
Long Info:
I use the recent client from last friday (force an update via satnogs-setup client).
The box is behind an lte upstream attached to the productive database.
Rotor is attached via network rotorctld and worked for weeks.
Issues arrised since last week and file system check didn’t show issues as well.
os update was also done last friday (apt-get update/upgrade)
Config is as follows:
Configuration Key Value
BACKEND_FEEDER_PORT 5023
BACKEND_LISTENER_PORT 5022
CLIENT_LISTENER_UDP_PORT 5015
CURRENT_PASS_TCP_PORT 5005
DEFAULT_LOGGING {…}
DEFAULT_SQLITE_PATH /tmp/.satnogs/jobs.sqlite
DISABLE_DECODED_DATA False
ECSS_FEEDER_UDP_PORT 5031
ENABLE_IQ_DUMP False
GNURADIO_APT_SCRIPT_FILENAME satnogs_noaa_apt_decoder.py
GNURADIO_BPSK_SCRIPT_FILENAME satnogs_bpsk_demod.py
GNURADIO_CW_SCRIPT_FILENAME satnogs_cw_demod.py
GNURADIO_FM_SCRIPT_FILENAME satnogs_fm_demod.py
GNURADIO_FSK9K6_SCRIPT_FILENAME satnogs_fsk9600_g3ruh_ax25.py
GNURADIO_IP 127.0.0.1
GNURADIO_SCRIPT_FILENAME satnogs_fm_demod.py
GNURADIO_SCRIPT_PATH [’/usr/bin’, ‘/usr/local/bin’]
GNURADIO_UDP_PORT 16886
IQ_DUMP_FILENAME None
LD_DOWNLINK_LISTEN_PORT 5033
LD_DOWNLINK_RETRIES_LIM 5
LD_DOWNLINK_SMALL_TIMEOUT 2
LD_DOWNLINK_TIMEOUT 5
LD_UPLINK_LISTEN_PORT 5021
LD_UPLINK_TIMEOUT 2.5
RF_SW_CMD_OFF_CHAR_SEQ None
RF_SW_CMD_OFF_INT None
RF_SW_CMD_ON_CHAR_SEQ None
RF_SW_CMD_ON_INT None
RIG_FILE
RIG_MODEL
RIG_PTT_FILE
RIG_PTT_TYPE
RIG_SERIAL_SPEED
SATNOGS_ANTENNA None
SATNOGS_APP_PATH /tmp/.satnogs
SATNOGS_BB_GAIN None
SATNOGS_COMPLETE_OUTPUT_PATH
SATNOGS_DEV_ARGS None
SATNOGS_DOPPLER_CORR_PER_SEC None
SATNOGS_IF_GAIN None
SATNOGS_INCOMPLETE_OUTPUT_PATH /tmp/.satnogs/data/incomplete
SATNOGS_LO_OFFSET None
SATNOGS_NETWORK_API_POST_INTERVAL 2
SATNOGS_NETWORK_API_QUERY_INTERVAL 1
SATNOGS_NETWORK_API_URL https://network.satnogs.org/api/
SATNOGS_OUTPUT_PATH /tmp/.satnogs/data
SATNOGS_POST_OBSERVATION_SCRIPT None
SATNOGS_PPM_ERROR 46
SATNOGS_PRE_OBSERVATION_SCRIPT None
SATNOGS_REMOVE_RAW_FILES False
SATNOGS_RF_GAIN 31
SATNOGS_RIG_IP 127.0.0.1
SATNOGS_RIG_PORT 4532
SATNOGS_ROT_IP 172.x.y.100
SATNOGS_ROT_PORT 4533
SATNOGS_ROT_THRESHOLD 4
SATNOGS_RX_DEVICE rtlsdr
SATNOGS_SERIAL_PORT None
SATNOGS_SQLITE_URL sqlite:////tmp/.satnogs/jobs.sqlite
SATNOGS_STATION_ELEV 1641.0
SATNOGS_STATION_ID 36
SATNOGS_STATION_LAT 46.839
SATNOGS_STATION_LON 15.012
SATNOGS_VERIFY_SSL True
STATUS_LISTENER_PORT 5032
TASK_FEEDER_TCP_PORT 5011
WOD_UDP_PORT 5023
Interessting part is that jobs.sqlite doesn’t get updated anymore. somehow the cronjob went away…
Digging through the logs with journalctl -xe tells regarding a sql space issue. I narrowed it down to a simple ran put of space for the tempspace:
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: Exception in thread APScheduler:
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: Traceback (most recent call last):
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/usr/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: self.run()
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/usr/lib/python2.7/threading.py”, line 754, in run
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: self.__target(*self.__args, **self.__kwargs)
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/apscheduler/schedulers/blocking.py”, line 30,
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: wait_seconds = self._process_jobs()
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/apscheduler/schedulers/base.py”, line 979, in
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: jobstore.update_job(job)
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/apscheduler/jobstores/sqlalchemy.py”, line 10
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: result = self.engine.execute(update)
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py”, line 2075, in exe
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: return connection.execute(statement, *multiparams, **params)
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py”, line 948, in exec
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: return meth(self, multiparams, params)
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py”, line 269, in _ex
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: return connection._execute_clauseelement(self, multiparams, params)
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py”, line 1060, in _ex
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: compiled_sql, distilled_params
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py”, line 1200, in _ex
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: context)
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py”, line 1413, in _ha
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: exc_info
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py”, line 203, in rais
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: reraise(type(exception), exception, tb=exc_tb, cause=cause)
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py”, line 1193, in _ex
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: context)
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py”, line 507, in d
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: cursor.execute(statement, parameters)
Apr 15 19:57:59 satnogs-oe8rke satnogs-client[2716]: OperationalError: (sqlite3.OperationalError) database or disk is full [SQL: u’UPDATE apscheduler_jobs SE
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: 2018-04-15 19:58:00,261 - apscheduler.executors.default - ERROR - Job "get_jobs (trigger: interval[0:01:
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: Traceback (most recent call last):
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/apscheduler/executors/base.py”, line 125, in
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: retval = job.func(*job.args, **job.kwargs)
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/satnogsclient/scheduler/tasks.py”, line 171,
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: job.remove()
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/apscheduler/job.py”, line 103, in remove
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: self._scheduler.remove_job(self.id, self._jobstore_alias)
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/apscheduler/schedulers/base.py”, line 606, in
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: store.remove_job(job_id)
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/apscheduler/jobstores/sqlalchemy.py”, line 11
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: result = self.engine.execute(delete)
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py”, line 2075, in exe
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: return connection.execute(statement, *multiparams, **params)
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py”, line 948, in exec
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: return meth(self, multiparams, params)
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py”, line 269, in _ex
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: return connection._execute_clauseelement(self, multiparams, params)
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: File “/var/lib/satnogs/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py”, line 1060, in _ex
Apr 15 19:58:00 satnogs-oe8rke satnogs-client[2716]: compiled_sql, distilled_params
pi@satnogs-oe8rke:/tmp/.satnogs/data/files/EXT_WOD_LOG $ df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/root 15G 2.6G 12G 19% /
devtmpfs 460M 0 460M 0% /dev
tmpfs 464M 0 464M 0% /dev/shm
tmpfs 464M 18M 446M 4% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 464M 0 464M 0% /sys/fs/cgroup
tmpfs 464M 464M 0 100% /tmp/.satnogs
/dev/mmcblk0p1 42M 22M 20M 53% /boot
tmpfs 93M 0 93M 0% /run/user/1000
So upsizing the temp space should help. any ideas where to tweak that?
also a cleanup, compression and sql restart would be a help i guess.
sorry for long post but the trail might have collegues with similar issues.
br Robert
hi, since i still suffer on abbandoned files which are a left over in the temp space, i look for a hint how to manual resubmit or force an upload.
Also the teak to resize the temp space would be cool to know. my first idea of fstab didn’t do the trick…
asking for ideas and help,
br Robert, 73 de oe6rke
Hey @oe6rke,
Can you give us more details about these files?
Are these files inside /tmp/.satnogs/data/incomplete directory? If yes, then moving them one directory up (in /tmp/.satnogs/data/) will allow the uploading task to try to upload them.
Are these files in /tmp/.satnogs/data/? What type of files are they and with what size (ls -la /tmp/.satnogs/data/ will help you see their size). If there are .dat files with .png files of zero size then probably something is wrong with the waterfall generation.
Are these files in /tmp/ and they are crc fails files? Then you can remove them without any problem.
Thanks for the insights Freddy. indeed there were a Zero waterfall which blocked further processing and filled up the temp space until Zero space.
well, i Monitor it after upgrading to the latest Version. checking of file System showed no further indication. hope the ghost is gone via help of the update,
thx, br Robert
is there a way to Restart the waterfall Generation and upload process? have some .dat with size waiting in the data Directory which are untouched there.
any idea how to perform that manually?
tnx, br Robert
You can generate manually the waterfall by running
gnuplot -e "inputfile='./waterfall.dat'" -e "outfile='waterfall.png'" -e "height=1600" /usr/share/satnogs/scripts/satnogs_waterfall.gp
Both double and simple quotes are needed. waterfall.png should have the format of waterfall_<id_of_observation>_<date_in_iso_format>.png for example waterfall_133443_2018-05-12T03-19-11.png. .dat filenames should have the needed info.
If satnogs_waterfall.gp isn’t in /usr/share/satnogs/scripts/ check in /usr/local/share/ directory.
Let me know if you get any errors during the process, sometimes there are memory issues.