Waterfall and audio files not uploaded [solved]

I’m running the latest git version of satnogs-client on a PC using OpenSUSE Leap 42.3. The data files created by the observer are in /tmp/.satnogs/data. They have not been uploaded, and I believe that the reason can be found in these lines from the log:

2018-05-11 21:07:17,357 - apscheduler.executors.default - ERROR - Job “post_data (trigger: interval[0:02:00], next run at: 2018-05-11 19:09:17 UTC)” raised an exception
Traceback (most recent call last):
File “/usr/lib/python3.4/site-packages/apscheduler/executors/base.py”, line 125, in run_job
retval = job.func(*job.args, **job.kwargs)
File “/usr/local/src/satnogs-client/satnogsclient/scheduler/tasks.py”, line 92, in post_data
for fil in os.walk(settings.SATNOGS_OUTPUT_PATH).next()[2]:
AttributeError: ‘generator’ object has no attribute ‘next’

I have explicitely defined the environment variable SATNOGS_OUTPUT_PATH to point to /tmp/.satnogs/data.

Any suggestion as to what I can do,


SATNOGS_OUTPUT_PATH is already defined in settings.py.
I think (I am a bit new here), that you should not have to define it.

What I think is your problem is a compatibility issue about
os.walk between python2 and 3. The idiom …next()[2]
does not seem to work in python3.

May be you could try with python2 to confirm the diagnostic.


this idiom for directory traversal seems to work with python3:

for root, dirs, files in os.walk(settings.SATNOGS_OUTPUT_PATH):
for fil in files:

or better, but without walk, the following idiom does not recurse through subdirs
as it was the case before:
for file in os.listdir("."):
if os.path.isfile(file):

It works under python2.7 and python3.4.

This stresses a question, for which I so far have been unable to find an answer to: Which version of Python is satnogs-client actually written for? version 2 or version 3?

The installation instructions talks about setting up a virtual environment, but did not specify what should go into this environment. I couldn’t find the program ‘mkvirtualenv’ but managed somehow anyway to get something set up. That environment is a Python 3 environment, whereas OpenSUSE still sticks to Python2.

So what is the best setup really?

On Ubuntu, it works well with python2.7.

I got the mkvirtualenv bash function from the python-virtualenv
package, it seems to be available on Suse.

I guess its better I expand a bit on how I got the virtual environment working :wink:
I installed virtualenvwrapper with pip. Then I executed this command:
source $(which virtualenvwrapper.sh)
and could now execute
‘mkvirtualenv satnogs-client -a’ and ‘workon satnogs-client’.

I guess I should alway do this before I work on the satnogs-client files?

Please don’t ask me how I found this out. I’m very bad at keeping logs. :frowning:

Only “workon satnogs-client” is useful after the environment has been made

Yes of course. My stupid.

I have made the error in tasks.py disappear. I used your first suggestion, so the relevant lines now read
for root, dirs, files in os.walk(settings.SATNOGS_OUTPUT_PATH):
for fil in files:
file_path = os.path.join(*[settings.SATNOGS_OUTPUT_PATH, fil])
print("Looking at: ", fil)
if (fil.startswith(‘receiving_satnogs’) or
fil.startswith(‘receiving_waterfall’) or
fil.startswith(‘receiving_data’) or
os.stat(file_path).st_size == 0):

(you will probably have to imagine the indentation)

However, nothing is uploaded yet. :unamused: The files creating during a pass (the ‘receiving…’ files) are still left in /tmp/.satnogs/data.

But the ‘continue’ lines in the code also tells me that these files are being ignored. Some other process must the create the files that do get uploaded. The question of course is ‘which?’ So the search continues.

Anyway, you helped me get rid of the error and thank you for that.

You will notice that I have put a print line into the code. I did this because I can’t get my head around satnog-client’s logging system. I’ll raise that question in a separate thread.

The conversion (or maybe just a renaming) of the observation files after a completed task is done here:

1 Like

And “preformatted text” like codelines need 4 spaces infront (not a very nice solution):

#include  <iostream>
#include  <iomanip>

using namespace std;

int main()
    //    cout << "0x" << setfill('0') << setw(4) << hex << 0 << endl;
    cout << std::hex << std::setw (2) << std::setfill('0')
    << (uint32_t)10 << std::showbase << " ";

…and indeed not very useful :\

Filenames must have a correct naming to be uploaded to the network, for example:
-rw-rw-r-- 1 satnogs satnogs 4337800 mai 13 22:47 receiving_waterfall_134755_2018-05-13T20-45-38.dat
-rw-rw-r-- 1 satnogs satnogs 3792896 mai 13 22:47 receiving_satnogs_134755_2018-05-13T20-45-38.out
and the ident number must correspond to a registered one, I think.

When something went wrong, some steps may have been missed.
It is better to start again after the correction. Is it your case?
May be you can come back to your data after everything has come back under control.

And I almost forgot, have you started satnogs-network?

1 Like

There seems to be a lot of differences between the master and 0 branches.
It’s a bit surprising.

The 0-branch is the dev branch! Expect things to fail for any reason :smiley:

If the 0 branch is the dev, Why are tags on the 0 branch?
There has recently been a call to install version 0.6.1 and it is on
the 0 branch (dev) and not on master.

Which branch is considered stable enough for prod then?

I have at present these files in /tmp/.satnogs/data:

-rw-r--r-- 1 bent users 250871808 maj 13 20:24 receiving_satnogs_134569_2018-05-13T14-30-50.out
-rw-r--r-- 1 bent users    155648 maj 14 20:23 receiving_satnogs_135326_2018-05-14T14-06-47.out
-rw-r--r-- 1 bent users    147456 maj 14 20:27 receiving_satnogs_135327_2018-05-14T14-25-46.out
-rw-r--r-- 1 bent users 452775300 maj 13 20:24 receiving_waterfall_134569_2018-05-13T14-30-50.dat
-rw-r--r-- 1 bent users  50782600 maj 14 20:29 receiving_waterfall_135326_2018-05-14T14-06-47.dat
-rw-r--r-- 1 bent users  56157700 maj 14 20:29 receiving_waterfall_135327_2018-05-14T14-25-46.dat

The ones for observations 135326 and 135327 are recorded after I updated tasks.py. They correspond to the ‘needs vetting’ observations on my ground station’s page

I thought that satnogs-network is what I connect to when I schedule observations etc. Should I have a local copy of that running alongside my client?

According ‘git status’ I’m on the master branch. Is that OK?

I have checked about satnogs-network. I am not sure if it is needed at this point.

After completion of the observation the “receiving_” prefix is dropped.
The post method is waiting these files begin with
satnogs, waterfall or data to send them on the network.

satnogs-network is the instance that runs on:

There’s no need to run it locally!

If there have been no more commits, master branch is ok, because it is the same as 0.6.1, otherwise do a

$ git checkout 0.6.1

Don’t forget to update gr-satnogs to v1.3

@oz6bl: I cannot find the point why you are trying to manipulate the code - is it because you want to use python3 or is it just because “it’s not working” for you?

1 Like

in the other thread

there is a discussion about another case of pile up in tmp dir (with real gurus :slight_smile: )
It may not be related but have you seen this thread?

I continue to follow in case I can dig something from the sources.