The new waterfall plotting mechanism is almost ready! There are still some improvements that have to been done, but are related with the display and the .png image creation. You can check it out on my fork https://gitlab.com/surligas/gr-satnogs/tree/waterfall
How it works:
Instead of the gnuplot it now uses the python matplotlib and generates directly the .png image at the end of the observation.
The C++ block waterfall_heatmap is now responsible to compute the power spectral density and map the resulting energy into int8_t values for every frequency bin of the FFT. These values are passed during runtime to the Python block waterfall_plotter. This block just appends the received values into a 2-D numpy array. For a 7-minute observation, this will take up to few dozens MB of memory. At the end of an observation, before the termination of the flowgraph GNU Radio automatically called the stop() method of each block. The stop() of the waterfall_plotter generates the waterfall image using the stored 2-D array and exits.
What to test:
We have to confirm that this method uses less memory than the gnuplot method during the image creation. I can confirm this for my x86 setup, but I did not tried it on a Raspberry.
If we are ok with the current setup, we can then test higher FFT values for better resolution. Moreover, we can test higher number of rows per seconds. Eventually, we could set it to such resolution that CW signals can visually be decoded, which with the existing gnuplot images is not possible.
I run satnogs_fm_demod.py --waterfall-file-path=/home/surligas/test.png and then I use Ctrl+C to terminate the flowgraph. After a while, the waterfall is ready.
It has nothing to do with loss of data from the network. jpeg does compression that leads to loss of data from the original image in terms of less bytes.
It’s like audio compression. A song on a lossless format may be 60-100 MB whereas an mp3 is about 3-4 MB. To achieve this compression mp3 cuts off some frequencies. You cannot reconstruct the original audio.
Why not? 2MB instead of 0.5 is not that much. Also .png is ways easier for possible feature visualization enhancements that the guys from the network may want to apply.
There are still quota limited ADSL plans!!! I see your concern, but the audio file for example requires ways more bandwidth. Reducing the observation time by a minute may give you significant less upload files.
Anw, this thread is not the proper one for such discussion.
I was trying to find any design docs or discussions, but haven’t found . I have more questions, like “why create waterfall at all”. Since all observations have audio file, it will be fairly easy create waterfalls at any time. Moreover they could be created on-demand on server, rather than on slow raspberry pi.
The difference between the generated waterfall and the one that you get from the audio is that the first one covers bigger bandwidth.
The goal behind this is to check if the frequency of a transmitter has been drifted so much that it gets outside the audio bandwidth and fix it if this behaviour is repeated.
You can check that by going to an observation and compare waterfall with the spectrogram of the audio in audio tab.
Totally agree on this. In general, audio may generated after some corrections, eg filtering, costas loop etc, whereas the spectrum waterfall is generated right after the doppler compensation.
Server side processing, should be avoided at all cost to my opinion. The server is only one (at least for now :P). On the other hand, distributing the workload, even in low-end processing units, is extremely efficient. It may take longer. That’s true. But these units most of the time are idle. Why not use them? One will argue that it should be ready for a next observation. That’s also true. But for now,you can just prioritize observations. In the future if the observations are dense enough, we can make the client to postpone the processing and upload until it becomes idle.
But again, this thread was about testing the new waterfall. You can create a new one about the data size and your concerns.
I see your concern, but the audio file for example requires ways more bandwidth. Reducing the observation time by a minute may give you significant less upload files.
. I have more questions, like “why create waterfall at all”. Since all observations have audio file, it will be fairly easy create waterfalls at any time. Moreover they could be created on-demand on server, rather than on slow raspberry pi.