Hello.
I am new to the community - so please correct me if I´m wrong
It seems that only demodulated audio is stored for an observation.
This causes imho two major problems:
If the satellite frequency is off-center, the data payload is lost, especially for CW
non-standard modulations can´t be decoded from the stored data, so the observation is useless, although stored
Is it possible to have a raw audio stored of the band around the RX frequency?
Could be implemented using existing libs, for example:
If the bandwidth of the signal at Frequency f is 30kHz, demodulate as USB with Frequency f-15kHz and bandwidth 30kHz. The resulting audio would contain all the information needed to decode unknown data at later times.
The problem right now is 75% storage space issues and 25% bandwidth considerations. We are up to over 1,000 network observations being collected per day now (w00t!). To store the raw IQ would be too much data to keep around on the network side of things. Additionally, we have some stations who are either on metered or slow network connections and this could prove problematic for them. The latter problem could be solved by making this an “optional” contribution, of course.
That said, there is a setting on the client itself, ENABLE_IQ_DUMP that will store the raw IQ of an observation locally.
Now, I’m not saying that raw IQ handling is off the table completely, it is a frequent topic that comes up and we may figure out a way to make it feasible - I just wanted to call out the issues that make it difficult today.
We are getting close to improving this specifically for CW decoding, and the decoder itself has a decent amount of sway to it (enough to account for, say, PPM or TLE issues). Major frequency corrections should be made in db as we find them (and you are free to create an account there and submit corrections as you see them!)
Maybe a small change could solve some aspects of this issue:
If you could optionally choose your demodulation scheme to DSB (or single side band?), you could keep the information in the signal intact for later use, plus this demodulation scheme is a standard which gnuradio can handle out of the box.
You could make CW, AM, FM or any other demodulation from these sidebands, without the need to have giant IQ-files.
Would be a real good thing, because 1000s observations every day are really valuable, it would be a pity to lose the information in them.
I´m pretty sure that (lossless) compression could bring the DSB files into similar size than actual demodulated WAVs.
Cheers, Paul
PS: Maybe saving a single sideband where the received signal fits in will work better, alternatively saving a compressed I/Q file where the desired signal has been bandpass filtered (e.g. FIR), many ideas there.
This maybe a good idea to implement that into the well discussed “scanning mode”. I’ll try to find some time to implement kind of a proof of concept in the next days.
I will try to develop a lossless compressor for I/Q data from the captured WAV, so the complex baseband can be stored undisturbed to very small file sizes. Bandwidth will be of course limited to most common signals - let´s say +/- 20kHz from center.
In case this works out, how many kilobytes/second would be acceptable for SatNogs to store?
My guess is that the easiest way to store IQ data is to make the OGG files stereo and put the in-phase and quadrature signals in the left and right channel respectively. You then also get the Vorbis lossy compression, but that may be sufficient for most purposes. Currently the OGG encoder in gr-satnogs only takes floats as inputs, so it would have to be changed to handle complex input data.
You´re right. One could try out different compressors, maybe also FLAC as lossless.
As they are made mostly for music, the lossless ones could squeeze it only to 65%. The lossy ones could destroy the phase information in the IQ-data, but it´s worth a try.
I´d like to implement one myself based on Huffmann/RLE and setting a treshhold for noise to avoid storing empty spaces.
Currently I´m working on a signal processing routine to extract the desired band from the IQ-baseband before starting with compression.
Direct floating point complex I/Q downmix from a high-bandwidth file 8/16bit to 0 Hz and bandpass filtering works.
Bandwidth of +/- 20 kHz would take 80kb/sec uncompressed.
Now I´ll take care of lossless compression. Lossless because it should also work for weak signals, probably buried in noise, which someone eventually could be able to decode even years after the data has been stored on SatNogs servers.
I won´t use existing codecs, because they don´t fit into the typical signature of the RX signals, where there is mostly noise around a pattern. I try to squeeze it at least to half, hopefully much more. Will use Huffmann, RLE etc. and blockwise reading/writing to achieve a fast processing with low memory consumption on a Pi. Everything just C code - no special libs needed.
Very motivated now. Let´s see how close it comes to the 10 kb/sec used now.
Tho it would be nice if we did have some sort of SSB demodulater. So that it could be used with the FSK and a few other modulation types to allow them to be demodulated later from audio?
If the raw RF I/Q is available, you can use any demodulator on it. You can load it into HDSDR for example.
Or use own demodulation on it. I/Q demodulation algorithms are far easier to do on complex wave than on real wave - the only problem now is that the desired band can´t be efficiently extracted and saved.
If it can, new horizons will open.
SSB is just the imaginary part of the I/Q signal in time domain, so from my understanding information will get lost.
You can restore CW and AM from SSB though. For other signals I´d need to look for the demodulation formulas.
I am not that deep in signal theory - over 20 years that I heard some lectures :-D, so please correct when I´m talking BS.
I am not from signal domain either but I have seen some algorithm to do SSB demodulation
and I think it is possible to reverse it. Some gurus should have ideas about it.
SSB can be taken from both the real and the imaginary part, there is no difference there and no information is lost. In such case, the sideband is decided by the filtering. There are also other methods to demodulate SSB, though probably usable only when complex filtering is unfeasible.
You can generate complex I/Q from a real signal, after all, that’s how the complex I/Q is generated from “real RF” in the first place. In fact, many soundcard based modems start with transforming the real audio signal into complex I/Q. A common way to do that is to use a Hilbert transform.
Useful, yes, but it isn’t really a demodulator. If we use a complex bandpass filter to select the desired sideband like we do in gr-satnogs, then SSB demodulation doesn’t require anything else than taking either the real or the imaginary part of the complex I/Q signal.
