Removal of auto-vetting when decoded data exists

[Acinonyx]

Vetting of observations has long been a source of misconceptions. A feature intended to get human feedback whether a signal actually exists in the waterfall has been abused to infer the status of a satellite or even the station. This has become even worse by the introduction of the auto-vetting. An observation which has decoded data is automatically vetted as good even when the decoded data and waterfall is just noise. Building onto wrong assumptions, we did try to eliminate the auto-vetter false positives by removing the “noise frames”, making the demodulators/decoders stricter. Nevertheless, this proved to be a poor choice and a fix to a problem at the wrong level. Some transmitters have become undecodable recently due to this change which affected operations on a few satellites.

We are about to correct this issue and revert back to looser decoders. Decoded data will be accepted, either valid or not. To alleviate some of the vetting misconceptions, we are also about to remove the auto-vetting feature. Let me remind that the vetting is supposed to be done based on the existence of a signal in the waterfall and not the existence or validity of the decoded data.

P.S. Check this year old discussion on vetting and the need for a reform: Observations vetting - Need for a reform

[lu7did]

Currently auto-vetting isn’t available for CW signals, but automatic decoding is.

I do concentrate my pass collection priorities on satellites doing CW telemetry, at least on some modes of operation.

I do observe most of the decoding are glibberish, but when resorting to the actual audio the signal is strong. As a frequent contester I can decode fairly fast CW signals under very marginal conditions, have play with automatic decoding algorithms as well.

The dynamics of the CW decoding is that it requires at least 6 dB of SNR to be decoded comfortably by ear (marginal receiving with great effort can be lower than that) but this is using the ear-brain combination which is known to add some advantage on the equation. Some claims this advantage is as high as 10 dB, by experience with decoding algorithms I would be more conservative and say it’s circa 6 dB.

So, automatic decoding algorithms should have a strong barrier on a SNR of 12 dB, being able to copy above and not below.

Obviously SNR goes up as the bandwidth goes down, but there is a practical limit on the nature of the DSP filters required for very narrow processing, specially with FIR filters. For slow signals a 15-25 Hz filter should be enough, and most satellite’s telemetry data is at 10-15 wpm or so to increase the likelihood of being took by novice operators.

The signals I’m hearing on the pass sound track is for moments quite solid, well above the 12 dB level (by trained ear) still the decoder reports plenty of “E” and “T”, which is the fingerprint of a filtering problem as noise is usually confusing algorithms to be either Es or Ts (perhaps also Is or Ms showing fast or slow adapting algorithms).

Modern algorithms override this condition with the so called “bayesian” or “adaptive” filters, some tests with Kalman filters also has been performed with success. Examples of actual implementations are at the top of the food chain CW-Skimmer (on top of which the global Reverse Beacon Network has been built, reporting thousand of spots per minute worldwide), and also CWGet and FLDigi, pretty much the rest are either copycats of the above or old generation “length counting” algorithms. Algorithms are not public, but the general principles are kind of known.

I’m interested into understanding the architecture of the satnogs decoder in general, the CW decoder in particular and ways to evaluate (even test on my own) some alternatives.

¿Is there any pointer to documentation where a description of these aspects of the system are commented? Or perhaps pointers to discussions and forums? Starting afresh on a topic on this system on some topic pretty much looks like trying to find a needle on a haysack.

73 de Pedro, LU7DID

[xlevel]

Is vetting an opportunity to leverage the power of the new cloud Vision AI APIs I wonder?
It is something I have been thinking about, having seen examples of business using these APIs to identify everything from tumors too broken insulators on power lines.

[cshields]

Let’s make an API endpoint for vetting. DB could, theoretically, call back and vet an observation as good once data passes a decoding struct.

[cshields]

Already filed (yay) as https://gitlab.com/librespacefoundation/satnogs/satnogs-network/-/issues/724

[lu7did]

Modern CW algorithms uses learning algorithms that could benefit from AI tools. See http://ag1le.blogspot.com/2020/04/new-real-time-deep-learning-morse.html for further details.

If the purpose is to identify if “some” valid signal is present these algorithms are more than adequate once properly trained, not even requiring a whole lot of computing resources. To actually perform some decoding is a different matter, but limited success can be obtained, at least with some modes.

73 de Pedro, LU7DID

[csete]

Are there any numbers for how bad the auto-vetting was? With 5500 observations per day, the chance for making mistakes while manually vetting is IMO non-negligible.

[Acinonyx]

This is a change towards a new model in which vetting is only the human input of whether there is signal in the waterfall and does not indicate the overall result of the observation. It does require more UI changes to make it more clear.

The status of a satellite or ground station is dynamic and should (at least) depend on:
a) multiple human vetting input
b) whether there is decoded data or not
c) error detecting methods of transmitter mode (or absence of them)
d) quality of the decoder used
e) statistics of older observation results

[Acinonyx]

The problem is that we cannot find those number unless we make this change! Almost nobody vets an already vetted observation. So there is no indication if the decoded data came from a signal visible in the waterfall or it is just decode noise.

[lu7did]

The way to handle hundreds of vettings per day is very rudimentary with the current GUI. Ctl+G and Ctl+F are close keys and it’s not unthinkable to be pressed by mistake, creating a potential whole lot of Tipe I/II problems in the current vetting system.

But the need for an improved vetting GUI or the availability of an API to programatically at the very least be able to fail out passes without a Waterfall/Sound is still missing in the feature list.

73 de Pedro, LU7DID

[mat]

@lu7did

I’m interested into understanding the architecture of the satnogs decoder in general, the CW decoder in particular and ways to evaluate (even test on my own) some alternatives.

You made me curious, so I’ve just had a look. The CW flowgraph at satnogs-flowgraphs/generic/cw_decoder.grc looks like this in GnuRadio Companion:

The code for the actual CW decoder is at gr-satnogs/lib/cw_decoder.cc and other files nearby. Notably cw_decoder_priv.cc lines 132-137 seems to sum up the general algorithm:

/**
 * The decoder work as follows. The moving sum of the FFT triggers are passed.
 * This moving sum should exchibit a plateu when the CW is on. Depending on the
 * width of the plateu, it could be a dot or dash. In addition during a plateu
 * a small standard deviation should be observed.
 */

@lu7did

¿Is there any pointer to documentation where a description of these aspects of the system are commented? Or perhaps pointers to discussions and forums? Starting afresh on a topic on this system on some topic pretty much looks like trying to find a needle on a haysack.

I’m not sure there’s much documentation other than the code, but there is some info on getting started with dev on the wiki under [SatNOGS_Client_Development](https://wiki.satnogs.org/SatNOGS Client Development) and other pages on the Contribute menu.

It should be relatively straightforward to record an IQ file and feed it back into the above flowgraph before modifying the decoder to try alternative algorithms.