We would definitely store and offer the kaitai structs in db - the issue is that the python decoder compiled by the ksc compiler does not offer a function to retrieve the data aside from addressing the object attributes directly - and we aren’t going to know what those attributes are… Say for instance we have a class instance called decoder, there could be decoder.battvolt1 and decoder.battvolt2 for one satellite, but not for another… so I was initially going to scrape these by going over the instance’s __dict__. This won’t work for more complex structures…
So, I see the process changing as such: someone generates the kaitai struct for a satellite, then compiles that into the python decoder. An additional step will need to be made to add a function to that decoder script which we will then call (which would then return {‘battvolt1’: ‘3.3’, ‘battvolt2’: ‘3.2’ …})
I’m open to other ideas here, I wasn’t too happy with the __dict__ method to begin with.
I think it is doable with __dict__. We can safely drop all the “private” attributes. Then we can check if any of the are instances of a subclass of KaitaiStruct and iterate again over those until we read all the tree.
I think the problem is how to properly serialize a structure into InfluxDB without adding any restrictions on how to write a Kaitai YAML.
with the right recursion, that might just cover it…
now… we still have some satellites that may break the kaitai mold (FOX DUV for one)… but, done right, we can still have custom scripts for them that plug into this model.
In the spirit of “release early, release often” we have a dev instance of our dashboard framework up and running. It decoded all of its data from db-dev.satnogs.org and will continue to do so for anything that comes in to the API.
Some things to note:
this will only have data for 8/1-8/19 (with the exception of any newer data we get to db-dev.satnogs.org/api)
so far we have working decoders for UNISAT-6, CAS-4A, CAS-4B. There will be more to come!
don’t expect this for prod anytime soon, as we have some infrastructure work to take care of before we have the resources to handle this at a larger scale.
I envision the creation of new decoders (kaitai struct schemas) and dashboards to be a crowdsourced activity. Right now its a little rough but we’re getting there.
If you would like to try your hand at creating some dashboards and playing with the data, let me know. I have some rough documentation available but I myself am no grafana expert, and would love for someone who is to jump in and help.
To match the metrics in grafana up you will need to reference the kaitai structures for each satellite, attached earlier in this thread.
Hi!
Yay, finally SatNOGS is able to close the decoding gap! Really fun to see what data lies dormant in the db.
Based on the documentation given by @sputnix in their Telemetry Viewer I wrote a kaitai struct for Siriussat-1 and -2 (beacon & extendedbeacon, see Kaitai struct for siriussat). To help in generation of the struct I wrote the python script siriussat_json2kaitai
A few remarks based on this experience:
To make it easy to collaboratively create the kaitai structs, we should consider to store them in a VCS/git-repository. There will be changes, issues, etc with the structs which can be handled nicely with a gitlab-repo.
We need a policy for the attribute identifiers.
Kaitai expects ^[a-z][a-z0-9_]*$ for the identifier. Beside this hard limitiation, it has to be decided how exactly we adopt the identifier from given documentation. The original identifier can be kept with the kaitai key -orig-id (see User Guide: ksy-documentation)
Two examples:
A documentation provides the flag for the status of an heater, called fl. Should we adopt this identifier or the more descriptive flag_heater?
When the documentation provides identifiers for subsystems, should we use a prefix or postfix (solar1_U, U_s1, Us1, U_solar1,solar1_voltage`)?
We should try to always link/provide the documentation on which the kaitai structs are based to be able to verify/fix/understand them later. Specifying references to the original documentation can be done with the kaitai key doc-ref. We could also consider to save a copy of the documentation together with our kaitai structs to prevent it to get lost (see User Guide: ksy-documentation).
Another update… If a satellite has a struct (or multiple structs even) associated with it, they will appear on the satellite’s page. You can see an example of this here
More decoders are making their way in thanks to @kerel and more testing is ongoing. Thanks for all of the feedback I’ve gotten so far!
Not necessarily… a single transmitter may transmit multiple types of frames, and we should also not assume that a frame type is limited to a single transmitter (say for instance, a frame is typically sent via UHF but then for testing purposes gets shifted to S band once in a while)
I used the original db model for this, “Telemetry” and the only association is with the satellite itself… and if you think about it, SiDS frames that come in to db are only associated with a satellite, not with a transmitter, so any further association isn’t necessary (IMO).
I almost agree. The struct should not hold any relations to the transmitters. But, the transmitters should have a unidirectional relation of one to many (possible) structs.
