Working on "Telemetry Dashboards"

Hey all,

as a lot of you already noticed I am working on telemetry dashboards for some days now.
As a feedback to the whole community and to start some discussions and maybe get some useful hints on doing this and that I wanted to summarize my work in this thread.

The start of interpreting telemetry frames already stored in the data tab of an observation has been requested already some time ago. In these days there was a decision to use kaitai struct for the telemetry decoding. @cshields Corey already did some experiments with it on CAS-4A/B and I started very optimistically with the telemetry format of one of the AMSAT FOX satellites. Corey’s approach went very well where mine turned out to be a bit too advanced to start with (no worries, it is still work in progress :wink: ). Corey also did “some” work on “Closing the loop” between satnogs-network and satnogs-db to get frames from SatNOGS stations into the db (not only SiDS frames from other frontends). He also added grafana to visualize the data inside that frames into satnogs-db. As a proof of concept there were some dashboards made available to see if there are some showstoppers with the concept that was chosen (kaitai, influxdb, grafana, etc.).

To get back to my efforts I decided to dig deeper into that kaitai stuff by going one (or even more) steps back and tried to decode the AX.25 header in the dataframes oduced in satnogs-network. I started with a simple kaitai struct, following the AX.25 documentation on TAPR. The first one looked liek the following:

meta:
  id: ax25_decoder
  endian: be
seq:
  - id: ax25_header
    doc-ref: 'https://www.tapr.org/pub_ax25.html'
    type: hdr
    size: 16
  - id: ax25_info
    type: str
    encoding: ASCII
    size-eos: true
types:
  hdr:
    seq:
    - id: dest_address
      type: str
      encoding: ASCII
      size: 6
    - id: dest_ssid
      type: u1
    - id: src_address
      type: str
      encoding: ASCII
      size: 6
    - id: src_ssid
      type: u1
    - id: ctrl
      type: u1

Which gave pretty good results if you keep reading on the AX.25 docs: source and destination callsign are 7-bit ASCII charcters and are enhanced by a so called extension bit. This is why you won’t ever get readable callsigns without converting the raw binary data in this fields. It took a while to get behind that and in a first step I was satisfied to convert these values by hand, masking the single charcacters and shift them one to the right

  c = (c & 0xFE) >> 1

Some time later I got a really good hint from one of the kaitai-guys to use a built-in process called “ror” (rotate right) but I just didn’t know how to use this and startet poking around. As one of the last things I did these days, and with the story being told much further, I finally got the point how to use it. To decode the AX.25 header’s callsigns one can do (in kaitai struct):

meta:
  id: ax25_decoder
  endian: be
seq:
  - id: ax25_header
    doc-ref: 'https://www.tapr.org/pub_ax25.html'
    type: hdr
    size: 16
  - id: ax25_info
    type: str
    encoding: ASCII
    size-eos: true
types:
  my_str:
    seq:
      - id: callsign_str
        type: str
        size: 6
        encoding: ASCII
  hdr:
    seq:
    - id: dest_address
      type: my_str
      process: ror(1)
      size: 6
    - id: dest_ssid
      type: u1
    - id: src_address
      type: my_str
      process: ror(1)
      size: 6
    - id: src_ssid
      type: u1
    - id: ctrl
      type: u1

Using this kaitai struct I could get what I wanted: read callsigns of the AX.25 header in human readable ASCII inside the kaitai web IDE. But it was more than that! I suddenly got a tool on my hands to process raw telemetry data and I wanted more!

Time came and fortunately one of the team members of the UCLA ELFIN STAR team joined the IRC chat where lots of the SatNOGS team and contributors are around to talk. He was very interested in seeing observation data from their satellites first orbits. He was very cooperative and gave informations essential for further decoding of the satellites telemetry data.

Thanks for that man :wink:

I went off to the awesome #oscw18 to meet the core SatNOGS and Libre Space members and even some of the contributors beside me made it to Madrid. It was a great experience to meet all that people and it was really fun.

One evening I started poking around with the telemetry frames of the ELFIN satellites. I had great talks about that with all the guys staying with me. So I decided to go for a complete dashboard of these two satellites: complete the already started kaitai struct, get behind the secret of compiling them :wink: and have a start in grafana for the visualization. But you want to validate that structs with some data and it was nearly impossible to get valid frames from these satellites. I decided to write a message to their twitter account and see what happens. I also had that guy in the IRC chat in mind, but he was busy and not very much around to talk (what is absolutely understandable in such times). But guess what? I got answer on twitter. He told me about the complete frame structure (and that there are some very special specialities) and gave a link to the conversion rules for the telemetry values. What a great conversation! I was trying to figure out how I could put all that information into the kaitai struct, so I went back to the guys on the kaitai chat. One told me: 'this is the reason we have implemented ‘user-processes’ and I was remembering the “ror”-built-in. Poking around a bit and reading a lots of docs finally gave me the idea behind that. I made an implementation of a so called “ELFIN-preprocessor” in python and that worked just out of the box! Great! :smiley:


Need to make a break here now, to be continued asap!

P.S.: for the current results, but still WIP, visit: dashboard-dev.satnogs.org and watch the graphs :slight_smile:

7 Likes

I have to say that this stuff is absolutely brilliant. It is a missing piece of the SatNOGS jigsaw that is crucial. I have no idea about the technical stuff, it’s way over my head but as a bystander I am blown away with it.

Enough of the nice words…get back to work and make SatNOGS even more awesome. We’re right behind you (miles behind :grinning:)

1 Like

Great work.
I think that once ax.25 is decoded, you will see a lot of amateur radio ops (hams) joining this group.

APRS should also fall into place.

Thanks for supporting my favorite hobby.

Hey @DL4PD – awesome work! (And nice to meet you at OSCW :smile:)

Poking around, I’m guessing these are the two dashboards you’ve made for the Elfin satellites:

Is that correct?

(Man, I love Grafana…I’m trying to convince $WORK that we should be using this instead of the truly terrible product we’re using now.)

1 Like

Yes these are the ones.

@DL4PD great write up!

I want to mention here, how important is the satellite owners/teams give details about the encoding of their transmissions. Unfortunately many teams don’t give in public these details, which (and correct me if I’m wrong) are obligated to share when using amateur frequencies. Hopefully, and we are working on this direction, in the future these details will be available.

2 Likes

Transmissions of amateur radio station signals cannot be encrypted according to FCC in the USA.

Some, or most, amateur bands/frequencies are shared by government and or commercial entities whose transmisstions are regulated by different parts of FCC rules and regulations in the USA.

As a rule of thumb, private and collegiate owned/operated stations must be licensed and transmitt via FCC rules part 95 and part 97 of the rules and regulations.

Most other countries have different rules/laws.

Its not unusual to find pirated transmissions on the amateur bands.

I suspect that some of the satellite engineers choose the amateur frequencies because the fines and penalties are less than using the commercial frequencies when operating without a license and outside the law…

This is why I stated in a comment a few weeks ago that outlaw sats should not be treated the same as sat teams that follow internationa law.

just my 2c